Google Drive upload speed suddenly throttled?

[ccarcit]

I discovered this weekend (10.12.2024), after months of trouble free backups using FFS from server to Google Drive, that my syncs from several servers to Google Drive using FFS have slowed down to a fraction of what they were. On the server I am testing, it was cut to about 10% of what the previous speed was, from 10-15 MB/s to 1 or 2 MB/s.

I ran speed tests and the upload speed was reported at what was expected. I used the Google Drive web client to perform an update of one of the files to sync, and THAT was fine. My conclusion is that the connection mechanism that links FFS to Google Drive has experienced a change or reached some limit that has caused this reduction in upload speed. I have seen this phenomenon at multiple sites, after multiple modem, router, and server reboots, so I am confident this is not a network issue.

Anyone else?

[ccarcit]

Forgot to mention this is on Windows Server 2019 hosts

[ccarcit]

[Edit by Zenju: Leo2's post has been removed]

I am not sure how this observation is supposed to help me or anyone else experiencing this. I have not had any issues with the FFS connection to Drive until the last week.

One of the tests I tried was using the Google Drive client on a server. I did not get far enough to definitely say it would transfer files at the expected speed because the Drive client caches the files on the system drive of the machine in the user's appdata folder. In my case I am transferring a little over a terabyte of data, and about a third of the way thru the transfer my system drive filled up and completely ran out of space.

Additionally, using the Drive client in this case is useless because it doesn't run as a service, only when the user is logged in. That makes it worthless for a system admin like me who needs these backups unattended every week.

I say all that to tell you that your observation may be correct on the surface, but has no relevance in this case. I need insight specific to my use case, which I seriously doubt is unique.

[ccarcit]

[Edit by Zenju: Leo2's post has been removed]

Can you please just stop replying? You have now twice ignored the glaringly obvious point I made that ALL OF THIS WAS WORKING FINE UP UNTIL THE LAST WEEK!

You're also disregarding the fact that I am not talking about the syncs failing because I got a data transfer limit, of which I am well aware, and have set up this backup using more than one account to avoid this cap. This has NOTHING TO DO WITH DATA TRANSFER SPEED!

So to reiterate: this was working as expected at my broadband provider's published upload speed until the last week.

AND: Stop responding. You're missing several points and it's not helpful. Change data storage providers? Seriously? It's been like talking to first level support when you've laid out all the pertinent details and explained that you've taken the first five steps they ask you to do, and STILL being told they want you to reboot them modem.

[ccarcit]

And just to make sure someone doesn't direct me to information about Google's upload speed cap, I know it's 45MB/s. So please, from here on out, relevant replies only.

https://www.google.com/search?q=google+upload+speed+cap&oq=google+upload+speed+cap&gs_lcrp=EgZjaHJvbWUyBggAEEUYOdIBCDYzODZqMGo0qAIBsAIB

[ccarcit]

@zenju? Been emailing you about this and haven't gotten a response. And no input here so far that's useful. Starting to wonder if it's time to find another product, which would suck, because FFS is the best option I've found so far.

[xCSxXenon]

Considering FFS hasn't updated in four months, it seems obvious to me that nothing in FFS has to do with your issue. https://freefilesync.org/archive.php
Also, nothing has changed with Google Drive's API in over a month either. https://developers.google.com/drive/docs/release-notes

I have seen this phenomenon at multiple sites, after multiple modem, router, and server reboots, so I am confident this is not a network issue.

I would say this rules out anything on your side, which leaves Google to blame. Have you tested with a different GDrive account?

[ccarcit]

Thank you for responding!

Yes, I tried a different account this morning. No difference. All my sites use different accounts to perform the uploads, so I didn't suspect it was any sort of user-based restriction either.

And I agree with you that it's not FFS and not an issue with the network on my side. I just got done checking the changelog for the Drive API and saw that there hasn't been a change. Many Googlings later, I am seeing that there is a long-standing issue with the API throttling transfers to keep abuse low, but even at my top upload speed I am only hitting about 15MB/s.

I can accept that this is something that nobody has any control over. I was just looking for acknowledgement or corroboration. This is tremendously frustrating, because it took me months to get the FFS to GDrive uploads working consistently after fine tuning timeouts, threads, and internal server complication, and then this happens.

I neglected to include the fact that this is a Google Workspace for NP domain I'm working with.

[xCSxXenon]

Considering you have a contractual subscription with Google, you may look into getting direct support from them. This could at least result in an official answer as to whether it is quickly fixable or if it really is an issue with Google's entire API framework. In the end, this is a big reason I moved to OneDrive. The black box known as Google would always have weird things like this that were dealt with by waiting and praying, not actual definitive actions. I hope you can figure out a resolution. I'm guessing all the different accounts are under the same Workspace domain, so having an issue with multiple accounts in that configuration is basically still the same account.

[ccarcit]

Yes, I was hoping to avoid having to open a support ticket but that seems like the only path to answers. A solution would be awesome. Answers at least provide more direction, even if it's not the one I found the most effective.

All the accounts are indeed under the same domain. I have a couple less elegant options I can pursue (SFTP > Synology NAS or MultCloud account > Google Drive sync job) but when I had these in place in the past, it was problematic. Establishing a different cloud storage provider is very, very last resort based on the volume of data being transferred every week (> 2TB). As an NP, cost is a primary concern.

I really appreciate the feedback here. As a one person IT department, I rely heavily on the experience and insight of others to help me get information on arcane issues like this. If Google support can give me information that is relevant to this situation, I will add it to this post so that others who might have this issue can benefit.

Thanks again!

[ccarcit]

I submitted a support ticket to Google for this, but I am quite confused about a discrepancy between what the FFS client shows as the upload speed vs what perfmon shows when I did a comparative test using SFTP. The FFS client shows 3-4 MB/s but perfmon shows 14-15 MB/s. The perfmon on the NAS being transferred to shows the same D/L as the FFS client.

https://photos.app.goo.gl/vMwyEC6MnDg3uYFv7

[xCSxXenon]

Check what that 120Mbps is actually going to from the processes tab, and sort by network usage by clicking the column header. There's clearly some overhead playing a factor

[ccarcit]

Well this was unexpected. The overhead, inexplicably at 50% of the available bandwidth, is allocated to the VPN service I used for this particular experiment (rather than reopening a port on my router). It is at least running closer to expected speed than yesterday.

https://photos.app.goo.gl/PRiLFog7AD5a6Vin7

When I connect through the NAT to the NAS, all the numbers line up as they should.

https://photos.app.goo.gl/7Se8NZEnY7YuViQh8

[ccarcit]

I submitted a support ticket to Google for this...

And their reply...

Hello,

Thank you for your prompt response.

To help me troubleshoot your recent issue, I'll need your Google Cloud Platform (GCP) project ID and number.

To answer your queries, think of your GCP project as a container that holds all your GCP resources, like virtual machines or storage buckets. The project ID is a unique name you assign to the project, while the project number is a system-generated identifier.

You can find both the project ID and number within the GCP Console. This article provides a detailed guide on identifying your project [1].

Additionally, could you please mention the API method you are currently using?

Once you have this information, please reply to this email, and I'll be happy to assist you further.

Kind regards,

Nicolas
Google Workspace Support
Bucharest, Romania

I explained the details to the chat support (including the fact that this was not a project I owned or had the details for) before I got the ticket generated, but that didn't seem to have made it into their notes. So I sent the chat transcript to them rather than type it all out again.

[therube]

(screenshot has your .org name in it, if that matters.)

[ccarcit]

(screenshot has your .org name in it, if that matters.)

I appreciate you bringing that to my attention. I will redact the SFTP details I didn't catch in one of them.

[xCSxXenon]

This sounds like an issue with the VPN host/client at this point am I correct? Something in that is handling the data in an odd way. Do you know if TailScale has had any changes?

[ccarcit]

This sounds like an issue with the VPN host/client at this point am I correct? Something in that is handling the data in an odd way. Do you know if TailScale has had any changes?

I should have more explicitly explained that I was using the Tailscale connection strictly for the purposes of this experiment to further verify the upload speed was performing as expected using FFS with an SFTP connection vs the Google Drive connection. I have never actually used Tailscale for this purpose, so the fact that it was slower was more of an "oh that's interesting" observation. I only ever use Tailscale for remote desktop connections to servers. It isn't something I intend to further investigate in the context of the SFTP connection.

When I connect directly to my NAS via SFTP by using the custom NAT port on my router, it behaves as expected and achieves the maximum upload speed using FFS with the SFTP config. Just a further confirmation that it isn't FFS that's at the center of the reduced speed mystery, it's the Google Drive API.

[ccarcit]

Not surprised by support's response. Looking like it's time to try rsync, which is just another level of complication.

Hello,

Thank you for your prompt response.

I understand your concern and want to assist you in any way I can.

However, I would like to clarify that we generally do not offer direct support for third-party applications that are not developed by us or our customers. Please note that even if third-party tools, such as GAM, Clasp, FlashPanel (developed by BetterCloud), Marketplace or Chrome apps, are using Google APIs, they are not supported, as highlighted in this Google Support article [1].

While we value your business and are committed to providing excellent service, our primary focus is on supporting our own products and services. We recommend reaching out to the developer of the application you’re having trouble with directly for specific troubleshooting advice and technical support.

You may find helpful information and troubleshooting tips on the developer's website or in their online forums.

Should you have any other questions, please do not hesitate to reach out.

Kind regards,

Nicolas

[Zenju]

our primary focus is on supporting our own products and services.

This is really the key information.

We recommend reaching out to the developer of the application you’re having trouble with directly for specific troubleshooting advice and technical support.

How should the developer know, if and why Google is applying bandwidth throttling?

FreeFileSync is 100% subject to Google's good will and graces. Google's own software is using a slightly different API. So they have the capabilities to disadvantage third-party access at any time they like.

[ccarcit]

How should the developer know, if and why Google is applying bandwidth throttling?

Since I know nothing about how the API is utilized with FFS (is it a GCP project with the identifiers that support was asking for?) then I have no idea.

FreeFileSync is 100% subject to Google's good will and graces. Google's own software is using a slightly different API. So they have the capabilities to disadvantage third-party access at any time they like.

My final question to them was "does Google apply such throttling under certain circumstances, and if so, what are those circumstances and what is the threshold to trigger that?" Still waiting for their response but I am not holding my breath for anything concrete.

My final question here is whether a similar issue has ever been reported from other FFS users. I can't wrap my head around why this process worked flawlessly for MONTHS and then this issue manifested so uniformly, across multiple sites and accounts, out of the blue, given that every other method of data transfer is operating as expected.

Absent any concrete information, I am forced to surrender and rebuild my most critical backup schema from scratch. Again.

[Flowers4]

Hi,

Zenju can clarify this but my research shows that:

FreeFileSync doesn't directly use the Google Drive API for connectivity
. Instead, it uses its own method to sync files with Google Drive
. While this works well for basic file synchronization, it might not offer the same level of control or efficiency as using the API directly.

The Google drive API is configured here: https://console.developers.google.com/

FFS would need to be updated, I believe, to support using your own Google account Drive API. Some applications let you connect to your Google account and have the ability to specify your own Google Drive API configured in the developer console. When I do this I get vastly superior performance Over using the built in Google connectivity button.

I use my own tool to connect to Google Drive that incorporates an API that I activated in my developer console at Google.
Going this way, I get 600 megabits transfer rate when synchronizing files to Google with FFS.

[ccarcit]

Hi,

Zenju can clarify this but my research shows that:

FreeFileSync doesn't directly use the Google Drive API for connectivity etc

You have my attention and I would like to subscribe to your newsletter! 😂

I'd be very interested in finding out more about what you're doing and start digging into that. Not sure if we're allowed to put our email here but if you're willing to provide more info I'd like to take this offline.

[ccarcit]

I swear to FSM that trying to navigate all the anomalies, glitches, and numerous random bitflips I have encountered in my IT career has pushed me to the edge of sanity more times than I can count. This has definitely been one of those times.

After numerous server, router, and gateway reboots to try and resolve this with no success, I decided to try this sequence one more time with the server that's been most impacted by this problem, because what did I have to lose.

And tonight, that worked. I am sitting here watching this server pump data to Google Drive using FFS at the full upload bandwidth capacity of ~13MB/s. I keep checking expecting to see the transfer speeds plummet, but it's holding steady.

I want to understand why this really happened but I want to move on with my life more. So on we go.

Thanks for the input, all.

[Flowers4]

@ccarcit,

Hi,

First, here is the Google doc for authorizing apps to both login, and use the API.
https://developers.google.com/identity/authorization

The catch is that each user would need to login to the console using their Gmail account and enable it to use the Google Drive API. Part of the setup lets you create the necessary client ID and secret key for the app to use the Google Drive API in your account. This is not very straightforward for the novice. Basic overview instructions could be written and provided in the docs.

I don't have a newsletter to subscribe to however I did generate a temporary alias that you can use and I will disable it immediately after you contact me so it doesn't remain a valid email I on the website.

I will expire this alias address making it invalid/useless after you reach me.

[ccarcit]

Possibly the most hilariously appropriate alias ever created! Well played, you legend!

[Zenju]

Hi,

Zenju can clarify this but my research shows that:

FreeFileSync doesn't directly use the Google Drive API for connectivity etc

You have my attention and I would like to subscribe to your newsletter! 😂

Sign me up as well! Because this claim doesn't make any sense! :D

[ccarcit]

I spoke too soon about a reboot of all the components fixing things. At some point between when I checked the server I've been testing and saw that it was hitting max upload speed and about 2 hours later, the speed had dropped from ~13MB/s to sub 1MB/s speeds. Somehow it actually got worse.

And none of the other servers shoed any improvement at all after all the hardware was reset.

[therube]

(

So they have the capabilities to disadvantage third-party access at any time they like.

Not saying it is the issue here, & maybe it is not deliberate :evilgrin:, or maybe it is, & they'll never say so, but...

Google disrupted YouTube video playback on Firefox, again
)

[Flowers4]

Hi,

Zenju can clarify this but my research shows that:

FreeFileSync doesn't directly use the Google Drive API for connectivity etc

You have my attention and I would like to subscribe to your newsletter! 😂

Sign me up as well! Because this claim doesn't make any sense! :D

@Zenju,

What makes the method that I use difficult for you to implement and deploy to your customers is that it requires each person to log into the Google developer console and activate the Google Drive API which is not active in each person's account by default and then that creates a client ID and secret code for accessing the drive API.

FFS would need to be modified to be able to capture and use the client ID and secret code provided by Google for the drive API and this is just not something that's easy to do for the everyday user that is using FFS. The average user is not going to want to or be able to easily follow the procedure in the Google developer console.

It's really only for advanced users.