FreeFileSync To Google Drive - Stop Sync When Receiving "User Rate Limit Exceeded", Other Errors Ignore

[itman2000my]

Hi,

Im currently using Google Drive as a backup option.

My batch sync option, I checked "Ignore Errors". However, when sync hits the "User Rate Limit Exceeded", FreeFileSync will still continue to try to upload to Google Drive. So, this will be a never ending retries until all the files are processed.

Hence, my advice is:
1. Checked "Ignore Errors" and FreeFileSync will ignore other errors, ie. cannot open file, etc
2. If error is "User Rate Limit Exceeded" from Google Drive, stop sync

Thanks

[Zenju]

Maybe enabling "retry" and setting a delay would be a better solution?

The "User rate limit" error may also be an indication that you're using too many parallel threads.

[itman2000my]

FFS will then retry x # of times per file x n seconds wait time. So, if you hv 100k files, then will be 10 retries x 10sec x 100000 files.FFS will continue for another 115days till complete.

[itman2000my]

For Google Drive, there is a 750GB upload limit for users to upload their files. If using API calls, then there is a set of other quotas involved.

[itman2000my]

Under "Drive upload & copy limits"

https://support.google.com/a/answer/172541?hl=en

[Zenju]

"User Rate Limit Exceeded" is not related to storage limits: https://stackoverflow.com/questions/68483840/how-to-resolve-a-403-error-user-rate-limit-exceeded-in-google-drive-api

[itman2000my]

Correct. Not related to storage limit. However it is related to UPLOAD limit. Google Drive allows upload of 750GB per day.

Once you hit the upload limit, that's where the User Rate Limit Exceeded error occur.

[Zenju]

I haven't tested, but if "User Rate Limit Exceeded" is returned by Google not only when hitting requests per second, but also for the 750GB per day limit, then both reactions, retry or abort, would be inappropriate in some cases.

[itman2000my]

There are 2 error codes returned by Google Drive which is found here:
https://developers.google.com/drive/api/guides/limits

1. Error 403 is critical. Hence we can abort the process
Quotas:
a. 750GB Upload Limit

2. Error 429 is not critical. Hence, this can be based on number of retries and delays. Eg. 12000 queries per 60 seconds.


QUOTE

If you exceed a quota, you'll receive a 403: User rate limit exceeded HTTP status code response.
Additional rate limit checks on the Drive backend might also generate a 429: Too many requests response. If this happens, you should use an exponential backoff algorithm and try again later. Provided you stay within the per-minute quotas below, there's no limit to the number of requests you can make per day.

Note: Depending on your type of Google Workspace account, there are additional Drive storage limits. Google Workspace users can only upload 750 GB per day between My Drive and all shared drives; this limit also applies to copies. Users who reach the 750 GB limit or upload a file larger than 750 GB can't upload or copy additional files until 24 hours have passed. The maximum file size that users can upload is 5 TB; only the first file that breaks the limit completes uploading. The maximum file size that users can copy is 750 GB.

END QUOTE

[Zenju]

If you exceed a quota, you'll receive a 403

Both "Queries per minute" and "Queries per minute per user" are quotas. And so is the 750 GB per day limit.
So the "403: User rate limit exceeded" alone isn't enough to distinguish these cases, unless the error json structure contains additional details.