Compare by just file time

[robnicholson-malt]

Love FreeFileSync - used it for years. Even got last company to contribute. One quirk I've come across is around OneDrive and Office files. It's a known problem with all synchronisation methods including good old robocopy.exe. The problem is simple - when Office files are uploaded to OneDrive (or SharePoint), extra metadata is added to the file. You can watch it in OneDrive - file uploads to OneDrive and then almost immediately downloads. The downloaded file has a different file size but identical timestamp. The contents are obviously different as well.

What this means is that if you run FreeFileSync again with file time and size as the compare option, it wants to re-copy most Office files. The issue can be seen in this screenshot:



It's only Office files that are affected. PDFs, JPEGs etc are fine. Now one can argue until the cows come home that OneDrive shouldn't be altering files like this. None of the other cloud services I've used do this. But you'd be banging your head against the wall.

The fix with robocopy is easy - /xc to ignore changed files. A changed file in robocopy is classed as one where it exists on source & target, has the same time stamp but different file sizes.

So it would be really cool if FreeFileSync you add File time alone to the compare options.

[xCSxXenon]

I do not have this issue with my OneDrive and Office files. I just confirmed with a docx that I saved locally, copied to OneDrive, and then compared the size and SHA-256 results. Files were identical
With that said, it wouldn't make sense to add a feature to a sync program that makes the program not sync anymore, just for very few isolated incidents.
I don't know what is causing your size discrepancy, but that is where I would focus my efforts

[robnicholson-malt]

Trust me, it's not an isolated incident. You will come across it at some point if you get involved in migrating data to SharePoint/OneDrive. It's documented here by Microsoft specifically "OneDrive adds additional metadata to Office documents after uploading, which changes the size of the file".

https://techcommunity.microsoft.com/t5/onedrive-developer/why-different-used-size-of-onedrive-in-my-files-app-and-onedrive/m-p/3968344

I'm currently migrating a lot of data from one system to SharePoint/OneDrive. The workaround is to one-way mirror from local source folder to OneDrive. Let OneDrive finish. Then swap over in FreeFileSync and copy the updated files back to source folder.

Once the metadata has been added, it's not added again. So maybe that's why you don't see the problem. The files you're comparing already have the metadata added.

[xCSxXenon]

Nothing on that page is officially from Microsoft. The OP is a Samsung app developer and the only response is from "Anonymous". Their response shouldn't be regarded as fact

[xCSxXenon]

In fact, searching the entire forum for anything related to changing size doesn't bring up anything related,which seems like it isn't a common issue

[robnicholson-malt]

It's a fact - I've migrated 20+ systems from a local file system to OneDrive/SharePoint. It happens with them all. Documented here:

https://myce.wiki/news/microsoft-onedrive-for-business-modifies-files-as-it-syncs-71168/

And also here by me four years ago where I contacted Microsoft and they confirmed OneDrive for Business (at the time) adds metadata to Office files alone thus changing the file size. Any system that uses hashes or compares the contents will think it's changed.

https://superuser.com/questions/1345933/why-is-onedrive-changing-the-size-of-files

And another discussion with a Microsoft agent where he says " I am afraid it is by design". So please stop suggesting I'm making this up. Why would I?

https://answers.microsoft.com/en-us/msoffice/forum/all/sharepoint-modifies-files-on-its-own/98c9644a-e614-401f-9ad6-1b12329fe87c

[xCSxXenon]

Based on Myce testing, we found that the consumer version of OneDrive (formerly SkyDrive) does not appear to any modify files

This is a very important distinction, as it seems the root issue is SharePoint and not OneDrive.
https://github.com/29039/29039s-useful-stuff/blob/main/m365-sharepoint-online-set-parserdisabled.md
This is for SharePoint, but I wonder if it or something similar is out there for OneDrive for Business.

[armirol]

It's a fact - I've migrated 20+ systems from a local file system to OneDrive/SharePoint. It happens with them all. Documented here:

I'm facing an issue with sharepoint and wonder if i can replace it with a solution that would use freefilesync instead.

we reach almost 500k files, 1.2TB of data and sharepoint sync is broken....
sometimes those files are updated, sometimes not.. we send links to clients and the files are updated...or not! Very hectic.

MS has a 100k (300k?) file limit and make you pay for extra storage and say they don't guarantee anything above 100k files... without talking about the rights modified issue (i stopped years ago the both way sync)

I wonder if you thought about or tested a solution replacing sharepoint.... any insights would be much appreciated, in particular how to have users sharing files easily... (2 days, and i have to say i wasn't able to find a similar solution, even commercial... AWS, oracle, IBM, and backblaze (still trying to test that one) are way to complicated for me and our need)

Thank you

[robnicholson-malt]

Is that 500k files in a single document library? If so, then yes, you're going to struggle. I see varying recommendations for maximum capacity per document library but I recommend not exceeded 50k in a single library. SharePoint itself can handle more but OneDrive starts to struggle.

MS might limit the number of documents in a library (every file system has limits) but they don't charge you based on the number of files. That 1.2TB figure is the basic you get with a Microsoft 365 tenant plus a few GB per user. You can buy additional storage as required and it's not that bad. IMO 1.2TB is a reasonable amount of space for a company of up to 50 if they're creating office (e.g. Word) documents. If you're generating a lot of graphics or video, I'd look elsewhere for cloud storage. Most big graphics companies still have on-premises servers and back up to the cloud.

I don't think FreeFileSync will help you significantly here. It still has to synchronise via OneDrive.

[armirol]

I don't think FreeFileSync will help you significantly here. It still has to synchronise via OneDrive.

We are an architecture company in healthcare... we only have huge and heavy projects.... i mistyped... it's 1.4Tb. We don't use onedrive, only sharepoint and i cannot reduce the number of files as the owner is asking for those.

How about we have an online external cloud storage and still use frefilesync... people are updating files on the fly and they need to be synchronized right away... i cannot find anything matching our needs. I do IT for the company as side part of my engineering job (though i built and manage the whole system). i have to acknowledge my limit of competences here.

I appreciate any advice.

thank you

[David.P]

I have kept advocating for this feature request for some time already.



I have tried basically every other synchronization program that comes to mind, and there is not a single one that can handle the SharePoint file size difference problem well, if at all.

I would love to use FreeFileSync as it is the best synchronization tool by far overall. So +1 one from my side once more on this feature request.

[robnicholson-malt]

Ahh glad I'm not alone. I was starting to feel paranoid when I kept been told above this isn't an issue/doesn't happen! Nowhere as nice an interface but good old robocopy can handle this situation with the /xc switch which ignores changed files: different file size but same name & timestamp.

[David.P]

Thanks for the info about RoboCopy. I suppose RoboCopy wouldn't be able to at the same time recognize renamed/moved files and folders, would it?

[robnicholson-malt]

Probably not as it's "dumb" in that it simply compares the the source and target files each run. It doesn't have a database like FreeFileSync to track file system changes.

[David.P]

Just wanted to briefly note that Syncovery recently has introduced an automatic feature that supposedly solves the file size change problem with SharePoint, apparently even without any special size ignore settings.

However, the setup and user interface of Syncovery is so convoluted and confusing that I have not yet been able to successfully verify this in Syncovery. And that's despite having used Syncovery for other purposes for years 💁‍♂️

Granted, Syncovery probably is the most complete (...and complex) Sync solution in existence on this planet.

[robnicholson-malt]

Thanks for the heads up on Syncovery. As a long IT support guy, cost is always a key factor but it's reasonably price even for the premium version.

[mopani]

Interesting, thanks for the heads-up about file system limits in SharePoint.

Syncthing might be a possible solution for getting files quickly synced between systems, but it isn't helpful when you only want to sync a subset of a complete library, or provide external web-links to documents.

[David.P]

I have just found out and tested that at least GoodSync seems to be able to solve the present issue.

GoodSync has a synchronization mode that directly connects to SharePoint via the "Microsoft Graph" interface. If this connection method is used, GoodSync uses checksums provided by SharePoint which apparently don't change even when the file sizes are changed in SharePoint.

This way, GoodSync can recognize files as identical even when they have different sizes.

[meteorquake]

Just to mention that Winmerge has an option that matches just on Modified Date (meaning Name+Modified) and may perhaps be of help in your task. You could easily do a folder (with subfolders) match and copy across all unmatched files from the one side and a delete on unmatched files on the other side.

[David.P]

One more note on the synchronization with GoodSync.

Somehow, GoodSync manages to upload files to SharePoint in a way that doesn't change any file's size after uploading. This seems to have to do with GoodSync using the Microsoft Graph interface.

This interface also seems to have the advantage that it is incredibly fast. It takes about five seconds for GoodSync to analyze the sync state of 80,000 files in 15,000 folders using the Microsoft Graph interface, over an internet connection which is not even particularly fast.

Anyway, maybe this helps to one day implement a working SharePoint synchronization with FreeFileSync as well.

[therube]

(Beyond me, but, Overview of Microsoft Graph.)

[anon828683]

Hi guys. I came to this topic late but I found a solution that works for me. Mind you, I only use mirror mode to backup my data so if you use a different mode, it may not work.

Anyway this is what I did:
- In the Synchronization tab, choose Custom.
- Uncheck "Use database file to detect changes".
- In the Difference section, choose "Do nothing" if "Right side is newer".
- The files changed in size by Onedrive will be marked as "conflict" and won't be synced, hence stopping the vicious cycle.

Clipboard Image.jpg (43.07 KiB) Viewed 961 times

Hope this helps!

[Plerry]

Which is exactly what is described in viewtopic.php?t=11742#p45588.

This does however not change the Compare (as requested in this topic),
but only affects the (proposed) sync actions, based on the Compare results.
Nevertheless, the outcome may be the same.

[robnicholson-malt]

No I don't think this will help as the problem is the file size changes, not the timestamp when OneDrive adds metadata to the office file. Hence the reason /xc (exclude changed) switch in robocopy solves the problem. This switch ignores the file if the timestamp is identical but the file size changes. It's not 100% safe but probably 99.99% as I can't think of any instance where the file would be changed without changing the time.

I mean yes, you could change the contents of a file and then change it's timestamp back to the original but why would you do that!

[Zenju]

Apparently SharePoint can be fixed! This is the real solution rather ignoring something as fundamental as file size: viewtopic.php?t=9562#p47161