Sometimes I encounter a problem after synchronizing: some files, which are basically text files (html, css, js, php, ...), are some sort of corrupted on the destination drive (environment: two Windows-PCs and a NTFS formatted HDD as NAS) - in fact they are no more text files. It only happens when I sync my folders with FFS - using FreeCommander for instance this problem doesn't occur.
After reading about the way FFS transfers files ((S)FTP) I assume that sometimes text files are transferred in binary mode - could this be?
The last time it happened I collected the "corrupted" files as well as their originals - maybe this is helpful for you to find out the reason for this problem (see attachment).
I googled for descriptions of this problem or solutions but couldn't find any. Of course I tried suggestions I found in your FAQ's or in your forum (e.g. verification) but nothing helped.
I really like FFS and would be happy to continue using it (and then donate to you for your work) - but fixing this problem is crucial to me.
Problem with "corrupted" text based files
-
- Posts: 5
- Joined: 13 Dec 2022
- Attachments
-
- ffsCorruptedFiles.zip
- (6.51 KiB) Downloaded 72 times
-
- Posts: 2287
- Joined: 22 Aug 2012
FreeFileSync does not (or at least: is not supposed to) change/modify file content when copying.
So it seems there must have been a corruption during copying.
You can try to do a verification after copying, by setting the VerifyCopiedFiles flag to true.
So it seems there must have been a corruption during copying.
You can try to do a verification after copying, by setting the VerifyCopiedFiles flag to true.
- Posts: 944
- Joined: 8 May 2006
Is this a local drive (direct attached or on a LAN)?NTFS formatted HDD as NAS
If it is a local drive, I would think you are not using FTP (though it is possible you are)?the way FFS transfers files ((S)FTP)
Are you using FTP?
(If you were, you would have had to click the "cloud" icon & specifically set it up.)
Since you have found corrupt files (& they certainly are, complete with the same date/time, i.e. silent corruption), I would do a more general compare of source & NAS held files & see if there are not more that are in fact corrupt. Might turn out to be a dying (NAS) drive. Similarly run a CHKDSK on the drive...
What type of NAS is this?
https://louwrentius.com/what-home-nas-builders-should-understand-about-silent-data-corruption.html
https://support.google.com/cloud/answer/10759085?hl=en
-
- Posts: 5
- Joined: 13 Dec 2022
Dear Plerry, dear therube,
thanks to both of you for your quick answers.
To answer your questions/suggestions:
VerifyCopiedFiles: I enabled this feature after I discovered this problem the first time - but it didn't make any difference (in fact I don't know how the verification works).
NAS: My network consists of two Windows PC's (one 10, one 11), a FritzBox as modem and router, and a NTFS-HDD connected to the FritzBox via USB - on my PC's this drive is integrated as a network drive, so it acts like a local drive.
And: No, I'm not using FTP - I'm sorry, this was just a quick shot from my mind.
But the hint of a "dying drive" is interesting: A couple of months ago I had an issue with this drive (I forgot that, but you brought it back to my mind) which caused me re-formatting it and I also ran CHKDSK on it - but it said that everything is ok.
And thanks for the article about "silent corruption" - I had no idea of that.
I will follow the suggestion of searching for other corrupted files - I know a way to find corrupted text-files, but not how to do so with binary files ... I'm gonna search for solutions, but if you know any tool I'd be very grateful.
I'm gonna observe this issue, make tests with other drives and whatever comes to my mind ... and I'll let you know if I'll find something.
Cheers ... and merry Xmas and a happy new year
patman
thanks to both of you for your quick answers.
To answer your questions/suggestions:
VerifyCopiedFiles: I enabled this feature after I discovered this problem the first time - but it didn't make any difference (in fact I don't know how the verification works).
NAS: My network consists of two Windows PC's (one 10, one 11), a FritzBox as modem and router, and a NTFS-HDD connected to the FritzBox via USB - on my PC's this drive is integrated as a network drive, so it acts like a local drive.
And: No, I'm not using FTP - I'm sorry, this was just a quick shot from my mind.
But the hint of a "dying drive" is interesting: A couple of months ago I had an issue with this drive (I forgot that, but you brought it back to my mind) which caused me re-formatting it and I also ran CHKDSK on it - but it said that everything is ok.
And thanks for the article about "silent corruption" - I had no idea of that.
I will follow the suggestion of searching for other corrupted files - I know a way to find corrupted text-files, but not how to do so with binary files ... I'm gonna search for solutions, but if you know any tool I'd be very grateful.
I'm gonna observe this issue, make tests with other drives and whatever comes to my mind ... and I'll let you know if I'll find something.
Cheers ... and merry Xmas and a happy new year
patman
-
- Posts: 3607
- Joined: 11 Jun 2019
May want to look at the SMART data for that driveBut the hint of a "dying drive" is interesting: A couple of months ago I had an issue with this drive (I forgot that, but you brought it back to my mind) which caused me re-formatting it and I also ran CHKDSK on it - but it said that everything is ok. patman, 14 Dec 2022, 08:59
-
- Posts: 5
- Joined: 13 Dec 2022
Hi there,
first I want to say that I love your forum: quick and friendly answers!
Sencond I want to thank xCSxXenon for the hint looking at the SMART data of my NAS-HDD: I used HDDScan and it said that everything's ok.
So I made a number of tests ... and some progress:
I copied the source directory (more than 2,500 files) from one Windows-PC to my NAS-HDD several times into different (empty) directories - the pheonomenon of "corrupted" files only occured when copying with FFS! Everything was ok when I used Windows Explorer or FreeCommander (every time I did a "quick search": looking for ASCII-files containing characters that only occur in binaries).
After that I made a "deep comparison", which means a directory sync with FreeCommander comparing the file's contents - the result was that I discoverd binary files (png) which were different, too.
What a surprise when I opened them in a text editor: their contents were parts of svg-files!
Please see the attachment: You'll find the "original" png's as well as the "corrupted" ones and the origins of their contents - I was able to find the svg-files containing the contents of the "corrupted" png's - they come from a completely different subdirectory:
png's subdir:
svg's subdir:
Can you give me an explanation for that?
Cheers
patman
first I want to say that I love your forum: quick and friendly answers!
Sencond I want to thank xCSxXenon for the hint looking at the SMART data of my NAS-HDD: I used HDDScan and it said that everything's ok.
So I made a number of tests ... and some progress:
I copied the source directory (more than 2,500 files) from one Windows-PC to my NAS-HDD several times into different (empty) directories - the pheonomenon of "corrupted" files only occured when copying with FFS! Everything was ok when I used Windows Explorer or FreeCommander (every time I did a "quick search": looking for ASCII-files containing characters that only occur in binaries).
After that I made a "deep comparison", which means a directory sync with FreeCommander comparing the file's contents - the result was that I discoverd binary files (png) which were different, too.
What a surprise when I opened them in a text editor: their contents were parts of svg-files!
Please see the attachment: You'll find the "original" png's as well as the "corrupted" ones and the origins of their contents - I was able to find the svg-files containing the contents of the "corrupted" png's - they come from a completely different subdirectory:
png's subdir:
sites\pms\views\janko\img
devEnv\xtres\content\img
Cheers
patman
- Attachments
-
- fileCorruption.zip
- (530.78 KiB) Downloaded 66 times
-
- Posts: 2287
- Joined: 22 Aug 2012
This sounds like a case where you have multiple (at least 2) left-right base folder pairs and a full or partial overlap between those base folders, or any of their (sub)folders.
FFS should give you a warning if that is the case, but you may have selected to ignore warnings.
Example:
If you sync A => C
and sync B => C
C forms said overlap.
If there are identically named files (e.g. your png's) in the same relative folder tree position in A and in B, there is no telling which file (the one from A or the one from B) ends up on C.
FFS should give you a warning if that is the case, but you may have selected to ignore warnings.
Example:
If you sync A => C
and sync B => C
C forms said overlap.
If there are identically named files (e.g. your png's) in the same relative folder tree position in A and in B, there is no telling which file (the one from A or the one from B) ends up on C.
- Posts: 291
- Joined: 13 Apr 2017
I have seen this type of corruption when I had my pc set to dual boot with different Windows versions.
The mistake was to hibernate then boot with the other Windows version, or power off but with "Fast startup" active in the Power options.
This caused a lot of cross-linked cluster on disk resulted in file corruption very similar to your.
Does this ring a bell?
EDIT:
Sorry, I missed that the copy destination is a NAS, so my post is irrelevant.
The mistake was to hibernate then boot with the other Windows version, or power off but with "Fast startup" active in the Power options.
This caused a lot of cross-linked cluster on disk resulted in file corruption very similar to your.
Does this ring a bell?
EDIT:
Sorry, I missed that the copy destination is a NAS, so my post is irrelevant.
Last edited by Gianni1962 on 22 Dec 2022, 13:29, edited 2 times in total.
- Posts: 944
- Joined: 8 May 2006
That FFS "caused" the issue, I'd say is sheer coincidence.I copied the source directory (more than 2,500 files) from one Windows-PC to my NAS-HDD several times into different (empty) directories - the pheonomenon of "corrupted" files only occured when copying with FFS! Everything was ok when I used Windows Explorer or FreeCommander
Try doing it again & again & again - with FC or WE & see if they too then don't end up with corrupt files.
Other then for a casual look, which is fine for as far as it goes, a "deep compare" (either by file content or by hash) is the only way to know.After that I made a "deep comparison", which means a directory sync with FreeCommander comparing the file's contents - the result was that I discoverd binary files (png) which were different, too.
Again, I'd attribute that to coincidence.What a surprise when I opened them in a text editor: their contents were parts of svg-files!
Take a look at your earlier files that also corrupted, yet they contain absolutely "nonsense" data (where these /happen/ to have, somewhat related, ascii data - coincidence).
I had a Team USB Flash Drive, viewtopic.php?t=9681#p35680, that corrupted data silently & regularly - without regard to anything else. I could CHKDSK the drive till I turned blue. I could format & reformat. I could copy file upon file upon file to the drive - & everything checked out - absolutely.
And then... out of the blue, at any point in time, even just after having verified everything, I'd copy something else to the drive, & I'd catch that it was corrupted. And then I could go back to those other files that I had just confirmed were valid, & I'd find corruption in there too. I could be a few days into using the drive, or it could be immediately after a format that I'd find corruption. Simply absolute garbage.
Tell us about the drive.NTFS-HDD connected to the FritzBox via USB
Are the two PC networked together, set up to be able to talk to one another, outside of their common connection to the FritzBox?
If you were to connect the NAS drive directly to one of the PC's (via USB), can you still confirm corruption?
-
- Posts: 5
- Joined: 13 Dec 2022
Hi there,
thanks to everybody who was interested in my problem - now I have to say that I'm gonna return to FreeCommander's sync - it's a real shame, because FFS is much more comfortable ... but for me it's unreliable. I can't remember having issues with corrupted files, so this had to be a very rare situiation - but with FFS I'm regularly confronted with it.
After making a bunch of tests it seems that therube is right - but not the way you might think: it seems to be coincidence that FFS doesn't work for me.
Let me tell you - I set up a simple test scenario:
• I took one source folder (A) on one (and only one) of my local machines and a newly created (therefor empty) destination folder on my NAS-drive. And for each test-run I created a new destination folder and kept the previous ones - let's call them Bn (B1, B2, ...).
• Then I asked my applications (FFS, FC) to synchronize A and Bn - and for the fact that the destination folder was empty the whole content of the source folder had to be copied.
• The result of the tests with the first source folder was always the same: no corruption when using FC (or WE when doing a simple copy), corruption every time I used FFS (I tried 3 times with FFS and 3 times with FC or WE). And even more than that: every time the same files were corrupted.
• After these tests I switched the source folder (in the end I used 5 different source folders): some of them were copied correctly by FFS, others weren't - and the same phenomenon: re-running the sync with FFS resulted in the same corrupted files.
• After that I plugged the NAS-drive directly into my local machine (USB): No corruption when using FFS!
In the end I had more than 20 destination folders.
So it seems to be a problem with the communication of FFS and my NAS-drive via WLAN. Or a problem with the specific files that were corrupted, but also only for FFS. Or ... well, I don't know.
@Gianni1962: My problem occurs in a less complex situation.
@Plerry: I think that your example doesn't meet my tests, does it?
@therube: As I said, this seems to be coincidence in my environment - but it's definitly no coincidence which files are corrupted for there were always the same ones.
And to answer your questions:
Cheers
patman
thanks to everybody who was interested in my problem - now I have to say that I'm gonna return to FreeCommander's sync - it's a real shame, because FFS is much more comfortable ... but for me it's unreliable. I can't remember having issues with corrupted files, so this had to be a very rare situiation - but with FFS I'm regularly confronted with it.
After making a bunch of tests it seems that therube is right - but not the way you might think: it seems to be coincidence that FFS doesn't work for me.
Let me tell you - I set up a simple test scenario:
• I took one source folder (A) on one (and only one) of my local machines and a newly created (therefor empty) destination folder on my NAS-drive. And for each test-run I created a new destination folder and kept the previous ones - let's call them Bn (B1, B2, ...).
• Then I asked my applications (FFS, FC) to synchronize A and Bn - and for the fact that the destination folder was empty the whole content of the source folder had to be copied.
• The result of the tests with the first source folder was always the same: no corruption when using FC (or WE when doing a simple copy), corruption every time I used FFS (I tried 3 times with FFS and 3 times with FC or WE). And even more than that: every time the same files were corrupted.
• After these tests I switched the source folder (in the end I used 5 different source folders): some of them were copied correctly by FFS, others weren't - and the same phenomenon: re-running the sync with FFS resulted in the same corrupted files.
• After that I plugged the NAS-drive directly into my local machine (USB): No corruption when using FFS!
In the end I had more than 20 destination folders.
So it seems to be a problem with the communication of FFS and my NAS-drive via WLAN. Or a problem with the specific files that were corrupted, but also only for FFS. Or ... well, I don't know.
@Gianni1962: My problem occurs in a less complex situation.
@Plerry: I think that your example doesn't meet my tests, does it?
@therube: As I said, this seems to be coincidence in my environment - but it's definitly no coincidence which files are corrupted for there were always the same ones.
And to answer your questions:
Western Digital My Passport Ultra 2TB, several years old.Tell us about the drive:
No - mostly I only work with one at a time, the other one's turned off. But having my tests in mind, I think that this is irrelevant.Are the two PC networked together, set up to be able to talk to one another, outside of their common connection to the FritzBox?
Cheers
patman
-
- Posts: 3607
- Joined: 11 Jun 2019
I wonder if the NAS has an update that would fix this issue. That's been a common occurrence in these parts
-
- Posts: 5
- Joined: 13 Dec 2022
Good idea, but I fear that I can't force that for my FritzBox is provided by my internet provider, so I'm dependent on updates by them - on the other hand the device info says that OS is up to date ... it will remain a secret for an unknown time
