ffs_tmp files created

[mcore]

Hello,

When trying to synchronize a local drive with a network drive (WD My Cloud), the sync UI is stuck, while the engine keeps creating an enormous amount of .ffs_tmp files (like a.jpg.ffs_tmp, a.jpg_1.ffs_tmp. a.jpg_2.ffs_tmp and so on, with a rate around 100 files per minute).

It used to work, but not recently.
I also have LiveDrive software to backup that network disk online, so I thought it might interfere with FFS. But closing it doesn't change anything. I start FFS (usually in non-interactive mode), wait for the scan to complete and then observe, that the amount of files to synchronize doesn't change. I then force-close FFS and search for ffs_tmp files on my network drive to delete them. Nothing else happens.

Please, help.

[Zenju]

What does the logfile show at the end?

[mcore]

Where can I find the log file?

[alexkamp]

Hi Zenju,

First, what a wonderfull program you've made. I am using it a few years now. Great work!

I still have this issue with latest version of FFS, 6.8.

I sync my files over Ethernet (Windows 8.1 Pro -> Samba server on Ubuntu server).
In my case it tries to copy an Hyper-V virtual machine XML description file. In the same directory an directory containing the 'basename' of XML filename exists. Hyper-V VM's are not active during syncing - turned off a few months.

Source and destination path both are not too long:
G:\Hyper-V\Linux x64\Ubuntu 12.10\Virtual Machines v.s. Z:\G\Hyper-V\...

Source file: 545EF19F-F107-4A10-8124-6E08CD9DEA64.xml
It creates the following:
545EF19F-F107-4A10-8124-6E08CD9DEA64.xml.ffs_tmp
545EF19F-F107-4A10-8124-6E08CD9DEA64.xml_1.ffs_tmp
[...]
545EF19F-F107-4A10-8124-6E08CD9DEA64.xml_28229.ffs_tmp
[...and so on...]

The log you referred at is located at C:\Users\(User)\AppData\Roaming\FreeFileSync (?). No log references of the error are shown there. I tried to find more information about the crash via Event Viewer (Windows), but it just showed details about the 'Application Hang'. This is actually not the error. Windows thinks the application is not responding any more.

The following options are tested - but resulting in same behavior as above.
- Removing XML at destination (operation update -> creation)
- Renaming directory having basename of XML file
- Removing parent directory 'Virtual Machines' (new creation of all)

To prevent a infinity loop, you can create a 'breaker' which breaks the tries f.e. after 10 times. Do you have some magics in the software so we can test this scenario or show more debug data?

During editing some other files and syncing I received the 'file cannot be copied' messages - which are correct. So that part of code is working...

[Zenju]

Hi Zenju,

First, what a wonderfull program you've made. I am using it a few years now. Great work!

I still have this issue with latest version of FFS, 6.8.

I sync my files over Ethernet (Windows 8.1 Pro -> Samba server on Ubuntu server).
In my case it tries to copy an Hyper-V virtual machine XML description file. In the same directory an directory containing the 'basename' of XML filename exists. Hyper-V VM's are not active during syncing - turned off a few months.

Source and destination path both are not too long:
G:\Hyper-V\Linux x64\Ubuntu 12.10\Virtual Machines v.s. Z:\G\Hyper-V\...

Source file: 545EF19F-F107-4A10-8124-6E08CD9DEA64.xml
It creates the following:
545EF19F-F107-4A10-8124-6E08CD9DEA64.xml.ffs_tmp
545EF19F-F107-4A10-8124-6E08CD9DEA64.xml_1.ffs_tmp
[...]
545EF19F-F107-4A10-8124-6E08CD9DEA64.xml_28229.ffs_tmp
[...and so on...]

The log you referred at is located at C:\Users\(User)\AppData\Roaming\FreeFileSync (?). No log references of the error are shown there. I tried to find more information about the crash via Event Viewer (Windows), but it just showed details about the 'Application Hang'. This is actually not the error. Windows thinks the application is not responding any more.

The following options are tested - but resulting in same behavior as above.
- Removing XML at destination (operation update -> creation)
- Renaming directory having basename of XML file
- Removing parent directory 'Virtual Machines' (new creation of all)

To prevent a infinity loop, you can create a 'breaker' which breaks the tries f.e. after 10 times. Do you have some magics in the software so we can test this scenario or show more debug data?

During editing some other files and syncing I received the 'file cannot be copied' messages - which are correct. So that part of code is working...

This could be a bug in the Samba implementation giving back incorrect error codes. When creating a new temporary file and the creation fails due to "file already existing", FreeFileSync retries using a different file name like `<number>._ffs_tmp`.
Currently there is no limit of the number of these files since a collision with an unrelated ffs_tmp file is extremely unlikely.
To verify this theory and in case you are able to reproduce the scenario in a minimal way, you can send me a Process Monitor log at the time while FFS endlessly creates these files on the Samba share.

[Zenju]

Hold on: I'm able to reproduce the issue with Win 7 against an Ubuntu Samba share! Investigating...

[alexkamp]

Hi Zenju,

I did a Wireshark dump on my Windows 8.1 and got the following:
Request: (filename).ffs_tmp
Response: STATUS_EAS_NOT_SUPPORTED (but created the file with zero bytes size)
Request: (filename).ffs_tmp again
Response: STATUS_OBJECT_NAME_COLLISION
Request: (filename).ffs_tmp again
Response: STATUS_OBJECT_NAME_COLLISION
(... some other comparing regarding directory information ...)
Request and Response pattern repeating now with suffix _#.ffs_tmp
After pattern the # will be incremented.
Loops until programm is force closed.

I see you've got a clue. The same clue as I found?

[Zenju]

Hi Zenju,

I did a Wireshark dump on my Windows 8.1 and got the following:
Request: (filename).ffs_tmp
Response: STATUS_EAS_NOT_SUPPORTED (but created the file with zero bytes size)
Request: (filename).ffs_tmp again
Response: STATUS_OBJECT_NAME_COLLISION
Request: (filename).ffs_tmp again
Response: STATUS_OBJECT_NAME_COLLISION
(... some other comparing regarding directory information ...)
Request and Response pattern repeating now with suffix _#.ffs_tmp
After pattern the # will be incremented.
Loops until programm is force closed.

I see you've got a clue. The same clue as I found?

It looks bad: Win32 "CopyFileEx" fails with ERROR_ALREADY_EXISTS although the target file did not exist - even worse, the function call in fact *created* the file itself.

This is either a Windows bug in CopyFileEx or a network driver bug that maps the Samba Share into Windows. Now FreeFileSync has to somehow find a way to deal with this mess...

[Zenju]

CopyFileEx makes two attempts at creating the target file: At kernel level it first calls NtCreateFile with extended attributes. This fails with STATUS_EAS_NOT_SUPPORTED. Then it calls NtCreateFile normally, but this also fails with STATUS_OBJECT_NAME_COLLISION (this is later mapped to Win32 error code ERROR_ALREADY_EXISTS).
It seems the first creation attempt failed to clean up and left a zero byte file behind, which is probably a bug in the Samba file system implementation on Windows.

[alexkamp]

It looks bad: Win32 "CopyFileEx" fails with ERROR_ALREADY_EXISTS although the target file did not exist - even worse, the function call in fact *created* the file itself.

This is either a Windows bug in CopyFileEx or a network driver bug that maps the Samba Share into Windows. Now FreeFileSync has to somehow find a way to deal with this mess...

Was your faulty file in your test linked to a service or 'hidden in-use activity'?
I checked the 'permissions' of my faulty file and it has an extra 'user' in UUID, I think related to Hyper-V. I can see file contents in Notepad++, copy the file and see file contents, but renaming it leads to a 'file is opened in 'Hyper-V Virtual Machine Management' message box. Maybe 'CopyFileEx' operation breaks by opening the file in 'some exclusive mode' or so.
I'm wondering if the initial post of Mike also had this issue...

[Zenju]

Was your faulty file in your test linked to a service or 'hidden in-use activity'?
I checked the 'permissions' of my faulty file and it has an extra 'user' in UUID, I think related to Hyper-V. I can see file contents in Notepad++, copy the file and see file contents, but renaming it leads to a 'file is opened in 'Hyper-V Virtual Machine Management' message box. Maybe 'CopyFileEx' operation breaks by opening the file in 'some exclusive mode' or so.
I'm wondering if the initial post of Mike also had this issue...

The trigger seems to be that the file must have some kind of NT extended attributes. I made tests copying files with ADS to Samba, but it works correctly. A certain file in my test however seems to have another kind of extended attributes and exposes the problem reliably when copying to Samba. In your case the extra user id may very well be saved as an extended attribute and so exposes you to the copying problems.
Interestingly there is no issue when copying the file with Windows Explorer, which is explained by the fact that Explorer does *not* use the default CopyFileEx API, but they use the alternative IFileOperation interface instead.

[alexkamp]

The trigger seems to be that the file must have some kind of NT extended attributes. I made tests copying files with ADS to Samba, but it works correctly. A certain file in my test however seems to have another kind of extended attributes and exposes the problem reliably when copying to Samba. In your case the extra user id may very well be saved as an extended attribute and so exposes you to the copying problems.
Interestingly there is no issue when copying the file with Windows Explorer, which is explained by the fact that Explorer does *not* use the default CopyFileEx API, but they use the alternative IFileOperation interface instead.

In addition to my post - and to feed the brainstorm regarding this issue:
Today I ran Handle (Windows Sysinternals) and got the following result.

[...]
vmms.exe pid: 2504 NT AUTHORITY\SYSTEM
[...]
5B0: File (R--) G:\Hyper-V\Windows XP\Virtual Machines\0C1E05B9-C8E6-4629-B66B-07B7BC2D97EB.xml
5B4: File (R--) C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines Cache\0C1E05B9-C8E6-4629-B66B-07B7BC2D97EB.xml
5C0: File (R--) G:\Hyper-V\Linux x64\Ubuntu 12.10\Virtual Machines\545EF19F-F107-4A10-8124-6E08CD9DEA64.xml
5C8: File (R--) C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines Cache\545EF19F-F107-4A10-8124-6E08CD9DEA64.xml
[...]

Looks like an file-locked-for-.... issue.

The problem can be solved in many ways, depending on available time for it, with or without successfully copy the file. Most important thing is that FFS should not crash. Some kind of 'better safe than sorry'.
Maybe it's a good start to limit the number of file copy retries to prevent hanging FFS. You can question yourself if an operation succeeds after 10 retries - even 10000.

Too bad I'm not familiar with win32 programming, but I can contribute choosing a different approach because my programming background.

[Zenju]

In addition to my post - and to feed the brainstorm regarding this issue:
Today I ran Handle (Windows Sysinternals) and got the following result.

[...]
vmms.exe pid: 2504 NT AUTHORITY\SYSTEM
[...]
5B0: File (R--) G:\Hyper-V\Windows XP\Virtual Machines\0C1E05B9-C8E6-4629-B66B-07B7BC2D97EB.xml
5B4: File (R--) C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines Cache\0C1E05B9-C8E6-4629-B66B-07B7BC2D97EB.xml
5C0: File (R--) G:\Hyper-V\Linux x64\Ubuntu 12.10\Virtual Machines\545EF19F-F107-4A10-8124-6E08CD9DEA64.xml
5C8: File (R--) C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines Cache\545EF19F-F107-4A10-8124-6E08CD9DEA64.xml
[...]

Looks like an file-locked-for-.... issue.

The problem can be solved in many ways, depending on available time for it, with or without successfully copy the file. Most important thing is that FFS should not crash. Some kind of 'better safe than sorry'.
Maybe it's a good start to limit the number of file copy retries to prevent hanging FFS. You can question yourself if an operation succeeds after 10 retries - even 10000.

Too bad I'm not familiar with win32 programming, but I can contribute choosing a different approach because my programming background.

Do you have Volume Shadow Copy active in FreeFileSync's global options (= default)? In this case the locked file should not cause trouble and will be copied like a normal file (but still one with extended attributes).

> Maybe it's a good start to limit the number of file copy retries

This is the least I will do, even in this sad situation of bugs in operating system core components. A break after like 5 or 10 retries should suffice.

Other ideas:

1. Have the Samba or Windows guys fix their code. This would be the best and from a global perspective cheapest solution for all parties involved. But is unlikely to happen soon, as this requires much more research, further most, find out contact information of developer who is respsonible or even go the long and dusty "official" road. Even then it may take years until the fix is rolled out on a large scale.

2. Use the same Windows Copy Routine like Explorer does. This may be the best solution from the user's point of view. Instead of an early failure (after a few retries), file copying will simply succeed. However replacing the core routine of a file sync tool should not be taken lightly and needs further extensive testing, reseach and performance tests.

3. Theoretically there's a hacky way to detect the problem before copying the file by checking for extended attributes. Drawback is one additional file access. And even if the situation is detected, it only helps reduce the number of retries to 1, i.e. helps to produce an early failure.

[alexkamp]

Do you have Volume Shadow Copy active in FreeFileSync's global options (= default)? In this case the locked file should not cause trouble and will be copied like a normal file (but still one with extended attributes).

> Maybe it's a good start to limit the number of file copy retries

This is the least I will do, even in this sad situation of bugs in operating system core components. A break after like 5 or 10 retries should suffice.

Other ideas:

1. Have the Samba or Windows guys fix their code. This would be the best and from a global perspective cheapest solution for all parties involved. But is unlikely to happen soon, as this requires much more research, further most, find out contact information of developer who is respsonible or even go the long and dusty "official" road. Even then it may take years until the fix is rolled out on a large scale.

2. Use the same Windows Copy Routine like Explorer does. This may be the best solution from the user's point of view. Instead of an early failure (after a few retries), file copying will simply succeed. However replacing the core routine of a file sync tool should not be taken lightly and needs further extensive testing, reseach and performance tests.

3. Theoretically there's a hacky way to detect the problem before copying the file by checking for extended attributes. Drawback is one additional file access. And even if the situation is detected, it only helps reduce the number of retries to 1, i.e. helps to produce an early failure.

Sorry for my late response on this issue. Today I'm able to test the options you've mentioned.

I can't find the 'Volume Shadow Copy' you mentioned, so I tested the first 2 options: 'fail-safe file copy' and 'copy locked files'. Third option (copy NTFS file permissions) should not be affected in this issue.

Having 'fail-safe file copy' checked, FFS keeps looping (crash scenario). Only scenario it doesn't crash is to uncheck it.
Having 'copy locked files' checked or unchecked, FFS shows 'Error 80: File already exists. (CopyFileEx)'. After skipping both XML files the sync can be finished.

I think the easiest way to handle this issue is simply accept that a file cannot be copied due to lack of bad Windows-Samba integration. FFS can handle this scenario by implementing the retry limiter, when reached the limit: show the Error 80. Don't forget to remove the created temporary files FFS created at the destionation.
A rebuild of the core for this problem is something you don't want to do - only in urgent cases which this problem isn't.

Worth a short investigation: create a function which bypasses the issue. Read the file and write it to it's destination via FFS instaid of using Windows' CopyFileEx. Maybe modification time and NTFS rights are not correct, most important the data is being copied.

[Zenju]

Windows 10 update:
The Windows 10 Explorer uses SHFileOperation, which internally uses the CopyFile2 API. CopyFile2 suffers from the same Samba bug like CopyFileEx.
In other words, Windows 10 Explorer triggers this Samba bug discussed here just like FreeFileSync.

[Mobius]

Hi Zenju,

Resurrecting an old thread because this problem has just surfaced for me. On Windows 10, using FreeFileSync 8.3 donation edition, and suddenly getting hundreds of these temp files, named filename0.ffs_tmp through filename9.ffs_tmp, and then the same sequence for the next file, and so on. Upgraded to FreeFileSync 10.3 donation edition, but still having the same issue, though now the temp files have a slightly different naming structure.

On examining the problematic files, both in the source and the destination (destination is on a Synology NAS connected via SMB,) I can detect no extended attributes, so that doesn't seem to be the issue.

Have been on Windows 10 for a long time without this issue showing up. The only recent changes I can remember are 1) recently updated the DSM on the NAS (but don't think this is the issue; the underlying Linux is the same,) and 2) recently finally turned off SMB version 1, so SMB version 2 is the minimum used by both Windows and NAS. All other NAS operations continue as before, without issue.

Does that provide any helpful clues? Are you still working on this issue?

Thanks for a terrific program!

EDIT: Just wanted to add that the ffs_tmp files are zero byte files, but that they are not being cleaned up; they are left behind.

[guemi]

Hi Zenju,

Resurrecting an old thread because this problem has just surfaced for me. On Windows 10, using FreeFileSync 8.3 donation edition, and suddenly getting hundreds of these temp files, named filename0.ffs_tmp through filename9.ffs_tmp, and then the same sequence for the next file, and so on. Upgraded to FreeFileSync 10.3 donation edition, but still having the same issue, though now the temp files have a slightly different naming structure.

On examining the problematic files, both in the source and the destination (destination is on a Synology NAS connected via SMB,) I can detect no extended attributes, so that doesn't seem to be the issue.

Have been on Windows 10 for a long time without this issue showing up. The only recent changes I can remember are 1) recently updated the DSM on the NAS (but don't think this is the issue; the underlying Linux is the same,) and 2) recently finally turned off SMB version 1, so SMB version 2 is the minimum used by both Windows and NAS. All other NAS operations continue as before, without issue.

Does that provide any helpful clues? Are you still working on this issue?

Thanks for a terrific program!

EDIT: Just wanted to add that the ffs_tmp files are zero byte files, but that they are not being cleaned up; they are left behind.

This exact behaviour has started happening here. Using RealTimeSync on a windows server machine moving files from a local folder to a samba share.
¨

[guemi]

This is most likely due to the new Samba security update launched a few days ago.

Zenju: Any ideas if this is something you can solve with a patch?

[Mobius]

For me at least, the problematic temp files appear only for certain source files in certain batch jobs, although every file synchronized must create a temp file. Perhaps the Fail-Safe Copy option could be made a per job or per folder-pair option, so that it could be turned off for the problematic jobs, but left on for the other jobs? This would be a workaround until the issue is solved.

[Mobius]

Hmmm... 10.4 just released, but no mention of a fix for these problematic temp files. Zenju, is this something that is going to be addressed?

[guemi]

I second this. Error remains here.

[Zenju]

I don't think these new Samba issues are related to STATUS_EAS_NOT_SUPPORTED, so I'd appreciate if you'd open a new forum topic. If it included steps how to reproduce or a Process Monitor trace file, this would be even better.

[Mobius]

I don't think these new Samba issues are related to STATUS_EAS_NOT_SUPPORTED, so I'd appreciate if you'd open a new forum topic. If it included steps how to reproduce or a Process Monitor trace file, this would be even better.

Will do. My NAS is currently in the middle of a rebuild, but hopefully it will be finished by tomorrow and can do so.

[Mobius]

Well, this is very odd. Suddenly, the problem seems to have disappeared. I spent some time trying to reproduce it, but without any success. And, the ffs_tmp files that were formerly present have mysteriously disappeared as well. So, I re-enabled the job which had been producing these files, which had been disabled since 2018-07-18, and ran it. No problems. I still have the old log files, and for what it's worth, the problems always occurred on Microsoft files; but I think I mentioned earlier that I could find no special attributes on those files.

During that time period, I believe there was one minor Windows update--I have no idea whether that could be responsible. But in testing this, I notice that now, under Turn Windows Features On or Off, the SMB 1.0 entry which had been turned off, rather than a single entry, is now a tree containing three entries: Automatic Removal, Client, and Server, and only the Client is turned off; the other two are turned on. Anyone have any thoughts on that?

[tvprod]

This seems to be a problem similar to that discussed above.
I am running Windows 11 and backing up a WD NAS to a WD external drive. Each time I run file sync, .tmp files are being created with the same name as the new files to be copied onto the backup drive and not copying the files to be backed up. When I run file sync the next time when the .tmp files are there it gives me an error and won't sync because it thinks the file is already there, but it's a 0kb .tmp file:
ERROR_FILE_EXISTS: The file exists. [CopyFileEx].
I can manually delete the .tmp files but they are created again the next time file sync is run. Any thoughts?

[Zenju]

This seems to be a problem similar to that discussed above.
I am running Windows 11 and backing up a WD NAS to a WD external drive. Each time I run file sync, .tmp files are being created with the same name as the new files to be copied onto the backup drive and not copying the files to be backed up. When I run file sync the next time when the .tmp files are there it gives me an error and won't sync because it thinks the file is already there, but it's a 0kb .tmp file:
ERROR_FILE_EXISTS: The file exists. [CopyFileEx].
I can manually delete the .tmp files but they are created again the next time file sync is run. Any thoughts?

viewtopic.php?t=7898&start=30#p32404

[christiansenk]

I have this error, but whilst the error is *.ffs_tmp there are no ffs_tmp files present.. no matter where or how hard I look these files don't exist - but I can't sync the files across.
the erro message is
"D:\IMG_3286~b0d5.ffs_tmp" to
"D:\IMG_3286.jpg".
ERROR_ALREADY_EXISTS: Cannot create a file when that file already exists. [MoveFileEx]

but the file does not exist to delete.

[Zenju]

I have this error, but whilst the error is *.ffs_tmp there are no ffs_tmp files present.. no matter where or how hard I look these files don't exist - but I can't sync the files across.
the erro message is
"D:\IMG_3286~b0d5.ffs_tmp" to
"D:\IMG_3286.jpg".
ERROR_ALREADY_EXISTS: Cannot create a file when that file already exists. [MoveFileEx]

but the file does not exist to delete.

Could be a corrupted disk, fixable via Windows error checking: viewtopic.php?t=9776#p36079

[therube]

@Zenju

So your thought here is that it is a corrupted disk structure (which theoretically a CHKDSK could rectify) rather then a failing disk?

[tvprod]

When I did the last update to version 11.27 around 10/17/22 that seemed to fix this issue. I've been dealing with it for most of 2022 and it seems to work again.