Hello again - and no, I do not believe I have asked this question before (grin).
We have the feature for transactional file copies enabled per the recommendation (<TransactionalFileCopy>true</TransactionalFileCopy>), which means that the file copy will create a file with the .ffs_tmp extension and then once the file copy is complete, the file is renamed.
The problem we are seeing is a lot of dangling or leftover .ffs_tmp files, and we cannot figure out why that might be happening. We are running v5.12, we are using the automatic variant for all our data directories, and we are relying on the transactional copy feature for safety's sake.
What might cause FFS to leave a .ffs_tmp file behind?
Is there some cleanup routine that is supposed to run once the sync is done, or during the sync, to delete these files, much like the directory deletion feature you added in 5.12?
If FFS sees .ffs_tmp files in the target directory, does it just start the file copy over or does it try to delete the .ffs_tmp file first? What if the copy failed the first time, but then the file was not included in the sync the second time, I assume there is nothing that would know to clean up the .ffs_tmp files, but is that really true?
Is it also a safe assumption that if you have a file size filter applied, that the file would not be included in the copy from the outset, rather than somehow copying up to the filter size and then exiting before it is done (because the file size limit has been exceeded)? We have one directory where we see this a lot on the server and it is the same directory where we have a 1 Gb limit on the file size.
Just for context, the last time we ran a sweeper program to find these files and delete them (assuming that if they could be deleted, they were not being actively written to by FFS), we deleted over 10 Tb of files alone in one of our shares, so we are quite interested in getting a handle on what might be causing these orphaned .FFS_TMP files.
Thanks in advance for any help or insight!
Orphaned .FFS_TMP files in target directories
- Posts: 74
- Joined: 17 Mar 2008
-
- Site Admin
- Posts: 7052
- Joined: 9 Dec 2007
> this question
Hm, I count *five* questions :) ... but they're good ones about program reliability:
When fail-safe file copy is active, FFS will first try to find a temp name that is free. Usually this is the `<file name>.ffs_tmp` one, but if this is already taken it continues using more cryptic ones until it has one.
Then it starts copying the file. If file transfer is interrupted within the FFS application, e.g. the user clicked cancel or an error is reported by the Win32 API, FFS will cleanup automatically any .ffs_tmp files it created. This is also and perhaps even more importantly true if fail-safe file copy is inactive: FFS will not leave garbage data behind.
Therefore when you see a .ffs_tmp file this must be due to one of two reasons:
1. FFS did not get a chance to cleanup: E.g. The FFS process crashed (very unlikely ;) or the network connection dropped or the system was somehow shutdown in the middle (power loss) or the user just pulled out the USB stick while it's being written...
2. FFS failed at doing the cleanup: I imagine this could happen if an AV software has the file open causing a sharing violation when FFS tries to delete the file. If this case happens a lot, it needs to be investigated for example with Process Monitor why deletion fails.
In order to handle the two low-level problems above FFS has a second higher-level cleanup: .ffs_tmp files (and *directories*, if recycle bin is selected for "deletion handling") are automatically scheduled for deletion during the next sync. This is a special rule for .ffs_tmp files/directories only and irrespective of any other configurable sync settings.
> quite interested in getting a handle on what might be causing these orphaned .FFS_TMP files.
Could it be that you excluded .ffs_tmp files via the filter? This would thwart the high-level cleanup handling above and explain why they were not removed on subsequent syncs.
> file size filter applied [...] from the outset
The file size filter is applied directly after comparison, so FFS won't try to copy excluded files.
Hm, I count *five* questions :) ... but they're good ones about program reliability:
When fail-safe file copy is active, FFS will first try to find a temp name that is free. Usually this is the `<file name>.ffs_tmp` one, but if this is already taken it continues using more cryptic ones until it has one.
Then it starts copying the file. If file transfer is interrupted within the FFS application, e.g. the user clicked cancel or an error is reported by the Win32 API, FFS will cleanup automatically any .ffs_tmp files it created. This is also and perhaps even more importantly true if fail-safe file copy is inactive: FFS will not leave garbage data behind.
Therefore when you see a .ffs_tmp file this must be due to one of two reasons:
1. FFS did not get a chance to cleanup: E.g. The FFS process crashed (very unlikely ;) or the network connection dropped or the system was somehow shutdown in the middle (power loss) or the user just pulled out the USB stick while it's being written...
2. FFS failed at doing the cleanup: I imagine this could happen if an AV software has the file open causing a sharing violation when FFS tries to delete the file. If this case happens a lot, it needs to be investigated for example with Process Monitor why deletion fails.
In order to handle the two low-level problems above FFS has a second higher-level cleanup: .ffs_tmp files (and *directories*, if recycle bin is selected for "deletion handling") are automatically scheduled for deletion during the next sync. This is a special rule for .ffs_tmp files/directories only and irrespective of any other configurable sync settings.
> quite interested in getting a handle on what might be causing these orphaned .FFS_TMP files.
Could it be that you excluded .ffs_tmp files via the filter? This would thwart the high-level cleanup handling above and explain why they were not removed on subsequent syncs.
> file size filter applied [...] from the outset
The file size filter is applied directly after comparison, so FFS won't try to copy excluded files.
