This is a theoretical question: Suppose the FFS process was unexpectedly
terminated while it was updating the database on either side. What could
happen? Is it possible that the database could somehow get corrupted, or at
least inconsistent? If so, how would FFS react on the next synchronization?
Could you elaborate a little bit on this topic please?
Database and data integrity
- Posts: 4
- Joined: 17 Sep 2010
-
- Site Admin
- Posts: 7212
- Joined: 9 Dec 2007
When it comes to data consistency FFS behaves (almost) completely
transactional. Here's an overview about the most important design
considerations:
1. Copying a new file: All binary data is copied into target directory as a *.ffs_tmp file. File attributes, file times and optionally file permissions are applied. At the end the file is transactionally renamed to its actual name.
=> Behavior on unexpected termination: A *.ffs_tmp file is left behind. As
this file ending is reserved for FFS, the next synchronization will
automatically mark these files for deletion by default, no matter what other
sync-settings are active.
2. Deleting a file (permanently): A transactional operation as provided by the operating system.
3. Moving file to recycle bin: Individual files/folders are not directly moved to the recycle bin, but to a temporary folder. This folder is then moved to recycler at the end of processing of current folder pair.
=> Behavior on unexpected termination: The temporary folder is left behind in
base sync-directories and needs to be cleaned up by the user. No risk of data
corruption.
4. Moving file to custom deletion directory: I) Custom directory is on same drive as to be deleted file/folder: Transactional behavior due to ordinary rename operations. II) Custom directory is on another volume: Moving is emulated via "copy + delete": Copy is transactional due to use of *.ffs_tmp files as explained above. Delete is naturally transactional. The combination, of course is not. However the time window after copy succeeded and before deletion is complete is very narrow.
=> Behavior on unexpected termination: 1. *.ffs.tmp file is left in custom
deletion directory or 2. (very unlikely) file exists twice => no issue
concerning data loss
5. Overwriting a file: Overwriting is modelled as "delete + copy". Copy is transactional via *.ffs_tmp mechanism. However the operation starts with a deletion in order to minimize peak hard disk space consumption.
=> Behavior on unexpected termination: I. file to be overwritten is deleted!
Note handling of <automatic> mode: Modification of source initially had lead
to overwriting target. Target is deleted and program terminated. Next sync
will detect that both sides have changed. Normally a conflict. In this case
however, this conflict is automatically resolved as one side is empty => File
is copied again in the same direction next time.
II. *.ffs_tmp file left behind -> cleaned up on next sync.
6. Handling during synchronization via <automatic> mode:
=> Behavior on unexpected termination: Database file is written at the end of
synchronization. Therefore all changes until termination are interpreted as if
they were manually executed by the user. As this is something that happens in
general, there is no conceptional problem or risk of any kind.
7. Writing the database file:
=> Behavior on unexpected termination: Similar to file copying the database
file is first completely written to *.tmp files on BOTH sides. These changes
are then committed by deleting old files and renaming new db-files on both
sides sequentially. In general the time window of deleting old files and
renaming new ones is very narrow. Additionally changes of delete or rename
operations failing are very low, as the *.tmp files could be created
successfully proving full file write access permissions.
In the very unlikely event of a crash at least one of the db-files is missing
(note that first both files are deleted, then both *.tmp files are renamed to
actual db-files). With one db-file missing the tool will behave just like on
first synchronization of this folder pair. Ergo: No conceptional issue.
So to sum up: FFS leaves no gaps open for data loss or data inconsistency at
any step in the synchronization process.
-Zenju
transactional. Here's an overview about the most important design
considerations:
1. Copying a new file: All binary data is copied into target directory as a *.ffs_tmp file. File attributes, file times and optionally file permissions are applied. At the end the file is transactionally renamed to its actual name.
=> Behavior on unexpected termination: A *.ffs_tmp file is left behind. As
this file ending is reserved for FFS, the next synchronization will
automatically mark these files for deletion by default, no matter what other
sync-settings are active.
2. Deleting a file (permanently): A transactional operation as provided by the operating system.
3. Moving file to recycle bin: Individual files/folders are not directly moved to the recycle bin, but to a temporary folder. This folder is then moved to recycler at the end of processing of current folder pair.
=> Behavior on unexpected termination: The temporary folder is left behind in
base sync-directories and needs to be cleaned up by the user. No risk of data
corruption.
4. Moving file to custom deletion directory: I) Custom directory is on same drive as to be deleted file/folder: Transactional behavior due to ordinary rename operations. II) Custom directory is on another volume: Moving is emulated via "copy + delete": Copy is transactional due to use of *.ffs_tmp files as explained above. Delete is naturally transactional. The combination, of course is not. However the time window after copy succeeded and before deletion is complete is very narrow.
=> Behavior on unexpected termination: 1. *.ffs.tmp file is left in custom
deletion directory or 2. (very unlikely) file exists twice => no issue
concerning data loss
5. Overwriting a file: Overwriting is modelled as "delete + copy". Copy is transactional via *.ffs_tmp mechanism. However the operation starts with a deletion in order to minimize peak hard disk space consumption.
=> Behavior on unexpected termination: I. file to be overwritten is deleted!
Note handling of <automatic> mode: Modification of source initially had lead
to overwriting target. Target is deleted and program terminated. Next sync
will detect that both sides have changed. Normally a conflict. In this case
however, this conflict is automatically resolved as one side is empty => File
is copied again in the same direction next time.
II. *.ffs_tmp file left behind -> cleaned up on next sync.
6. Handling during synchronization via <automatic> mode:
=> Behavior on unexpected termination: Database file is written at the end of
synchronization. Therefore all changes until termination are interpreted as if
they were manually executed by the user. As this is something that happens in
general, there is no conceptional problem or risk of any kind.
7. Writing the database file:
=> Behavior on unexpected termination: Similar to file copying the database
file is first completely written to *.tmp files on BOTH sides. These changes
are then committed by deleting old files and renaming new db-files on both
sides sequentially. In general the time window of deleting old files and
renaming new ones is very narrow. Additionally changes of delete or rename
operations failing are very low, as the *.tmp files could be created
successfully proving full file write access permissions.
In the very unlikely event of a crash at least one of the db-files is missing
(note that first both files are deleted, then both *.tmp files are renamed to
actual db-files). With one db-file missing the tool will behave just like on
first synchronization of this folder pair. Ergo: No conceptional issue.
So to sum up: FFS leaves no gaps open for data loss or data inconsistency at
any step in the synchronization process.
-Zenju
-
- Site Admin
- Posts: 7212
- Joined: 9 Dec 2007
PS: "unexpected termination" is really the worst case scenario. In general
e.g. when user aborts operations the credo is "no file is left behind", that
is the tools cleans up completely.
e.g. when user aborts operations the credo is "no file is left behind", that
is the tools cleans up completely.
- Posts: 4
- Joined: 17 Sep 2010
Thank you for a quick and very informative answer!
-
- Site Admin
- Posts: 7212
- Joined: 9 Dec 2007
Item 5 will be refined in v3.15 to provide "full" transactional behavior:
viewtopic.php?t=1957
viewtopic.php?t=1957
