Feature Request: Atomic Mirror

[cmicat]

The Atomic in title is the same Atomic as Database "ACID".

1 Backgroud

I use FreeFileSync to mirror my internal hard drive's file to external hard drive. And I find out if I set "Delete files" option to "Permanent", when FreeFileSync is mirroring file, it will first delete target's orphan file, then copy source file into target. For Data Safety, I need the data copies >= 2 at any time (Or why should I do backup at all, right?). Due to this behavior, when FreeFileSync is syncing file, the target folder's data will not be complete. If the source drive is broken during the process, I will not be able to recover even old version from the mirror.
I have a 4T external hard drive for backup, sometimes I modify or move a large amount of files inside internal drive, so it spends hours to copy all the modified data. It worries me that in this few hours, my mirrored backup is simply incomplete.
Yes, I can use Versioning to keep all the old versions. But I use windows file history to do versioned backup, and use FreeFileSync to do mirror backup. I hope the mirrored version simply be 1.

2 What Behavior Do I Expect

In the "Delete files" Options, add an additional "Safe Atomic Permanent" option. When selected this option, all deleted and modifed files will be moved to a seperate versioned folder, after and only after one successful mirror, the versioned folder will be deleted.

3 My Homebrew Solution - Overview

Since Now FreeFileSync do not have the above option, I use custom config & commandline to do the same thing. I think this might help others, so I decided to post it here.

Tested FreeFileSync version: 11.17
Tested Operating System: Windows 10
Test Setup: Copy file from internal hard drive to external hard drive. I haven't tested any other connection method like sftp.
Test Result: Both modified and deleted file will be moved to the version folder, so old version will be safe while mirroring.
Test Result: Both modified and deleted file will be moved to the version folder during mirroring, after and only after a successful mirror, old version will be automatically deleted.

4 My Homebrew Solution - Config FreeFileSync

1.png (118.8 KiB) Viewed 1222 times

(1) source & target should use volume label

source directory: `[TestData]\source\atomic mirror`
target directory: `[TestData]\target\atomic mirror`

2.png (148.01 KiB) Viewed 1222 times

(2) Move files to a user-defined folder

[TestData]\Revisions-freefilesync\atomic mirror


(3) Naming convention
Use "Time stamp [Folder]". Every failed sync's file will be located in a seperate directory, so there will be no confliction, so old version file will be safe.

(4) Limit file versions
Do not limit it! On successive failures, you will not want your original version of mirrored file being deleted. On successful sync, the command will delete all old versions.

(5) Run a command - On Success
Make sure select "On success:". Do not use on completion, completion include success and failure, you'll only want to delete old version on success, rather than some failures like disk failure or anything break your sync process.

(6) Run a command - the command
The command will remove the version directory only after a successful sync.

(6.1) internal drive only
You can use fixed drive letter

rmdir /s /q "E:\Revisions-freefilesync\atomic mirror"


(6.2) removable storage
Windows May change the drive letter when you reconnect the removable storage, so you cannot use drive letter.

I tried use volume label, it's not working.

rmdir /s /q "[TestData]\Revisions-freefilesync\atomic mirror"


So you need to use volume GUID

1. First find you drive's GUID. You can get volume GUID and driver letter mapping info by execute following command in cmd (do not require admin rights)

mountvol.exe


2. Then the command in FreeFileSync will look like this.

rmdir /s /q "\\?\Volume{576f2736-1404-46b6-b46e-cf34cc7260f0}\Revisions-freefilesync\atomic mirror"


[cmicat]

Following content is something I discovering while trying to get atomic mirror. I hope this will be helpful to someone else.

1 Implementation Considerations

(1) filesystem features

At first, I thought about filesystem features. But this only works with directly attached storage, rather than sftp.

(2) log

The second best solution I thought will be use append only file as a log. Like redis' append-only file. You can read this file and know what write request already happened, and commit or rollback the whole transaction. But I believe sftp ftp have bad append-only file support.

So I believe if user is uploading local computer's data to a NAS provided sftp server, the second solution will not work.

(3) version folder & auto delete

Like my homebrew solution, but integrated into the software, and becomes an option.

This solution will require user recover files by hand after source disk died and mirror is interrupted. But hard to recover data is better than no data at all.

2 Filesystem features can be used to support snapshot

2.1 windows filesystem can be used on removable media

Mirosoft officially do not support use ReFS on removable media

If you try to use it anyway, you may get a RAW partition after some windows OS update:

KB5010691: ReFS-formatted removable media may fail to mount or mounts as RAW after installing the January 11, 2022 Windows updates

Considering other windows supported filesystem's functionality is much more weaker than NTFS and ReFS. The only viable filesystem in removable media is NTFS.

2.2 NTFS features related to snapshot

(1) ✗ File-Level snapshots

NTFS do not support this. Only ReFS can use.

(2) Volume snapshots (Volume Shadow Copy)

I think this is full volume (partition) level snapshot, not single file or directory, so cannot rollback single directory back. But can take a snapshot first, and copy the snapshot back to main filesystem to do rollback?

Shadow Copy
Volume Shadow Copy Service
Volume Shadow Copy Service
Volume Shadow Copy Service Overview

(3) Transactions

The lock will be too big.

3 Other Tested Software

🤔 Following software can change config to partialliy support atomic mirror:

1. FreeFileSync 11.17
see the post's main article

2. Syncovery 9.46k build 358
Uncheck "Profile Settings" - "Advanced Settings" - "Files" - "Deletions" - "Delete Files Before Copying"
But all partially synced data will be mixed together, make recover very hard.

✗ All following software cannot handle atomic mirror, and cannot make a homebrew solution:
1. AllwaySync 21.1.5
2. Microsoft SyncToy 2.1
3. DirSync Pro 1.54b1
4. FileGee

[cmicat]

And thank FreeFileSync's authors for the great software. Use FreeFileSync & the custom config is the closest thing I can get as atomic mirror.

[xCSxXenon]

So you disabled the existing ways to safely backup data and want another way implemented?
FFS, by default, does not delete the existing copy until the new copy is fully transferred with the .ffs_tmp extension appended. Once fully transferred, it deletes the existing copy in the destination and renames the new one with the correct extension. FFS also has the option to version, which you've disabled. You could also just turn off permanent deletions. You did a lot of work to solve the problem that not only you caused but you are the only one to have a 'need' for this solution.

[cmicat]

So you disabled the existing ways to safely backup data and want another way implemented?
FFS, by default, does not delete the existing copy until the new copy is fully transferred with the .ffs_tmp extension appended. Once fully transferred, it deletes the existing copy in the destination and renames the new one with the correct extension.

No, "Fail-safe file copy" do not satify my usage requirement.

"Fail-safe file copy" will:
1. Copy source file into target location with ".ffs_tmp" suffix
2. Delete the target's original file
3. Remove ".ffs_tmp" suffix.

But "Fail-safe file copy" only work with modified file scenarior, it do not work with delete file scenarios.

For example:
1. I may have a lot of game recordings, with over 1024GiB size.
For example, I use Nvidia Geforce Experience to record my whole game play with 4k resolution, the target bitrate is set to 50Mbps. With 50 hours game recording, it will reach 1,047 GiB.
2. I mirror it to a backup drive.
3. I cut the game recordings with sections I need. And delete all the 1024GiB uncut videos. The cutted video size is 200GiB.
4. I mirror it again to the backup drive.
5. FreeFileSync will delete the 1024GiB uncut videos in my backup drive first, then it will copy the new cutted videos into the backup drive.
In this time window, I will have only one copy of video files(uncut or cut), if the source drive, all data will be lost.
Assume the transfer rate is 150MiB/s, it will take:
200GiB * 1024 / 150MiB/s = 22.76 minutes.

This is just an example, sometimes I will just remux all the video files to change the mp4 web fast start option, and I do not keep original video files after successful remux. So it's very easy for me to generate hundreds GiB of data need to mirror.

FFS also has the option to version, which you've disabled. You could also just turn off permanent deletions.

Versioning will keep the old files, I want the old files be deleted after and only after one successful mirror. It will waste a lot of disk space if I keep the original video file after a successful remux or cut.

I use windows file history to keep versioned backup for my documents. But for video files, I only need mirror, so I use FreeFileSync. Maybe I should investigate some backup tool to do 1 version only backup, instead relying on synchronization tool? All the synchronization tools I tested do not have the desired behavior without a lot customization. Do you have any recommendations?

You did a lot of work to solve the problem that not only you caused but you are the only one to have a 'need' for this solution.

Well, it's actually very fast to find the correct "rmdir" command for FreeFileSync. Testing the other sync tools takes a lot of time.

[cmicat]

Ok, I just tested FileGee, if i set "incremental backup" to automic delete old version, it will delete old copy even after failed backup.