Why FFS Syncs With Priority To File Size?!

[Archivist]

Hey,

Since the latest update I have noticed of a few changes that became big problem to my workflow using FFS:

1. When syncing in "Update", FFS first opens ALL FOLDERS that were supposed to be synced, and then populates them with the files. Is there a way to cancel this way so FFS will go one-step at a time to each folder?

2. When syncing in "Update", Let's say there is no way not to open all folders, but then, I noticed FFS prioritize the order in which folder content is copied based on file size.
For example, folder1 has 2 files, one small, one big;
folder2 has 2 files, both bigger than folder1;
FFS syncs first the content in folder2, which is bigger than folder1 and then goes to folder1.
How come?
Syncing in the specific folder order and content is crucial.

Maybe going back to an older version will do the work?

Thanks.

[Archivist]

OK, as I was writing this post, I remembered a couple of weeks ago I have changed the settings in the "Comparison" tab, that the variant will be based on "File Size", I wanted to speed up the sync process time. I believe this gave priority to the file size in all folders, so that's why my syncs where not working well.

Still, I wonder why FFS don't sync in the folder order, on all occasions and variants, and why he has to open "empty folder" for everything that I wanna sync before it is synced?

[Archivist]

I left my FFS over night to copy files, and once again, even after changing the "Comparison" variant to "File time and Size" it copied my in order of Size and not folder order/content. So it wasn't this setting that caused the problem, please help.

[Archivist]

I just removed the current version of FFS and installed 9.6, indeed this problem is solved.

If there is a way to change or disable the option for FFS so it won't sync first all folders and copy based on file size, that would be great. I almost gave up on this software.

[Zenju]

FFS applies an order of sync directions that minimizes peak disk
space consumption. Folder pairs are synchronized one after another and each pair is handled in three steps:
1. process file move/rename operations
2. delete files and overwrite large files by smaller files
3. sync rest

PS: This sequence has been in place since FFS version 1.2.

[Archivist]

Hey, Thanks for replying.
I did some more tests and I traced this change/problem to version 10.0.

I might have not explained the problem correctly:
- I have bunch of folders: A,B,C...Z
- I use FFS to "update" an empty folder (for example)
- FFS first creates all folders from A to Z
- FFS populates all folders with the first file in each structure level, like A/A.txt,B/B.txt... Z/Z.txt
- FFS moves on to next folder structures like A/TEST/STUFF... B/TEST/STUFF... Z/TEST/TEST/STUFF

This creates some sort of multi-layered/structured syncing that I don't understand what its needed for. I have used the software for over a year and recently upgraded an older machine from 9.4 to 10.4, so that what made the sudden change.

[Zenju]

I see what you mean now. The folder hierarchy processing order used to be "depth first" but has been changed to "breadth first". By first processing all items at a folder level, e.g. creating all subfolders and copying files before recursing further, FTP syncs run faster because it avoids lots of needless CWD (change working directory) calls. In all other cases the new order shouldn't matter. What is the problem with the new processing in your case?

[Paul]

Archivist,

I don't understand (practically) what the issue is.

Can you please explain why the ability of FFS to "go one-step at a time to each folder" matters to you?

Why does it matter how FFS does the file sync (i.e., "depth first" vs. "breadth first")?

Asked another way, you wrote: "I left my FFS over night to copy files, and once again, even after changing the "Comparison" variant to "File time and Size" it copied my in order of Size and not folder order/content."

In the morning, assuming the transfer was completed, how could you tell the difference?

I'm not arguing or challenging you. I'm confused.

[Archivist]

It matters because I am using FFS with an LTO tape system so basically every 5tb it switches unto the next tape. For my workflow I must keep all associated files of a folder within the same tape and in order. What FFS did (what I call "problem") is that it started splitting a lot of my folders into 2 different tapes, when it not used to do that prior to version 10.0.

Example for this behavior:
I want to sync ("update") folders A,B,C,D,E,F,G,H which together has 8tb of info, with numerous files and sizes, so before it used to work that folders A,B,C,D,E will be synced on tape 1; and folders E (the files that are left),F,G,H will be synced on tape 2;
What FFS is doing now is it starts populating files from all folders depend on file size or folder level structure, so now I have on tape 1 all folders A,B,C,D,E,F,G,H and on tape 2 folders A,B,C,D,E,F,G,H with a mix of files in between.
What this means for my workflow, is that if I want to sync back my entire A folder, I must do that from 2 different tapes instead of 1 tape, which of course takes longer to do in an LTO tape system.

[root]

Any plans to fix this issue?

[xCSxXenon]

Doesn't sound like an issue, just doesn't work the way your scenario best needs. For a huge majority, it doesn't matter, and actually improves it for more people than going back to depth-first. Going back is not the best option, obviously, the only way forward I can see is adding yet another option. It would probably be best to do it per config and not globally though.

[root]

Adding it as an option would be great indeed!

[Archivist]

I would appreciate this option between "depth" and "breadth" synching. I still haven't upgraded from version 9.9 and recommending not to do so to my colleagues because we all use "depth" in our workflow.