1 to 1 copy with folder changes

Get help for specific problems
Posts: 23
Joined: 26 Apr 2020

ms308680

Hello,

I really did not know what to search for regarding this situation

Under the following situation of having a lot of structured data, for example, pictures, documents, and programs.

Pictures
Documents
Programs
Customers\Active
Customers\In-Active


Then under each of those folders are subfolders that are categorized to the best of your ability at the time. You make a 1 to 1 clone of the drive and put the backup on the shelf and use the active drive.

Over time you come up with better categorizations and organize things better, but while the same data is on the active drive, along with new data, since the folder structure has changed, syncing this again would result in duplicates and wasted space.

I know there are backup programs that can do what I am requesting, but I am looking for 2 things, 1 basically a cold spare but also a solution that I do not need to restore from since if the active drive starts to fail, I can just quickly backup the changes, change the clone to the active drive and create a new clone.

I am sorry if this seems rushed or unclear, I had to retype it from scratch when I tried to post it the first time it asked me to log in again and I lost my post. I normally copy my post to the clipboard and go to it, but I am a bit tired and I failed to do that this time.

Thank you very much for your time
Posts: 23
Joined: 26 Apr 2020

ms308680

Hello,

I thought of a "feature" that might make what I want to do possible...

Have a program "Pair" the drives, one as Active and the other as Clone

Have a service on the computer with the Active drive monitor any changes, deletes, moves, folder creation, folder rename, folder deletion, and such.

Then when you re-run the sync, with the paired drive, it can execute the script that is collected and move the files/folders accordingly, then sync the new changes.

does such a program exist?

Thank you
User avatar
Posts: 3549
Joined: 11 Jun 2019

xCSxXenon

That's exactly what the program does. You can have it mirror the source to the destination. If you have the backup connected at all times, you can use RealTimeSync to monitor for changes and sync when they occur
Posts: 23
Joined: 26 Apr 2020

ms308680

Then under each of those folders are subfolders that are categorized to the best of your ability at the time. You make a 1 to 1 clone of the drive and put the backup on the shelf and use the active drive.

Over time you come up with better categorizations and organize things better, but while the same data is on the active drive, along with new data, since the folder structure has changed, syncing this again would result in duplicates and wasted space. ms308680, 26 Apr 2020, 04:28
I quoted above where I was speaking about how it would be nice for this to work. I know how the software works I believe but I have not been able to think of a way to make it work for the way I am wanting to use it.

Thank you for your response. I do not believe I had enough information in this re-write I had to do.

I do not want an ONLINE sync, I want a 1 to 1 copy and the backup would go into a safe or safe deposit box and be updated "once in a while"

When I have tried to "Sync" by one way to update the 1 to 1 clone from the active drive, it puts files back in places that I previously sorted and the 1 to 1 clone, is now littered with duplicates everywhere.

From using this program, this relies heavily on a directory structure that does not change, and if it does, it relies on a modification to tell if something is newer or not. I have NOT seen this track folder moves and such like that for 1 to 1 sync that are not ONLINE syncs that are perpetual.


I hope this makes more sense now that you understand that I do not want to have the backup drive plugged in perpetually, only updated occasionally with the least possible stress to the 1 to 1 clone so I am not having to practically wipe it before I start the sync to make sure I have a 1 to 1 backup at X time.

I know there are backup programs that perform what I want, but I would like a solution that I do not have to restore from, pick it up, recover any data from the active drive, clone the backup to a new backup, and start the cycle anew.

I have been doing just what you are speaking of for years, but it is costing too much in replacement drives to keep spinning for perpetual changes. I am willing to lose certain data because it will actually be in a few other places at the same time before it is re-synced to the 1 to 1 clone. It is not like I am not backing up the active drive in some way, just not completely, so I am just trying to figure out a way to update the 1 to 1 clone that is not online and has no idea of the folder structure changes that have happened on the active drive to be mimicked to the cloned drive without having duplicate data. I have just spent the past 3 days (technically I am still organizing and combining even more) deleting over 500GB + of duplicate, or no longer useful, data and that is exactly what I want to avoid going forward.

If this feature exists, and I missed it, please help me.

I thank you for your time and help
Posts: 23
Joined: 26 Apr 2020

ms308680

I think I might have thought of a solution that will fit in my parameters, it is a bit more to set up than I was hoping, but it would do what I am asking...

It might take me a few weeks to a month to set up this and test it, but I will share my setup once I test it.

If anyone else has any other suggestions in the meantime, please feel free to share it still.

My idea is basically this:

Set up a local Nextcloud server (does not have to remain running all the time, for the purpose I will use it for)

Set up your normal active computer to do the first sync

Set up another account on the computer to Sync with the 1 to 1 clone drive, after the sync is done, you can disconnect the 1 to 1 clone drive and shut down the next cloud server until you want to sync the changes on the active drive again.

As I said, it is a little bit of work, but this setup would do what I want, and not cause excessive wear on drives but also it would give me a 3rd copy of my data.

Not my ideal solution, but definitely worth looking into. The only problem is that I do need to have 3x's the storage instead of only 2x's.

Storage for Active Drive
Storage for Clone Drive
Storage for Server Drive

I hope this helps everyone understand what I am trying to do better as well.

Thank you all for your help and I still appreciate any other thoughts on this matter.

Thank you
Posts: 290
Joined: 13 Apr 2017

Gianni1962

It seems that you have setup FFS to do a "Two way" sync, the default when creating a new sync job.

Go to Synchronization settings and select the "Mirror" variant, use the option [x] Detect moved files.
Done!.
This setup do exactly what you are looking for.

This way you will be "mirroring" you Active drive on the Clone drive:
moved files will be moved, deleted files will be deleted, changed files will be updated.
Posts: 23
Joined: 26 Apr 2020

ms308680

I will test this. I was actually using one way since I completely missed that setting.

Thank you so much! This will be a much nicer solution :)
Posts: 23
Joined: 26 Apr 2020

ms308680

Exactly what I was looking for. I apologize to everyone here for being so ignorant. I could not think of what to call this feature so looking it up on google got me nowhere.

:) Thanks
Posts: 23
Joined: 26 Apr 2020

ms308680

Hello again, when I tested this on a small scale before I posted my last post above, it worked wonders.

My test was a source and destination on the same drive (on the desktop), just different folders, I created a half dozen folders and put random stuff inside them, other folders, shortcuts, and some small files.

But when I actually use this for the use case I described above, I noticed that it was not moving folders and files, but deleting and re-transferring such items. I read the section and I am using an NTFS partition(s), my setup for the practical is:

2 separate external drives that I sync once every few days. While it technically is working, it is taking longer to do the sync since any moved files are being deleted on the target drive then re-copied from the source.

Any idea why this did not work like the test I did?

As I said, the source and target are 2 external drives, not network shares, both are plugged into the same machine.

Thank you for your time
Posts: 290
Joined: 13 Apr 2017

Gianni1962

Did you also select the option "[x] Detect moved files" while selecting the Mirror variant?

Anyway this option can be honored only after the first full Mirror sync because it needs its database to operate and that db is created the first time and updated then.
Posts: 23
Joined: 26 Apr 2020

ms308680

Yes, I understand that, and that is why I said my test worked correctly but moving to the practical use, even with detect moved files checked, it does not seem to be using this feature and falling back to only changes based on a delete and update method.
Last edited by ms308680 on 10 May 2020, 20:07, edited 1 time in total.
Posts: 290
Joined: 13 Apr 2017

Gianni1962

Anyway that option can be honored only after the first full Mirror sync because it needs its database to operate and that db is created the first time and updated then.
Posts: 23
Joined: 26 Apr 2020

ms308680

I understand that, thank you. I have been playing with this for two weeks to see if I was doing something wrong before posting back here.

Thank you
Posts: 290
Joined: 13 Apr 2017

Gianni1962

"When" you have check-marked the option [x] Detect moved files?
Because the db is created only after you have selected that option.
Posts: 23
Joined: 26 Apr 2020

ms308680

Thank you, yes I know. After you pointed out this feature when you originally answered the post 2 weeks ago, I played with it briefly on 2 folders on the desktop and it worked exactly as I wanted, and I posted a reply

Exactly what I was looking for. I apologize to everyone here for being so ignorant. I could not think of what to call this feature so looking it up on google got me nowhere.

:) Thanks ms308680, 26 Apr 2020, 22:21
When I tried to move this feature into practice on the practical that I explained in the opening post, it does not seem to use this feature at all, even with it enabled. Yes beyond the 1st compare and sync.

Thank you
Posts: 23
Joined: 26 Apr 2020

ms308680

It seems that you have setup FFS to do a "Two way" sync, the default when creating a new sync job.

Go to Synchronization settings and select the "Mirror" variant, use the option [x] Detect moved files.
Done!.
This setup do exactly what you are looking for.

This way you will be "mirroring" you Active drive on the Clone drive:
moved files will be moved, deleted files will be deleted, changed files will be updated. Gianni1962, 26 Apr 2020, 21:57

After you pointed out this, I did a small test on the desktop with 2 folders and some shortcuts in folders and small programs in the folders as well.

That test worked perfectly, which is why I responded with both:
I will test this. I was actually using one way since I completely missed that setting.

Thank you so much! This will be a much nicer solution :) ms308680, 26 Apr 2020, 22:11
and
Exactly what I was looking for. I apologize to everyone here for being so ignorant. I could not think of what to call this feature so looking it up on google got me nowhere.

:) Thanks ms308680, 26 Apr 2020, 22:21
Well, I just re-ran that test, and it is behaving the same way that it is on the external drives. The only 2 differences are:

1) When I ran the 1st test it was with the installed version, I am currently using the portable edition after uninstalling the full installed edition.

2)I am using the donation edition now as well.

I will try installing the donation edition and do a few more tests.
Posts: 23
Joined: 26 Apr 2020

ms308680

OK, I have re-ran this test on the installed donation edition and the free installed edition.

All of them now show the same results. They delete the items to move them instead of moving them. I remember seeing a "move" icon instead of delete when I first tested this feature. I do not know what else has happened between here and there, but the test that I thanked the poster for, now no longer works. I purposely tried a small scale test before moving to a production type use. I will continue other tests, but I am close to out of ideas.

Thanks,
Posts: 290
Joined: 13 Apr 2017

Gianni1962

In your test you are simply moving files or you are also changing those files?

To tell that those files are copied and deleted and not simply moved, you are looking at the log file?

Did you know that ...

"Detection is not supported by all file systems. Most notably, certain file moves on the FAT file system cannot be detected. Also virtualized file systems, e.g. a mounted WebDAV drive, might not support move detection. In these cases FreeFileSync will automatically fall back to copy and delete."

Synchronization Settings / Detect Moved Files
Posts: 23
Joined: 26 Apr 2020

ms308680

Thank you,

I read that notice and as I stated, that I tested this when you first pointed out this feature to me. As the post above states, when I tested it on April 26th, I was happy with the result and everything worked just fine.

As to how I know it is deleting and re-copying. There was a different icon after the compare called "move" and now the only way it happens is via delete and re-copy.

I have laid out how I tested it, and how I was planning on using it in the original opening posts.

again below is the FIRST test I ran on April 26th, 2020:

How I initially tested this after you made me aware of the "detect moved files" feature:

I created 2 folders on the desktop, one source, and one destination

I created a lot of folders inside that and copied some files, shortcuts and such to those folders

I performed the first initial sync via mirror with "detect moved files" on, with no files currently in the destination folder.

after this first initial sync, I moved some files around, and I noticed on the different icon that implied "move" instead of delete and copy.

This showed that the feature was working as I was hoping and I posted as such to thank you.



That is the end of the test.




When I went to put this to practical use and tried moving things AFTER the first sync, it wanted to delete and re-copy.

Today I re-setup my test and it no longer works, even the same way I performed it above.

I have already made quite clear that I read the stipulations, and I was using the same file system as the test, the same computer as the test.

I also pointed out the only differences above as well. Between this test and the production use.

To address your other question, I have tried it both ways, with making changes to the files, and without making changes to the files. Same result


Thank you,
Posts: 290
Joined: 13 Apr 2017

Gianni1962

I'm using FFS 10.23 free and I'm testing a file move.

After Compare I can see the moved file with a + icon and a - icon, but if I hover the mouse pointer over those icons the tool-tip tells "Move file on the right".
Posts: 23
Joined: 26 Apr 2020

ms308680

I just set up a brand new computer to do the same test, and I am still having the same results. Using the same version FFS 10.23 free or donation or donation portable.

I saw that Icon that you see, so I know that it WAS working, but why it no longer works, even for the test I ran, I am a little aggravated. Not at you, but as to me re-doing this test and it not performing the same whatsoever across 2 computers.

I have tried running the program normally and with administrative privileges

I have tried 2 local folders aka source and target

I have also tried moving the target folder to a separate drive as well

I have hidden files turned off, so I have also tried deleting the database file to re-create it on subsequent tries but I have not been able to get this feature working again.

All of these tests are on 2 different laptops. 1 of the 2 is brand new in all aspects. (so not virtualized)



I have tried no spaces in the files/folders

everything I can think of, but to no avail as to why this feature stopped working for me.

The new computer has not gotten all of its windows updates, it is running 1903 while the one I ran the initial tests on is running 1909 (the version of Windows 10)

The detect moved files feature is not working even on the two way setting.

For all of the tests that I have said have been failing, I have been telling you the result with it on Mirror as you told me.

I do look forward to more information on this issue, but I am more stumped now than I have been since...

Me trying my test again on a new system was my troubleshooting. If I can think of anything else, I will post back, but for now, a feature (detect moved files) that I tested and that worked while testing back on April 26, 2020, does not work for me in production and no longer works for the same test.
Posts: 290
Joined: 13 Apr 2017

Gianni1962

You are speaking about a "Move" icon but there isn't such icon:
there are two icons, one with a green + and one with a red - and a tool-tip that tells "Move ...".
Posts: 23
Joined: 26 Apr 2020

ms308680

I do not have a screenshot of it because I did not think I would need it. The best I remember of the icon was a spiral arrow that specified the files/folder was moved and would move them accordingly instead of delete and copy
Posts: 290
Joined: 13 Apr 2017

Gianni1962

The moved file
The moved file
Annotazione 1.jpg (14.93 KiB) Viewed 3071 times
The tool-tip
The tool-tip
Annotazione 2.jpg (52.93 KiB) Viewed 3071 times
Posts: 23
Joined: 26 Apr 2020

ms308680

Yes, that is what I am getting now.

But when I ran my test initial test after you showed me the feature on April 26, 2020, there was a different icon that was being used when the (detect moved files) was working properly and did not delete and re-copy, it actually performed the move tasks which for thousands of files and for practical use, is faster and less wear on the drive as to deleting and re-copying files.

I wish I had a screenshot of what I saw and how my test went, no matter how much I have looked, I cannot find the icon I saw when I tested it.

It was a yellow icon and I believe it was a circular arrow that indicated that the files were moved and that they would be moved accordingly to the right(clone/backup/target)

Thanks for your help, but what you have shown above is what it does by default. and I do not want to MOVE files, I want a 1 to 1 copy. I want to detect file moves, and if I want to wear out the backup drive by constant deletes and re-copies I would not be here and would use it as it is.

As soon as you told me of that feature and I tested it on a small scale, I saw what it was capable, but could not get it to work in practice nor in any subsequent testing.

I am starting to understand why we have been not on the same page. My initial test of the feature showed me completely different results than what you are getting.

Again, I did not foresee any problems so I failed to take any screenshots of this working like I was intending for it to.
Posts: 290
Joined: 13 Apr 2017

Gianni1962

My move works, so probably your, try it!
You can then view in details whats done in the log file.

I doubt of a single icon showing a Move because the left and right view of the moved file cannot be on the same row due to the fact than now they are in different folders.
Posts: 23
Joined: 26 Apr 2020

ms308680

I am just trying to tell you how it worked for that first test. I have been trying it and it deletes and re-copies. This puts unneeded stress on the clone/backup/target drive since I did see the program function differently.

again, I don't want to MOVE, I want it to detect moved files and move them according to the source drive. (example, if you move a customer to inactive [on the source drive], but no data actually changes with that customer, you just moved them from active to inactive, then I want it [FreeFileSync] to send a move request [which FFS did do on my FIRST test and only my FIRST series of tests] for that folder to the target/clone/backup drive instead of deleting the data and re-writing it. Because that's what it is currently doing instead of tracking folder moves, it is deleting and re-copying.)

I saw it work, I know it works, I just don't know why it no longer works.

my, for instance, is my opening post.

I want to always have a 1 to 1 clone with minimal impact on the backup drive.

This program has the feature to detect moved files, but is currently only working as:


Mirror
Does it exist on the left, Does it exist on the right?
if they match do nothing
if left no longer exists, delete the right
if the right does not exist copy to the right



The program is behaving the exact same way right now whether or not I have "detect moved files" enabled or not
Posts: 23
Joined: 26 Apr 2020

ms308680

The other reason that I know that the files/folders are being deleted and re copied instead of moved with the "detect moved files" is folders that I am moving on the source drive, when I do a mirror sync with the feature turned on (or off) what is suppose to be moved ends up in the recycle bin, like the setting says it will do since I have it set to send deleted files to the recycle bin.

So this further proves my point that the detect file changes is not working properly and it is only detecting differences between the drives and deleting what is no longer on the left and re copying what is on the left but not on the right.

Even 1 step further, on the computer where I first had a successful run with it on April 26, 2020, I did a complete Windows reinstall of 1909 with all the updates, and I am still failing to replicate the test that succeeded on April 26, 2020

When I ran that first test, I was watching the recycle bin like a hawk and was making sure that when things were being "moved" via the database tracking instead of re-copied, it worked flawlessly and for the life of me I cannot replicate it.
User avatar
Posts: 2248
Joined: 22 Aug 2012

Plerry

Perhaps I overlooked it in one of the lenghty messages above, but ...
In order to allow FFS to detect moved files, both the old and new location of a moved file needs to be in the (scope of the) left- and right-base-location(s) or any of its subdirectories.
Posts: 5
Joined: 29 Mar 2020

s7xKsECrPLT7

I am in the same situation I think. One difference is that I have network bandwidth to deal with. No solutions have been found other than for me to figure out if my NAS is causing issues or not, which I have no idea how to do.

viewtopic.php?t=7167&p=24147#p24147