Hi guys,
This is my first post so hope I am doing this correctly.
I am hoping that what I need to utilise here is the Versioning functionality but firstly here is what I am trying to achieve....
I have a 6TB NAS drive with lots of artwork files that I wish to backup to an identical 6TB NAS drive. I am wanting the backup to be a one way mirror copy so to speak so that in the event of a complete failure of the primary NAS the backup could simply replace it as the daily artwork drive.
Now what I am concerned about is that if a file was to corrupt(an this is an event that has happened as opposed to a hypothetical) due to a hard drive error or some other means how can I have a method that would not mirror this corrupt file thus overwriting what was a good backup with a corrupt file leaving me no good copy?
At the same time I am wanting to keep the folder structure the same as the original NAS. My though was to do a daily 1 way mirror copy as I have been doing but also somewhat run a Versioning folder such that any changes to existing files (ie new corruption) would not only be overwritten but also stored in a separate Versioning folder that hopefully would never be needed but there in case.
So in summary:
DRIVE A - Mirrored to DRIVE B on a daily scheduled task
but also
Any changes to specific files (as opposed to just new files and folders) to DRIVE A also stored in an additional folder for example DRIVE A/Versioning - perhaps with a time date stamp?
I belive this is possible from reading othere threads but not quite sure how to implement both. I am assuming the Versioning would need to be done first in the scheduled batch of changes to work?
Many thanks in advance.
Nate
mirror sync with corruption protection
- Posts: 2
- Joined: 23 Mar 2017
-
- Posts: 2523
- Joined: 22 Aug 2012
In general, for NAS1 to NAS2 syncs it would probably be faster to use the rsync capability available in most NAS's,
as it involves a direct NAS-to-NAS connection/transfer.
FFS needs to run on a PC or laptop, so the data goes NAS1-to-PC and then PC-to-NAS2.
However, rsync does not support versioning.
Versioning in FFS does not need to be done "first", but is done "during" the FFS sync process:
In case of a Mirror-sync, if a file pre-exists in the Target location and would be overwritten or deleted by the sync, FFS will move/copy said pre-existing file to the Versioning location before overwriting or deleting it in the Target location.
The exact behavior in the Versioning location depends on the versioning method being used.
As you intend to sync the entire A-drive to the B-drive, and your Versioning location will be part of your A-drive,
make sure to exclude the Versioning location from the sync via the exclude filter.
as it involves a direct NAS-to-NAS connection/transfer.
FFS needs to run on a PC or laptop, so the data goes NAS1-to-PC and then PC-to-NAS2.
However, rsync does not support versioning.
Versioning in FFS does not need to be done "first", but is done "during" the FFS sync process:
In case of a Mirror-sync, if a file pre-exists in the Target location and would be overwritten or deleted by the sync, FFS will move/copy said pre-existing file to the Versioning location before overwriting or deleting it in the Target location.
The exact behavior in the Versioning location depends on the versioning method being used.
As you intend to sync the entire A-drive to the B-drive, and your Versioning location will be part of your A-drive,
make sure to exclude the Versioning location from the sync via the exclude filter.
- Posts: 2
- Joined: 23 Mar 2017
Hi Plerry,
Many thanks for the reply.
I have tested this and got it working and also ensured the Versioning folder was excluded (although the Versioning is actually being stored on NAS2 - maybe I should have this stored on both as well?)
I also tested one of the corrupt files I had (a jpeg - one fine version and one corrupt via HD CRC error that on the surface appeared to be identical on a time and size compare). The Versioning worked and I also added the tick box to DETECT MOVED FILES as I figured there is even less chance of corruption migrating to NAS 2 if the file is not copied.
One thing I realised after getting this all in place was that if the backup itself suffered from a HD failure (NAS 2) of some kind and was corrupting files over time I would not ever really know and this is more of an archive backup. So if it was ever needed (i.e NAS 1 died) I could come unstuck so I was trying to work out a method to do some form of monthly scan or hash or something to verify the integrity of the files - any ideas how I could go about this?
Also thanks for the advice about maybe using the NASbackup system - the issue is that not only is Versioning not supported but the incremental backup would keep duplicating folder every time the name changed even if the content didn't which is something that will happen repeatedly.
Many thanks Nate
Many thanks for the reply.
I have tested this and got it working and also ensured the Versioning folder was excluded (although the Versioning is actually being stored on NAS2 - maybe I should have this stored on both as well?)
I also tested one of the corrupt files I had (a jpeg - one fine version and one corrupt via HD CRC error that on the surface appeared to be identical on a time and size compare). The Versioning worked and I also added the tick box to DETECT MOVED FILES as I figured there is even less chance of corruption migrating to NAS 2 if the file is not copied.
One thing I realised after getting this all in place was that if the backup itself suffered from a HD failure (NAS 2) of some kind and was corrupting files over time I would not ever really know and this is more of an archive backup. So if it was ever needed (i.e NAS 1 died) I could come unstuck so I was trying to work out a method to do some form of monthly scan or hash or something to verify the integrity of the files - any ideas how I could go about this?
Also thanks for the advice about maybe using the NASbackup system - the issue is that not only is Versioning not supported but the incremental backup would keep duplicating folder every time the name changed even if the content didn't which is something that will happen repeatedly.
Many thanks Nate
