Plain-Jane Incremental Backups and "Update right item"

[kds-kds]

Hello,

I have read these threads:
viewtopic.php?t=4615&p=15426&hilit=update+right+item#p15426
viewtopic.php?t=1197&p=5239&hilit=update+right+item#p5239
but my question seems to be a new one.

I simply want to back up selected files from my internal drive to an external backup, and I want to have a baseline with subsequent incremental backups. (This is what WDBackup used to do and probably still does.) So I would choose 'Update >' in FFS.

Now, with reference to the attached screenshot, what does "Update right item" mean and if this is selected what will FFS do? If a 'right item' is found which is older than the matching 'left item,' will the 'right item' be preserved with a new version of that item created by copying the 'left item' (which should be in a corresponding new subtree) or will the 'right item' be overwritten?

Thanks,
kds-kds

[Plerry]

I don't know where FFS would refer to the quoted "Update right item".
The descriptions under the sync settings (F8) for the different Categories are pretty self-explanatory.
Simply position your mouse over the associated Action and FFS shows the description.

For Update, the Action for the Category "Left side is newer" is "overwrite right item".

What happens to the existing, older right-side item depends on the "Delete files" setting in the same sync settings window.
Depending on your choice there, it will be deleted, moved to the recycle-bin or versioned.
For versioning you can select different methods, see here.

[kds-kds]

Re "I don't know where FFS would refer to the quoted "Update right item"."
the answer is
"Simply position your mouse over the associated Action and FFS shows the description."
as shown in the attachment in my (original) post.

For my part, I cannot see the wording "overwrite right item" anywhere, including in balloons/tool-tips.(!) If you're on Windows, does this app's Mac and Windows versions have different wording???

Thank you for pointing me to the File Versioning page. Now I also found it as a link in the Sync Settings panel.

If "3. Save versions at certain intervals" with the timestamp macro works as advertised, it is exactly what I was/am looking for. This, coupled to 'Update >' would make the setup a classic incremental backup. It is so useful and perfect for home users with one or two computers that it should have a 'click-and-choose' 'variant' button.

[Plerry]

Could not find any attachment with said "Update right item" reference.
Added mine below, with the mouse over the green arrow (without the "+").

FFS_tooltip.png (15.72 KiB) Viewed 4251 times

This, coupled to 'Update >' would make the setup a classic incremental backup

FFS always makes incremental syncs (at file level, not at block-level),
i.e. it will not touch, copy or move files that have not changed (left and right) since the last sync.
This applies to all sync variants. Two-way, Mirror, Update and Custom.
The difference between Mirror and Update is that:
* the Mirror variant will delete any right-side items that do not or no longer exist on the left-side (with deletion handled according to the Delete Files setting)
while
* the Update variant will not touch (leave where they are) any right-side items that do not or no longer exist on the left-side
The Mirror variant, just like the Update variant, (for both: when combined with versioning) might therefore very well also fit the definition of a classic incremental backup.

[kds-kds]

I am trying to attach it again.

FreeFileSync-UpdateRightItem-Balloon.jpeg (85.55 KiB) Viewed 4242 times

Re "the Update variant will not touch (leave where they are) any right-side items . . ." not quite. It depends on the choice of the buttons in the lower left in Sync Settings. 'Recycle bin' will not delete them, yes, but it will not "leave where they are" either. 'Permanent' will delete them. That is why I wrote that it is the combination of 'Update >' and 'Versioning' that makes for a classic incremental backup. Without versioning the updation would be called 'replication.'

I don't think it's correct to call mirroring any type of backup ("The Mirror variant, just like the Update variant, (for both: when combined with versioning) might therefore very well also fit the definition of a classic incremental backup."). The two are not synonymous.

The engineer who designed and built this excellent utility is obviously brilliant but he is not a native English-language speaker. Consequently his title/instruction "Select a variant" has possibly caused you (and maybe others?) to 'think' incorrectly, hence your phrasing. Synchronization, replication / mirroring, and backing-up are not 'variants.' These are distinct operations or tasks.

Again, thanks for replying.

[kds-kds]

Oops – and not even only 'Update >' and 'Versioning' , but 'Update >' and 'Versioning' and timestamp macro.

[Gianni1962]

FFS is a sync tool not a Backup program.

You are misinterpreting all the options.

[Plerry]

@kds-kds
Delete Files settings are only relevant for a (pre-existing) file in the (left or) right location that will be overwritten or deleted when running the FFS sync.
For the Mirror variant:
* Overwritten because a more recent version (of that file) exists in the left location.
* Deleted because that file no (longer) exist in the left location.
For the Update variant:
* Overwritten because a more recent version (of that file) exists in the left location.
* Deleted never applies.
For the Two-way variant:
* Slightly more complex rules apply, resembling to those for Mirror.

In your reply, your "not quite" suggests my quoted statement would be incorrect.
However, I wrote (and note the part that you forgot to quote):

* the Update variant will not touch (leave where they are) any right-side items that do not or no longer exist on the left-side

For said items, Delete Files settings are not relevant, because such items will not be deleted (nor overwritten). Thus, FFS does "leave (said files) where they are".
Your following reasoning is therefore incorrect and not relevant in support of your whatever you earlier stated makes for a "classic incremental backup"; a vague term that you (again) lack to define.

The rest of your reply is mostly about semantics; a topic on which discussion tends to be futile.
Linking the wording and terminology in the FFS help-file and manual, and my consequential alleged "incorrect thinking" and phrasing to the FFS author (supposedly) being non-native in English, sounds condescending, both to the author and me.
Did you ever consider the possibility you might be misinterpreting or thinking incorrectly ... ?

[kds-kds]

Gianni1962, Just because this utility has 'Sync' in the name does not mean it is exclusively that. Do you think that the utility named 'Preview' in Mac is only a preview utility and must not be used for any other purpose?
In any event I hope you don't have a problem if I use it as a backup utility.

[kds-kds]

Plerry,

Re "Did you ever consider the possibility you might be misinterpreting or thinking incorrectly ... ?"
No, because I know what I am talking about. When I don't, I stay shut. You don't know who I am so you don't know my software engineering skills, my coding accomplishments (including, apropos to the subject in question, writing a VxWorks replication component for a switch) nor do you know of my knowledge of taking backups since the days of tape. Nor of my experience in writing sh scripts and C code to recover from RCS/CVS problems. I shall leave it there.

WRT my previous post, my intention was to have a civil and professional discussion towards what I thought may be some common purpose, not to show you that I'm right or get into a flare war. (The hoped-for common purpose being, among other things, a strict definition of terms and operations to preclude muddled thinking and miscommunication in view of what I have observed on this forum, i.e. a lot of unnecessary and avoidable confusion. Indeed, the very reason that I posted my question arose from precisely what you dismiss as 'semantics.') Your defensiveness and attitude preclude any discussion, civil or professional.

Re "a "classic incremental backup"; a vague term that you (again) lack to define." the very question is revealing. Nobody should need to define such a well-established term.

Finally, the stuff ("not deleted," "not relevant") you have written is incomplete and therefore wrong, both in theory and also per what FreeFileSync's documentation and GUI options suggest. Moreover, you have contradicted yourself. However, I have no desire to argue with you; I shall simply use the utility the way I wish to and, in addition, use it in all its feature-richness instead of the constrained way you would like me to.

I would ask you please to refrain from posting any further responses on my posts because, you will agree, it will help no-one and we obviously do not think highly of each other. In return, I commit not to posting on yours.

Thanks.

[Lady Fitzgerald]

FFS is a sync tool not a Backup program.

You are misinterpreting all the options.

Actually, you are doing some misinterpreting. When set to Mirror mode, a folder/file syncing program is excellent for backing up data; in fact, in most cases, it is the fastest and most efficient way to backup data (it will not work for System—OS and program—files so those files should be on a separate drive or partition to segregate them from data).

I use FreeFileSync to back up all my data drives in my desktop computer and the data partition in my single drive notebooks. I use it in Mirror mode and have Versioning enabled. I also set it to verify the backups.

[Plerry]

Actually, you are doing some misinterpreting. When set to Mirror mode, a folder/file syncing program is excellent for backing up data; ...

Why would this be a misinterpretation?
In Mirror mode, FFS "just" makes a copy of the most recent state.
Versioning adds a little extra, but backup is about much more.
Enjoy reading The Tao of Backup (and forgive them the twist at the end ...).

FFS claims to be a synchronization tool (with quite some extra's) and (i.m.o.) is excellent in that.
FFS does not claim to be a backup tool. And (i.m.o.) rightfully so because, although FFS can be used to perform some backup related tasks, it lacks several vital features to make it a true backup tool.

[Lady Fitzgerald]

Actually, you are doing some misinterpreting. When set to Mirror mode, a folder/file syncing program is excellent for backing up data; ...

Why would this be a misinterpretation?
In Mirror mode, FFS "just" makes a copy of the most recent state.
Versioning adds a little extra, but backup is about much more.
Enjoy reading The Tao of Backup (and forgive them the twist at the end ...).

FFS claims to be a synchronization tool (with quite some extra's) and (i.m.o.) is excellent in that.
FFS does not claim to be a backup tool. And (i.m.o.) rightfully so because, although FFS can be used to perform some backup related tasks, it lacks several vital features to make it a true backup tool.

You are not paying attention. First, I was referring to using FFS for backing up DATA only, not System (OS and programs) files. I stated clearly that one's DATA should be segregated from one's System files. System files are best backed up using imaging with a program such as Macrium Reflect Free. Those images should also be backed up as data. I do not recommend using incremental or differential imaging; full images are much safer.

All data backups are just copies of one's data. What the heck else would they be? FFS set to mirror mode is a fast, efficient way to make, then update those copies. One could use imaging to backup data but it would take much longer and take up far more disk space for multiple versions.

For one's data to be reasonably safe, it should exist in at least three places, such as on the computer, on an on site backup drive, and on an offsite backup drive. To be true backups, backup drives must be kept powered down and disconnected from the computer except when updating them.

I have a set of four backup drives—two onsite and two offsite in each set—for each data drive in my computer since even backup drives can fail without warning. I keep my offsite backup drives in my safe deposit box at my credit union and swap them out no less than once a month with my onsite backup drives. That rules out automatic backups since they would require that the backup drive be kept powered up and connected to the computer at all times.

Since data created or changed after the offsite backups were updated and placed into the safe deposit box, I also use a paid cloud backup service (NOT cloud storage!) to backup my data. If I should lose my original data and onsite backups, such as in a house fire as your silly little story suggested, I can quickly recover most of my data from the offsite backups and the remainder can be downloaded from my cloud backup. If the unthinkable should happen, such as an asteroid strike taking out my computer, onsite backups, and offsite backups while I'm fortunately out of town, I can recover all my data from my cloud backup, although it would take several weeks to download it (the reason why I do not depend on a cloud backup only to protect my data; people with less data than what I have ). Paid could backup services (I had been using Carbonite.com but I recently started using CrashPlan Small Business since it has unlimited versioning retention and has no file size and file type restrictions, making it worth the extra cost; for now, I still have both working but will be dropping Carbonite soon once I'm confident CrashPlan is doing the job).

As your silly little story hinted at, one should check to make sure one's data has not been corrupted or infected by malware before updating a backup. I have multiple layers of antivirus/antimalware protection in place, some of it with fulltime protection, and run scans before updating my backups; I also run scans on a regular, frequent basis.

[Plerry]

You are right: I was not paying (full) attention. You wrote:

... a folder/file syncing program is excellent for backing up data ...

and as far as FFS is concerned I can only fully agree, at least when used with Versioning.
I did not notice the fine distinction between a tool for backing up data and a backup tool (or backup program, as worded by Gianni1962).

Although FFS (when used properly) is an excellent tool for backing up data, I do not consider FFS to be an excellent backup tool, as (quoting my earlier post) "it lacks several vital features to make it a true backup tool".
The features lacking to potentially make FFS an excellent backup tool would be

- a true verification feature
As stated by the FFS author e.g. here, the verification method used by FFS (if activated) is not a real verification, as the verification might use the contents of a buffer or cache, rather than what is actually read from the target disk.
- a recover/restore feature
At individual file level, restoring is a manual, yet quite doable activity .
But just try to restore an entire folder tree to the state it was in
at a certain date/time (not being the state at the time of your latest sync) ...
All data is there, but restoring it is a nightmare.

N.B.: I do not criticize FFS for lacking these features. It is ultimate all about choices and priorities.
I love FFS for its excellent file-syncing properties and am meanwhile using it for that very purpose for quite some years to (almost) full satisfaction .

[Zenju]

a true verification feature
As stated by the FFS author e.g. here, the verification method used by FFS (if activated) is not a real verification, as the verification might use the contents of a buffer or cache, rather than what is actually read from the target disk.

I think FFS has this (and it was specifically designed with the backup scenario in mind): binary comparison. The main advantage compared to verification after file copy is that it's completely detached and for example can be run after a computer restart that makes sure that all buffers at various levels are clean.

a recover/restore feature
At individual file level, restoring is a manual, yet quite doable activity .
But just try to restore an entire folder tree to the state it was in
at a certain date/time (not being the state at the time of your latest sync) ...
All data is there, but restoring it is a nightmare.

Now this sounds like a very cool feature to have. Maybe it is as simple as to provide a context menu that is showing all the old versions of a file upon right-click. But it needs to be integrated with the Versioning functionality. viewtopic.php?t=3908

[kds-kds]

Lady Fitzgerald and Zenju, first, I hope you shall not misinterpret me or my motives ('condescending', 'futile', etc.) or get into a tussle to try to prove who is right.

Lady Fitzgerald's posts are spot on; thank you for entering this discussion; and Zenju, as the author of the utility, thank you for posting.

I am really pressed for time so a quick response, leaving out a number of points that come to mind.

This unusual utility in which related but distinct functions and operations are arranged in a single application and which, apparently, allows hybrid operations to be done cannot be pigeonholed into an 'x' or 'y' utility, and the word 'sync' may be doing just that. It is more of a make-of-it-what-you-can-and-will sort of software. It can be called 'All-Round Data Transfer-and-Redundancy Utility.' (I do not think that 'data' excludes binary files; it encompasses bits and bytes of all kinds.) What it lacks in specialization, which is not what a home user or lay user needs, it more than makes up in breadth and range.

I hope you realize that terminology can also cause misunderstandings. For example, the meaning of the word 'versioning' used here is different from what a configuration management professional ('CM guy') would understand it as. (For personal backups I by far prefer unversioned files as-is on the backup drive for convenience and also other reasons so FFS's backups are perfect for me.) However, a CM person would say, "Versioning? I don't see it in FFS."

You may also consider matching the various operations and functions with respect to their temporal usages: Episodic, Periodic (Scheduled or On-Demand), and Continuous. (It's not really a 'matching', I use the word loosely; it's a many-to-many.) That would allow persons to conceptualize each operation within the whole utility somewhat better.

I was going to write to you about a Restore function. It is Backup and Restore that are coupled; not the other operations and so it would be a bit of a peculiar coupling such a utility with 'FreeFileSync.' As the utility is already so 'catholic,' perhaps Restore could be rolled into it? Though that might engender further confusion.(?)

What someone else has written is not correct; if you have a proper backup tree, then Restoring is a simple and mechanical task. From each succeeding incremental backup directory, all dirents that are not directories, including links, should be copied regardless, whereas for a directory if it exists on the target, the two should be iterated through. (What I have written is based on my belief and understanding that most true Backup-Restore utilities 'see' data-storage devices as we do, as filesystems with discrete files, and not as a device with blocks or chunking.)

That said, there may be a tricky scenario. With periodic backups using a standalone third-party utility (as opposed to a suite that, say, IBM would supply), the coupled Restore may not be able to account for moved files and renamed files. (I use 'may not' even though I think 'would not' is accurate because there could be something I don't know.) It would not be difficult for FFS's 'Update,' given its existing features, to try to figure out when it encounters a renamed file and relay it to (its coupled) Restore utility, should it be used, via a persistent 'handshake file.' I don't know if it's possible for non-vendor-supplied utilities to determine when a file has been moved but perhaps modern O.S.'s make this information available to any ole third-party software on an on-demand basis.(?) In any event a user's confusion or annoyance caused by seeing an old version of a file he/she had moved or renamed is probably a small price to pay for a successful restore.

[kds-kds]

Just to correct myself. What I wrote about restoring and restore is irrelevant and does not apply to FFS as the way it does backups is not incremental backups on successive branches off the baseline backup, but keeps the backup directory itself up-to-date and current, while optionally archiving out changed files in whatever way you configure it to. And so if a restore is needed, it's as simple as copying the backup dir.