Feature Request: Limiting Versions

[prog9999]

A really great, usuful app!

I have FreeFileSync running on 50+ servers with most versioning. Unfortunately, FreeFileSync with versioning is eating up many of these disks (taking all the available space) because it seems there is no way to automatically limit the number of versions to keep. It requires a lot of manual maintenance and monitoring to keep FreeFileSync in check. Please, please add this feature!!!

P Prog

[Zenju]

Re-introducing the versioning limit is in planning. What specific kind of limit would you like to see for your scenario?

[stoff7]

A limit on the number of versions would be great. Also, automtic deletions of versions according to age. It would be amazing if these settings could be applied on an individual folder or file outside of the default.
LOVE this software.

Stoff

[Plerry]

@Zenju:
Great to see that you are planning to (re-)introduce version limiting.
The present options, i.e.
* just the latest previous version or
* all previous versions,
are a bit limited.

On topic:
What would probably do is
* A number of previous versions (say: X), and
* an age (say: Y).
Both for X and Y, 0 (zero) could qualify as infinite (=no limit).

But how to combine the above? Variants are:
A) Retain at most the latest X previous versions (if available) BUT ONLY those previous versions younger than Y
B) Retain at least the latest X previous versions (if available) PLUS all/any other previous versions younger than Y

Personally I would have a need for variant B), but I can imagine that the versioning options window would allow to choose not only the value of X and Y but also to choose between variant A) or B).

Variant A) can probably be referred to as the (logical) AND variant or as the MAX variant:
“Retain those previous version younger than Y AND belonging to the latest X versions”
Variant B) can probably be referred to as the (logical) OR variant or as the MIN variant
“Retain those previous version younger than Y OR belonging to the latest X versions”:

[Zenju]

I'm wondering if it would suffice to add a single option "keep last x days". As always I'm trying to minimize the impact on the GUI and to avoid implementing features that hardly anyone wil be using (while FFS's future development could be constrained)

Interestingly I hardly get any requests to implement a limit for versioning which is quite different compared to a few years back before I implemented the current versioning design. So it seems that giving users the freedom of using their own syntax for the versioning scheme e.g. by using macros like %timestamp% solved like 90% of the requirements of versioning. So apparently there's quite a lot right and very little wrong with the current design, therefore adding a limit of whatever kind must not break this advantage. I'm mentioning this because it might be necessary to limit the freedom of the versioning path in order to technically implement a limit.

Ideally a "limit" could be offered for both versioning naming conventions "replace" and "time stamp". I'm not yet sure how this could technically be done for naming convention "replace" with a versioning path like "C:\revisions\%timestamp%".

[bgregory01]

I'm wondering if it would suffice to add a single option "keep last x days". As always I'm trying to minimize the impact on the GUI and to avoid implementing features that hardly anyone wil be using (while FFS's future development could be constrained)

Interestingly I hardly get any requests to implement a limit for versioning which is quite different compared to a few years back before I implemented the current versioning design. So it seems that giving users the freedom of using their own syntax for the versioning scheme e.g. by using macros like %timestamp% solved like 90% of the requirements of versioning. So apparently there's quite a lot right and very little wrong with the current design, therefore adding a limit of whatever kind must not break this advantage. I'm mentioning this because it might be necessary to limit the freedom of the versioning path in order to technically implement a limit.

Ideally a "limit" could be offered for both versioning naming conventions "replace" and "time stamp". I'm not yet sure how this could technically be done for naming convention "replace" with a versioning path like "C:\revisions\%timestamp%".

Zenju,
You mentioned that "Interestingly I hardly get any requests to implement a limit for versioning".
I would welcome this in a hearbeat. I actually had to write my own program to clean up the old versioning folders. Not the right thread for this but I though I would sneak it in...How about separate checkboxes on each folder pair to include/exclude syncing?

[immortal192]

I spent ~15 minutes looking for this feature on the FFS GUI because I assumed it had what I thought was a basic feature of a program that is primarily used as sync/backup that supports versioning (in fact, I came across FFS when I was looking for a lightweight open source automated backup solution that supported versioning). Kind of surprised that such a request was made 3+ years ago and only recently gained some more popularity but I guess it's never too late.

Anyway, I don't think "keep last x days" is ideal because many people may run have an automated FFS batch script that is set to run "every x hours." A significantly more flexible option (and an expected one) would be to "limit to x versions/backups" or "limit to x size" so that the user-defined folder (these two are the only ones I think are practical/useful and both should be implemented in my opinion). .

If possible, do you have an ETA? This feature is essential for my purposes and I am tired of writing separate batch files to do my own auto-deletion of old version backups--I need to know whether it is worth learning/using another backup solution with this feature or just wait for FFS to support it if it can be implemented in the near future.

Of course, no rush. Still an excellent application and it is a good thing that developers remain open-minded and active to their products.

[Zenju]

It looks like a limit by date for individual files can only be efficiently implemented for a fixed naming convention like
C:\revisions\%timestamp%\file.doc

while a limit by number of versions only for a naming convention like:
C:\revisions\file.doc %timestamp%.doc

Considering that currently FFS has two naming conventions "Replace" and "Time Stamp" this means that we'd need a third one. A limit on the number of sessions instead of by date would also only be possible for the third variant:

Replace -> no support for a limit
Time Stamp -> limit by number of versions only
"New one" -> limit by date only, limit by number of sessions

I'm not sure if I like this design very much, hopefully improvements are possible.

Limit by file size? That's a re-implementation of Windows Recycle bin and would require a database file as an index for quick access.

[azatuenthome]

I too would like to have the ability to limit the number of previous versions of files. Thanks.

[Plerry]

> It looks like a limit by date for individual files can only be efficiently implemented for a fixed naming convention like
C:\revisions\%timestamp%\file.doc
while a limit by number of versions only for a naming convention like:
C:\revisions\file.doc %timestamp%.doc


I can understand why the C:\revisions\%timestamp%\file.doc naming convention is challenging in efficiently limiting the number of versions, as FFS would need to traverse all %timestamp% folders in search of any previous versions of the file in question potentially residing in that %timestamp% folder.

Conversely, I do not understand why the C:\revisions\file.doc %timestamp%.doc naming convention would be inefficient in (also) limiting by date. All previous versions of any given file are available in one and the same folder and easily identifyable by date by means of the %timestamp% part in the file name.
In a system/protocol that retains file-dates while copying and/or renaming, even the system file-date might be used.

[Squashed]

I would like to add my vote for a limit on the number of previous versions of files.

Age would be useful where a file changes rapidly. Even if you kept say 30 copies, you might only be able to go back one day.

SpiderOak (spideroak.com) has a useful utility to archive versions so for example you could keep one copy per hour for the first day, then one per day for a week, then one per week for a month and finally one per month for six months.

Thanks.

[bgregory02]

I've been hoping for version limiting for some time. My requirements are simple, just the number of days to keep would work perfectly for me.

[volker01]

i realize that with: C:\revisions\%Date% and "Replace"
results: One Revision Folder per Day, and one Version of changed (or deleted) files within.
A batch file is cleaning the Subfolders in the Revision Folder. Hold the newest "n" Folders - delete the rest.
One Problem are long file names - its not really possible to delete with a batch file.
For that you can use ffs: mirror a empty folder to the folder you want to delete.

[GSV3MiaC]

Re-introducing the versioning limit is in planning. What specific kind of limit would you like to see for your scenario?

I'm just installing it to replace 'second copy' which seems to be (paid for!) abandon-ware these days, and the one thing I miss is version limiting. Some way to prune by age, or some geometric limit on the way versions are kept (as someone suggested, one from the last hour, plus one from the last day, plus one from the last week, and so on
.. ideally configurable per backup job). My mediaplayer is apt to update 'last played' tags which can cause lots of versions of .mp3 files to be spawned (wanted .. just one or two), and some of my spreadsheets change daily, and I really would like to find one from last month, before I messed up the math.

Perhaps something like 'if number of versions >n' then when you go to create a new one, delete the (newest|oldest|a random one in the middle). User selectable. 8>.

A separate batch job to clean up would be OK, but as someone said long fie names are apt to annoy windows (and/or the NAS), and anything that looks like 'del *.*' is seriously scary. 8>. Having them not spawned in such numbers in the first place would be better.

[korleone]

Hi.

I am also interested in this function.

One possibility is to set general settings in versions folder within the destination and, of course, excluded from synchronization, which are saved in folders by% timestamp% specified number of versions. All this as an option when choosing versioning. This is what makes about "Allway Sync" which creates the "_SYNCAPP \ Versioning" folder with hidden attributes.

Otherwise it seems a fairly complete program.

[mariobrito]

Hi,

For the first time in my life, i'm going to donate for a freeware :)

I would too love to have an old version limit function. Either by max number of versions or by "older then xxxx date".

It's the only function i feel that's missing.

Thank you so much for your work !

[fluff]

I've just joined this forum to add my support for this feature.
It would be great if the pruning was more severe for older versions.
I run a daily backup. It would be great to keep:
- All backups for the past week
- Weekly backups for the past month
- Monthly backups for the past year

Great app.

[tistar]

If, after a versioning backup, you execute this code:

@echo off
forfiles /p "directory_of_versioning" /s /m *.* /c "cmd /c Del @path" /d -7


Where "directory_of_versioning" is something "z:\backup" where z:\backup\%timestamp% is you versioning directory?

[tommycrock]

I still use FFS 5.9 to enable version limiting. For example, the metadata in my music files can change often but the files are massive and I only sync once a week so I limit to 2 old versions. My documents are generally small but on instant sync so I limit to 10 old versions. Other things I keep only one old version of (which is easily achievable in other FFS versions).
I would love to see this version limiting be reintroduced so I can get up to date, though I haven't had any problems with this old version of FFS. As others have suggested, combining this with time limits would be amazing (e.g. automatically delete a version when it becomes two years old where another version exists).

Thanks for the amazing software

[Randy]

I love this program. However, I am also still using FFS version 5.8 because it has built in backup versioning. The new updates of FFS are not very useful without this feature...even if they improve other features. Each time I have tried the newest version over the last several years, I have always had to uninstall it and reinstall 5.8 in order to keep this feature.

I have versioning set for 5 times because I have many large files that would fill up my 1TB backup drive too quickly if it were unlimited. However, when these files are altered, we may not notice an undesired change until several syncs have run. One of the best features of FFS 5.8 is the ability to go back to a file from several versions ago and find a file that has the missing information.

The old versioning feature in FFS 5.8 was simple, worked well, and was easy to understand.

PLEASE, PLEASE bring back the versioning limit in a future update.

[Zargron]

I'm loving FFS and also feel "Limiting Versions" would be a very valuable feature to add. IMHO, as is often the case in this type of scenario, a small but significant change might accommodate 90% of user requirements. BTW: i haven’t used an edition of FFS that contains versioning. Needless to say, without having put too much thought into it, might i entertain thee…

1) GUI - Synchronisation Tab
This tab already has a section for “Delete files / versioning”. Proposed changes:

a) ]When select “Replace”: present numeric edit box titled “Number of Versions”, (default=1)
b) When “Number of Versions=1”, behaviour of the “Replace” versioning method to be exactly as it is now
c) With “Number of Versions>1”, keep number of specified versions.
d) Edit box “Version Storage Path” could be used as is, depending on nature of implementation.

2) Implementation
Without Database

- My thoughts here involve not letting user put any date/time macros into version storage path. (Zenju, you already hinted at the potential to “limit the freedom of the versioning path”.)
- Perhaps the only macros allowed are %UserName% and %ComputerName%.
- At start of synchronise job, whilst resolving version storage path, validate that it doesn’t contain anything “illegal”.
- Within resolved version storage path, create sub-folder “FFS_ %timestamp%”
- During synchronise, save files into resolved and time stamped version storage path, preserving source folder structure.
- Towards end of synchronise job, count number of folders within the resolved version storage path.
- If the number of folders exceeds “Number of Versions”, delete oldest versions.

For example, with version storage path of “X:\Revisions\%UserName%\”,
- X:\Revisions\Zenju\FFS_2016-10-15 171423\MyWork\...
- X:\Revisions\Zenju\FFS_2016-10-16 184344\MyWork\...
- X:\Revisions\Zenju\FFS_2016-10-17 165501\MyWork\...
- Count time stamped FFS folders within X:\Revisions\Zenju\, delete oldest in excess of “Number of Versions”.

With Database
- Allow user to put anything into version storage path.
- At start of synchronise job, resolve version storage path and create sub-folder “FFS_ %timestamp%”.
- Save resolved and time stamped version storage path to database.
- During synchronise, save files appropriately.
- Towards end of synchronise job, check database.
- If the number of entries exceeds “Number of Versions”, locate folder and delete oldest version.

For various reasons, i’d prefer without database. It appears from the comments in this thread that several users are writing scripts to keep versioning under control. I’m guessing they don’t use fancy database solutions to implement their requirements.

Obviously this suggestion only allows for count of versions. That is how i generally implement for my self and clients. You want 4 times a day backup and keep last 7 days, set version count=28. You want once a day backup and keep last 7 days, set version count=7.

BTW: is it by design that “versioning path” is allowed to be the same as “target folder”? For example, FFS allows you to set “D:\MyWork” as target folder in the main window whilst also letting you set “D:\MyWork” as the versioning path in the “Synchronisation Settings” window.

[jimlad68]

I'm hoping to move from an old version of ViceVersa to FreeFileSync but the lack of versioning/archive options is a bit of a stumbling block.

At present I use ViceVersa's Smart Archive: i.e.
ViceVersa keeps one file version every five minutes for the last hour, one file version every hour during the last day, one file version every day for the last month plus one file version per month.

The advantage of this is that it deletes versions in such a manner that there is always a good varied time selection.

So that would be my request:

I suppose someone with the knowhow could create a batch jobs that did similar.

For info ViceVersa also has these options:
Move Files to Archive Folders (instead of copying):
Keep Max. Copies per File
Remove Archive Copies older than:

[DrLove73]

Hi.

I am very happy with FFS so far, except for the fact versioning limiting is missing. I started using it about a month ago, and so far I built few scripts in pyton and windows batch script to create:
1. folders on the server that will be used for storing separate folders (if you decide to backup only "C:\Users", you need to already have existing folder "\\server\<backup storage forlder>\<PC folder>\C\Users"),
2. To create ".ffs_batch" files for each PC in a network/domain (from csv file) from standard templates,
3. Import Task to client PC that will schedule the FFS.
4. To create on client PC admin user/pass used for backup who has access to backup storage,
5. copy all necessary files (I create C:\Windows\Sync folder and copy personalized ".ffs_batch" to it),
6. add network drive for users server-stored documents (manual because if there is no domain, it is the only way),

But, syncing Outlook or Thunderbird mail profiles each 10-20GB in size every day is VERY problematic. I guess I will have to schedule a clensing program to keep sync storage light... Or I will have to experiment with those older programs, Vice Versa or FFS 5.8.

If everything goes as planed and I sell the system to few (smaller) companies here, I plan to donate.

[gandlz]

Hello,

just want to add my opinion: I really like the program and use it mostly as a copy tool. A few times I also used it as a backup tool but only without versioning and without scheduling. For example connecting an external HD and create or renew a mirror of the source. That’s OK, if your data is not very important and you only do this once a month or so.

But nowadays we have NAS and want to automatically have a backup after every day. That’s when you start needing versioning because you could find out a big mistake after a week but this mistake is already included in the backup.
And if you need versioning you NEED some kind of space management. I know hard disks are not that expensive anymore but a backup solution that needs unlimited space is absolutely unacceptable. And that’s what it is, as far as you need to monitor backup space manually or with a second program, you have to say that FreeFileSync with versioning enabled needs UNLIMITED space!

I understand that according to the different versioning schemes you may want or need different retention rules and that may not be easy, if you want to keep the program simple and understandable for every user.

But before doing nothing you should at least include a feature like “delete files older than …”
That’s enough to make a roughly space calculation and to avoid the need of unlimited space.

Thanks,
Gandlz

[DrLove73]

Actually, deleting files "older then" can delete ALL versioning files if file was not updated longer then set date. It is necessary to have "keep only X versions" of the file to protect few versioning files from deletion because they were not changed for several days, a months, whatever.

[gandlz]

Well, I agree this can be your need but it‘s not what I want. To explain you why, you have to know that I personally use a shareware imaging software to make backups. It makes a full image every Monday and incremental images every other day. Backup space is just enough to keep two backups (full and according increments). This means no more than 2 weeks! I use this PC for my business and also privat.

So according to my experience over the last about 10 years:

I never needed a backed up version that would be older than my backup was. Usually you need a backup just as long as you are on the current project. If you do audio or video stuff you may have really big files and in the end you don’t want to have several versions of a finished project in you backup. For me it is absolutely OK to have only the last version backed up if it is not changed for 2 weeks. Others may increase this time. If I come to a point during work where I think this should be saved to go back if needed later, I will make a copy on my data drive and have this until I delete it manually!

Also with your method, space calculation would be not possible or at least not so easy. For me personally this way would mean sizing up my backup with a lot of files I will never need again.

But I agree: if it is called “versioning”, its primary feature should be the creation of versions. Maybe we should have the possibility to choose this and the other way, let’s call it “retention rule” or “deleting method” or just “backup”.

Yes, I think I need a backup but not the creation of versions. Whenever I needed to go to my backups, I needed the most recent one. The question was “is this already in the backup” and not “is this still in my backup”. Since I am doing backups, I needed them when my hard drive fails, a virus came in, I deleted something accidently or I made a big mistake while working (I usually know this within 1-3 days).

Sorry for writing so much, but I just want to share my experience and backup philosophy!
Thanks.

[DrLove73]

Well, creating backups from one location and syncing two locations are slightly different things. Versioning is meant to provide different versions of the file in case we have to revert to earlier VERSION of the file. Versions in FFS are being moved to separate location, they are not meant to be in same folder as latest version.