Feature Request: Hard Links Handling the same as Symlinks

[johnkmc]

Hi zenju!

Firstly I have to say thank you for such a great product and it is almost perfect with
the inclusion of Symbolic Links handling options.

It would be perfect if Hard Links can be handled the same way too.

I read from a previous Hard Link request in which you had mentioned:

-------------------------------------------------------------------------------------------
I'm not sure if adding support for hard links would be a real benefit for most FFS
users. From a user's point of view they offer a subset of what is possible via
symbolic links, except they model shared ownership, whereas symbolic links model an observer. While
symlinks can be relative or absolute to any volume, hard links are absolute to the same volume, so have a few problems:
1. copy hard link or target file? (similar to symlink's "direct" and "follow") - only useful if source and target are the same volume
2. copy accross volumes: feasible at at all?
3. performance problem: Getting the hard-link refcount requires an additional disk access per file during comparison.
-------------------------------------------------------------------------------------------

My opinions as follows: (as a photographer and videographer using windows 7 x64)

Hard links have some real benefits over symbolic links in some applications:

A) Hard links add extra protection for important camera raw files. (e.g. from a wedding)
The raw file cannot be accidentally deleted easily, since one has to delete all the hard links before
the original file can be deleted. This is not true for symbolic links where the original
file can be accidentally deleted and the symbolic links would become useless.

B) Hard links can be moved around in the same drive into other folders within the same volume,
and the Hard Links are automatically maintained. This feature is very helpful for Adobe Photoshop Lightroom because one sometimes wants to group some pics together in some sub-folders.
For example, Ceremony_shots, Formal_shots, Reception_shots etc. A symbolic link will break
when you move it around and therefore you cannot reorganize the symbolic links yourselves between folders.

C) Shorter access time perhaps?

I am trying to provide inputs according to your comments quoted above:

1) Indeed, if possible, please handle hard links the same way as symbolic links, which you are doing perfectly right now in version 6. i.e. exclude, direct or follow. I love "direct" as it doesn't use any disk space :)
There is really no problem to impose this hard link restriction of having to keep the source and target in the same volume. When I use FFS, I am doing backup anyway and therefore I will backup the source files together with the hard links for sure. The hard links can be used to organize "the photos" into different folders without making duplicates to save disk space.

2) This one I have no clue. Instead of using copy within FFS, can mklink be used?

3) Perhaps an on/off switch is handy for this one. People who don't need the hard link
can turn this off to get the max performance. Perhaps we can make all hardlinks prefixed by
HL_ e.g. the hard link for video1.mov be named HL_video1.mov. This way, FFS will only check
refcount when it sees a prefix of HL_ on the filename. This way, very little performance hit
should result, as the percentage of HL_ files are probably less then 10% of all files to be backup.

Thanks a lot xenju and please keep up with the excellent work!

John

[johnkmc]

Found this wonderful free app (DSFS) to automatically search for duplicate files and one
can then choose to create a hard link for all the duplicates. (to save disk space)

see Duplicates & Same Files Searcher (DSFS):

Working well for me on windows 7 home premium x64.

Trying to fit this tool in the FFS syncing/backup work flow such that
hard links can be synced up as well. (FFS currently only supports symbolic links but
not hard links)

My initial attempt as follows:

Workaround for syncing/backup hard links using FFS & DSFS
(Note I only make hard links for .mov & .cr2 files as they are big)
---------------------------------------------------------

Source disk w:
Mirror disk g:

1) in w: use DSFS to convert all duplicate *.mov and *.cr2 to hard links

- search for *.mov;*.cr2
- select all files
- deselect file path that contains "Original" (Want files inside "Original" to be the "mother")
- convert all selected files to hard link

2) use FFS to mirror w: to g:

- unfortunately FFS will only copy the hard link as a full-size file but not
as a hard link

3) in g: use DSFS to convert all duplicate *.mov and *.cr2 to hard links

- search for *.mov;*.cr2
- select all files
- deselect file path that contains "Original" (Want Originals to be the "mother")
- convert all selected files to hard link


4) use FFS to compare between w: and g: again and hope they will sync up.

- step (4) is not quite working well right now. Sometimes it works but sometimes
it doesn't. When it doesn't work, it flags all the difference between identical files between w: and g: (The contents are identical but the time-stamps are probably different)


Any clue to help with step (4) to make this workaround work?

Thanks a lot! John

[johnkmc]

In step (4), FFS reported that the left side is newer for a number of files.
Is there a way for FFS to update the time stamps on the right side such that the compare becomes identical? This hopefully should fix the problem. Tks!

[johnkmc]

I have adjusted the hard link syncing procedure to yield better results:
(though not perfect and sometimes it doesn't sync well due to different time-stamps)


Source disk w:
Mirror disk g:

1) use FFS to sync w: and g: first.
2) convert duplicated files to hard links in w: (using DSFS)
3) convert duplicated files to hard links in g: (using DSFS)
4) use FFS to sync w: and g: again.

[sfxs]

Has anything happened on this? I'd like to know if this is going to be fixed/option added or will be left as is. I also need hard links to be passed as hard links, or in the worst case scenario, as sym links.

Copying hard links ( file link count >1) is possible, or at least detecting them by link count. Free file sync would just have to manually redo the operation. I.e: grab hard link targeting path h:\bla\bla\file.txt and recreate it to d:\backup\hdrive\bla\bla\file.txt or d:\bla\bla\file.txt. It would be slower, but im sure it wont be any slower than doing a comparison by content. They could also be converted into sym links and passed as "direct", extra hard link pointing to h:\bla\bla\file.txt is recreated as sym link pointing to h:\bla\bla\file.txt if not relative. It would save time in recalculating the new path, but it would be useful as a backup, as in, on a restore, the sym link would be useful towards the same root path (drive letter).

Perhaps something like
Hardlinks:
1)Ignore
2)Follow
3)Redo in target (slower)
4)Convert to Symbolic links

Most syncing software do not really bother with hard links, since in a sense, all files in a NTFS partition are hard links with a count of 1. Multiple entry points are a bit harder to detect than junctions or symlinks.

[ajax]

I share this exact same need. Only difference from Johnkmc is using hard links for both raw and developed files. There are several needs. One is to organize (the same) photos into different galleries without duplicating them. In this case photos are always developed in uncompressed .tiff format, which maximizes size, in order to maximize and preserve quality that even if NOT needed or useful now could become so in the future. Another is keep files that pertain to a particular post processing effort together in one folder, which can include different projects that use the same files for input to the work flow.

Insofar as this thread was started many years ago with NO apparent adoption I have to conclude that a solution is hopeless. Would be nice to find out otherwise.