Moved & Renamed clarification for network locations.

[JC-RFC]

Hi,

Terrific product.

I have searched everywhere but cannot find a definitive answer on moved/renamed when syncing between a local and a unc network location, both ntfs. I know the system uses file-id but I see some reports of people saying it works for their NAS connection and another comment on the forum that network locations don’t have consistent file-id.

If I move a folder of large files on the local machine, say from D:\Current to D:\Archive and the sync target is \\remote\

I find all the files are deleted from \\remote\current and then copied over the network to \\remote\archive.

Is this expected or should moving work for this scenario? If it should work any ideas why it’s not for me, eg unc vs mapped drive etc.

I am mirroring with database on etc, can see the database files are there and have synched this pair many times.

Finally if it is expected this should not work I’d love for it to be a new feature that could be made to work somehow, even if another instance need run on the remote machine for example?

[therube]

(I don't know enough, but...)

Is the sync.db (or whatever it is) created on both the source & target ends?
(And this assumes that you do have, "Use database file to detect changes" enabled in the Sync settings.)

If you take the UNC out of the equation...

If you have (folder pairs of)
D:\Current as source, &
D:\Target as target

If you move a folder of (small) files on the local machine, say
from D:\Current to D:\Archive,
and the sync target is (still) D:\Target\

Are the files deleted from D:\Target\Current\
& then copied over to D:\Target\Archive

or are then simply, moved?

[xCSxXenon]

It sounds like the OP has two folder pairs:
D:\Current <-> \\remote\Current
D:\Archive <-> \\remote\archive

In this case, moving files between D:\Current and D:\Archive will NOT move them on \\remote and will delete/recopy. This is because you are moving files between different synchronizations. The "Detect moved/renamed files" option only works within one sync pair. If you instead have it set up to sync
D:\ <-> \\remote
then moving files between D:\Current and D:\Archive will be detecting and the same move in the destination. In order to do this, you need access to the parent folder(s) of \\remote\current and \\remote\archive

[xCSxXenon]

I know you said \\remote is the target, so I may be wrong. Please clarify the exact source(s) and destinhation(s)

[JC-RFC]

Ok to clarify I have a sync rule of:
\\HTPC\Data <-> D:\
Note: \\HTPC\Data is a share on a remote machine that is sharing it's D:\ to the network
D:\ (The right hand side) is a drive pool on the local machine - The local machine name is Server.
This means the sync is effectively \\HTPC\D$ to \\SERVER\D$
I run ffs on Server.
I have write access to local(Server) and remote(HTPC) locations.
I have File time and size set.
8 threads on \\HTPC\Data and D:\
Two way sync, use database to detect changes, permanent delete
Both machines are running Windows 11 Pro and are connected via a LAN and a central unmanaged switch.

I checked and there is a sync.ffs_db file on D:\ (2974kb) and on \\HTPC\Data (803kb) not sure if the different sizes make sense.

If I move a subfolder of files from say D:\To Burn\SubFolder to D:\Archive\Subfolder then the files (and subfolder) are deleted from \\HTPC\Data\To Burn\Subfolder and copied over to \\HTPC\Data\Archive\Subfolder over the network. If subfolder contains say 100GB of data then it takes about 15 minutes at just over 100Mb/s on a 1GE LAN connection which is the expected speed if doing a delete and copy rather than a move.

Should I expect this to work as a move?

[xCSxXenon]

What happens if you map \\HTPC\Data to a drive letter instead of accessing it via UNC path?

[JC-RFC]

Was planning to try that, will do and report back.
The documentation recommends UNC paths and I prefer them but be good to know if this makes the difference I agree.

[JC-RFC]

Ok, I did a quick test setup on another pair of networked computers and moved files works flawlessly in all directions. Whether one side was UNC or a mapped drive did not matter either.
Now to hunt down the issue on my main setup.
But for anyone searching this forum, yes Networked locations definitely works for moved file detection.
Tested with a file that took about 2 minutes to copy across, moving is done in under a second on both sides.

[JC-RFC]

Did some comprehensive testing on my actual setup and confirmed it's drivepool.

I have a Stablebit Drivepool D:, a network location \\HTPC and an external drive E:
Moving files works between E: and \\HTPC but not between D: and \\HTPC or D: and E:
The common denominator is that D: is a Drivepool... https://stablebit.com/
Note I have CoveFs_OpenByFileId enabled in Drivepool and can read a Fileid from D: with:

fsutil file queryfileid D:\somefile.example.


I get

File ID is 0x00000000000000000000000000008745


for example on a file on D:

I then copied a file around and note that its fileid changes on the drivepool each time, same if renamed.
This doesn't happen on \\HTPC or E:

Also found I am not alone... viewtopic.php?t=7413&hilit=drivepool#p25045

Edit: Confirmed categorically here:
https://community.covecube.com/index.php?/topic/12577-beware-of-drivepool-corruption-file-deletion-performance-degradation-scenarios-windows-1011/#comment-47768

[JC-RFC]

I have discussed this with the devs of Drivepool and have the following response:

... incorrectly using File IDs as persistent file identifiers, which they should not be. File IDs in Windows can change from time to time on some filesystems.

Source: https://learn.microsoft.com/en-us/windows/win32/api/fileapi/ns-fileapi-by_handle_file_information

The identifier that is stored in the nFileIndexHigh and nFileIndexLow members is called the file ID. Support for file IDs is file system-specific. File IDs are not guaranteed to be unique over time, because file systems are free to reuse them. In some cases, the file ID for a file can change over time.

If this is the case, then it is expected behavior.

The correct API to use to get a persistent file identifier is FSCTL_CREATE_OR_GET_OBJECT_ID or FSCTL_GET_OBJECT_ID: https://learn.microsoft.com/en-us/windows/win32/api/winioctl/ni-winioctl-fsctl_create_or_get_object_id

Object IDs are persistent and do not change over time.

We support both Object IDs and File IDs.

I did check object-id and it does persist for moves and renames.

What are your thoughts on using object-id instead of file-id?

JC.