Mirroring of same sources multiple destinations

[tubemirror]

I am mirroring 3TB of data (400.000 files, 42.000 folders) from multiple data sources (disks) to
- 1x internal (10TB Sata) and
- 2x external (USB3.2) drives.
The file search takes a lot of time and needs to be repeated.

I would propose a mode where the file search is only performed once and being fed to multiple mirroring jobs (to different target disks) so that the mirroring can be performed in parallel.

I think by this you can generate a lot of time savings by
a) having to perform the file search only once
b) by performing the mirror jobs in parallel

I do not expect any I/O problem, because one destination is on internal SATA6 disk and the other two are connected to a separate / dedicated USB 3.1 gen 2 controller supporting UASP, the external case as well:
- StarTech Dual Port USB 3.1 Karte - 10 Gbit/s per Port - 2x USB-A - PCIe, ASM2142 (PEXUSB311A2)
- StarTech USB-B 3.1 Gen2 Enclosure, 3.5", USB-B 3.1 (S351BU313)

I think this could be useful for many people who mirror their data to multiple disks / sticks and want to speed up the whole thing.

Many thanks for considering such a mode to make an extremely well product even better.

Best regards
A.

[xCSxXenon]

As much as the protocols you are mentioning help latency and bandwidth, the more important factor is whether or not you are using an SSD or HDD. FFS does support parallel threads as well. This has been discussed before, but there is no way for FFS to "mirror" operations across multiple destinations without having the destinations mirrored themselves.

[Zenju]

Just use multiple folder pairs. If a source folder is mentioned more than once, it will still be scanned only once.

[downloadgemist]

[ Slightly different topic than the OP, but I tried searching the forum for once instead of making a new thread :) ]

I want to add a thought to this discussion. I sync a fast SSD towards 2 slower HDD's for 'cold storage' of video editing jobs. So i have:
(SSD)C:/folder-A => (HDD)E:/folder-A
(SSD)C:/folder-A => (HDD)F:/folder-A

So this is one source folder towards two destination folders, defined as two folder pairs.

Currently, multi-threading makes no difference, since the joblist will first execute copying all files to disk E, and then to disk F. Since the HDD's max out on writing speed before the reading speed of the SSD, all files are actually copied in series even with multi-threading. Setting multi-threading here actually hurts the performance, since the head has to move to different locations.

The optimal situation would be that both disks are accessed at the same time, but that would involve interleaving the joblist with E and F folders (maybe in the background?)

[xCSxXenon]

[ Slightly different topic than the OP, but I tried searching the forum for once instead of making a new thread :) ]

I want to add a thought to this discussion. I sync a fast SSD towards 2 slower HDD's for 'cold storage' of video editing jobs. So i have:
(SSD)C:/folder-A => (HDD)E:/folder-A
(SSD)C:/folder-A => (HDD)F:/folder-A

So this is one source folder towards two destination folders, defined as two folder pairs.

Currently, multi-threading makes no difference, since the joblist will first execute copying all files to disk E, and then to disk F. Since the HDD's max out on writing speed before the reading speed of the SSD, all files are actually copied in series even with multi-threading. Setting multi-threading here actually hurts the performance, since the head has to move to different locations.

The optimal situation would be that both disks are accessed at the same time, but that would involve interleaving the joblist with E and F folders (maybe in the background?)

This has been discussed before, but there is no way for FFS to "mirror" operations across multiple destinations without having the destinations mirrored themselves.*

* Such as a RAID1 setup.

Setting parallel options won't have any positive impact, as you've seen, since there are no artificial limitations you are encountering. You could have two FFS sessions running at the same time, but FFS locks each base location while operating on it. You would have to disable that feature to achieve this.