SOLVED: Btrfs compression is not applied when copying files via FFS 12.2

[MadMax]

Hello everyone! I am a big fan of FFS and use it almost everytime I have to copy or sync a large amount of data from one drive to another!

Today I noticed the following behaviour: When copying data to a btrfs partition with enabled compression (mount option compress-force=zstd:6) data copied with FFS does ignore the compression setting. When checked with the btrfs tool compsize the data is listed as "compression: none". This is also verified by the transfer speed which was way to high for zstd:6 - hence why I noticed it.

When copying the same data via a file manager or rsync (which I am using for the job right now) compression is applied correctly.

Is this a bug or am I missing some special setting in FFS?


Used versions: FFS 12.2, Btrfs progs 5.16.2, Linux Mint 21.1, Kernel 5.15.0.72 #79-Ubuntu

[Zenju]

I believe btrfs compression should be done transparently by the kernel, i.e. is handled automatically inside "read" and "write" Posix calls?
Since this is all FFS does, the target file should be expected to be compressed as well.
For further testing, you could use "cp" and see if the target file is compressed.

If it is, one could find out the difference between "cp" and FFS by testing with a single small file, while under "strace". https://freefilesync.org/faq.php#trace

[MadMax]

Thank you for the reply :)

I can confirm that both rsync -av as well as cp -v write the files in a compressed manner to my btrfs filesystem (or rather the kernel/btrfs itself does it right).

In the meanwhile I found out the "faulty" behaviour happens not only with FFS, but also with the program that creates the files in the first place. Moving newly created files to another file system and copying them back solves the problem.

From what I've read it might have to do with the call of a sync command during file transfers. But since Btrfs is a rabbit hole in itself I have not delved deeper into that yet...

Curiously there were also 5.4 G which did not get compressed during the initial sync with rsync. It might be that these 5.4 G are uncompressable which I doubt, though. The difference of 1.1 G have yet to be moved and copied over. Unfortunately there also seems to be no easy way to find these uncompressed files, but this topic is out of the scope of the FFS forum :)

Processed 7296 files, 4156665 regular extents (4156839 refs), 2 inline.
Type       Perc     Disk Usage   Uncompressed Referenced 
TOTAL       82%      420G         511G         511G      
none       100%      6.5G         6.5G         6.5G      
zstd        81%      413G         505G         505G 


[MadMax]

I also did a test with strace with a fresh btrfs subvolume to which I copied 5 files with 640 M.

strace freefilesync &> ~/ffstrace.txt


Result after the copy process with FFS:

sudo compsize /mnt/more
Processed 5 files, 5 regular extents (5 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced 
TOTAL      100%      639M         639M         639M      
none       100%      639M         639M         639M   


Then I deleted the files and copied them again with cp from within the directory where the files are:

strace cp * /mnt/more/ &> ~/cptrace.txt


Result as expected by previous tests (also note the number of extents compsize is showing is apparently bugged).

sudo compsize /mnt/more
Processed 5 files, 5120 regular extents (5120 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced 
TOTAL       80%      514M         639M         639M      
zstd        80%      514M         639M         639M 


I attached both strace results to this post. I am not sure if I did the strace for FFS right, since its a lot smaller than the cp one and apparently doesn't contain the copy procedure? I guess you will be familiar with FFS's behaviour though, so the cp one might be more important.

I'd be glad if this helped to improve FFS's feature set :D

[Zenju]

The strace for FreeFileSync should be run for the target process "Bin/FreeFileSync_x86_64", not the launcher.

The strace for cp looks like an ordinary read/write file copy loop. FreeFileSync does the same, but also fallocate(FALLOC_FL_KEEP_SIZE). Perhaps this throws btrfs off the tracks for some reason.

Let's test. Here's a version without the fallocate: https://www.mediafire.com/file/bzpjrsjnssqiznl/FreeFileSync_12.4_beta_Linux.tar.gz

[Zenju]

Looks like fallocate(FALLOC_FL_KEEP_SIZE) + btrfs is a problem: https://www.reddit.com/r/btrfs/comments/12zyzq8/rclone_ignoring_btrfs_compression/

[MadMax]

I just installed and tested the 12.4 beta version and it works flawlessly now. Compression is applied correctly.

With the rise of btrfs (Fedora even ships it with zstd:1 compression enabled by default) it would be a neat feature for FFS to have an option to omit the fallocate step or generally omit it by default when writing to a btrfs partition with enabled compression. I think fallocate is done mainly for performance reasons or does omitting it create new problems?

There is also a chance that the btrfs or kernel developers will fix the issue, since that's a bug on their end - but this will take a while...

Great work and thank you!

[Zenju]

I'll remove the fallocate call for the next release. Unlike on Windows and macOS, giving a hint to the file system about the expected file size doesn't seem to be a well established pattern on Linux. (= well intended, but asking for FS implementation bugs). "cp" doesn't fallocate, and a quick and dirty perf test didn't show any speed differences.

[MadMax]

Sounds good.

Since rsync also does not preallocate by default, this should be fine. I just had a quick look at rsync's manpage and there is a switch that has to be turned on specifically.

--preallocate This tells the receiver to allocate each destination file to its eventual size before writing data to the file. Rsync will only use the real filesystem-level preallocation support provided by Linux's fallocate(2) system call or Cygwin's posix_fallocate(3), not the slow glibc implementation that writes a null byte into each block.

Without this option, larger files may not be entirely contiguous on the filesystem, but with this option rsync will probably copy more slowly. If the destination is not an extent-supporting filesystem (such as ext4, xfs, NTFS, etc.), this option may have no positive effect at all.

If combined with --sparse, the file will only have sparse blocks (as opposed to allocated sequences of null bytes) if the kernel version and filesystem type support creating holes in the allocated data.