Compression / New Thoughts

[meteorquake]

I know compression has come up before but I think I may have some new thoughts, which I'll put in the form of an outlined suggestion regarding NTFS.
The settings themself could have a list of file extensions you want compressed for the backup. Obviously you may want doc,xls,txt,log,wav etc compressed but not docx,odt etc.
When the setting has a value and the filesystem is detected to be valid (NTFS here), when it comes to the copying process, FFS creates an empty folder on the destination with compression flag set.
Whenever it copies a file whose extension is earmarked for compression, it would copy to that destination folder (resulting in a compressed file) then moves it to its target location (thus retaining the compression). This is much better than copying to the destination then setting compression. You could also set it so that source files with C flag are copied in this way. Unless of course windows copying internally allows for copying directly with C flag set??
When copying a file whose extension isn't so earmarked, it copies to the destination in the usual way as a temp file and moves it to the target location.
So the two tasks are almost identical. The resulting files then present themselves in just the usual way as files so wouldn't cause any difficulty to the existing scanning setup, purely a slight adjustment to the copy-via methodology.
The reason for compression is that it saves space (sometimes a considerable amount), some SSD lifespan, and if the drive is on a slow link, then speed too. In this instance for example I am backing up my wife's computer into the spare space on my own and it makes a difference to have some compression, however I will have to make two backup folders - one with C flag set and one without - each with their own separate synchronisation task with different extension filters, but I'm presuming it would be ideal not just for me if it could be rolled into one synchronisation task by such a setting, whose implementation I think would be a fairly trivial change to the normal procedure.
David

[Plerry]

FreeFileSync (FFS) will gladly sync (file-level) compressed files for you.

Just like for encryption, compression is a special task in itself, for which there are several excellent programs available.
It is ultimately up to Zenju, the author of FFS, to decide whether to integrate compression and/or encryption into FFS, but if I were you I would not hold my breath on it ...

[meteorquake]

Cheers. A little bit of a long shot indeed but since the implementation should be trivial I thought it worth it :) :)

[therube]

Just some links.
In particular "some SSD lifespan" would seem to be wrong.
(And supposedly SSD lifespan, in general, is a non-issue these days.)

I'll also note that except for maybe Windows Explorer, other utilities (?) may not "show" the compression (so you may not particularly know what kind of compression ratio you're getting). (And W/E does not really "show" it either, but you could compare a compressed version of a directory against an uncompressed version - via W/E, & you could then "see" the size difference.)

https://superuser.com/questions/1719973/what-exactly-does-ntfs-compression-do-to-files
https://superuser.com/questions/1136329/ntfs-compression-on-ssd-ups-and-downs
https://superuser.com/questions/775042/ntfs-compressed-folders-is-it-possible-to-tweak-compression-ratio
https://superuser.com/questions/411720/how-does-ntfs-compression-affect-performance
https://superuser.com/questions/605398/why-does-ntfs-compression-take-up-a-lot-of-space
.

compressed vs uncompressed folder.png (13.67 KiB) Viewed 6949 times

[meteorquake]

For the SSD lifespan, I've tended to consider it an issue that can raise its head when you are in the later stages of the SSD filling up as there's more wear from juggling around storage to maintain wear-levelling in a diminished space.
I have compressed files set to show in a different colour. VoidTools "Everything" nicely shows the compression ratio as a column, and the Attributes column can also show it.
The CompactOS I find is extremely effective, with files shrinking hugely, sometimes to 20% of their size, which makes a difference for HDD particularly. But it can also be applied manually in bulk to cover all the program files which I do in that scenario.
NTFS compression is reasonable for compressible files and can be useful on SSDs for unusually large exe files or large .wav files or large numbers of e.g. .htm files, which usually compress well due to the repeated tags. For example I have a very large number of PDF files from archiveorg which I maintain TXT versions of as it helps with searching, so a text file might be 19 Mb which is reduced well and transparently by compression.
The main thing for compression is targetted file type ideally with a size threshold (which I do with Everything) - I wouldn't compress an odt or jpg file for example as they're already in a compressed form
David

[meteorquake]

I should provide a screenshot of the EXE compression option in action -

a.png (19.01 KiB) Viewed 6931 times

[therube]

(I'll grant you, that particular file does compress extremely well.
Didn't have it, so downloaded it & sure enough 700KB.
Threw libavcodec_plugin.dll & aticaldd64.dll, both ~16 MB to ARJ/7-zip, & size knocked down to ~6 MB.)

The CompactOS

Since I didn't know, Should you compress C drive or compact OS and how to do them.

Note that he is using a 48 GB drive - GB, not TB (& 48 GB I guess you might consider rather minuscule, these days), & says he gained "2.4 GB of space". Now while percentage-wise that is rather large, in the scheme of things you cannot even find a usb flash drive that small these days.

Here also, the same "savings" is noted, Windows 10 can efficiently compress system files. That gives back approximately 1.5GB of storage for 32-bit and 2.6GB of storage for 64-bit Windows.

On my computer at home, I do have literally a few *GB* of free space on my C: partition. (OS is installed on E:.)

That said, it runs just fine. And until it farts or until I feel like cleaning things up, I'll live with those few GB of free space. (It's been that way for a long time now.)

(Maybe if you're on some sort of "ipod"-like thing, where you might have a tiny drive... [but I know nothing of those types of devices].)

On my current computer, OS drive (Win7) is on a 90 GB partition with 52 GB free.
(And I'm thinking my home computer, Win7 also, is also partitioned that way, 90 GB.
And with such a "small" partition, my OS partition - is for the OS & essentially nothing else.)

[meteorquake]

Yes. CompactOS seems normally to compact just the OS files, but I use it to compact all the program files too; some programs are quite large normally.
What's available for people in a country will depend on matters like the country's economy, or individual finances; people often may get the most out of old machines they have.

[FreeFileLove]

FreeFileSync (FFS) will gladly sync (file-level) compressed files for you.

Using FreeFileSync in Windows 11, I backed up some compressed files (file-level) from an NTFS partition to another NTFS partition, but the destination files are not compressed.

Is there a trick to make the destination files compressed too?

[Plerry]

This sound like you are using the NTFS compression feature on a folder (or perhaps on the entire drive), and that folder (or drive) is part of your sync.
When using the NTFS compression feature, files are compressed at the time of writing, and uncompressed at the time of reading. As a result, the user, or any software tool reading such a file is simply presented with the uncompressed data, and unaware it comes from a compressed folder (or drive).
So, in these cases FreeFileSync (FFS) will sync such files in their uncompressed form.

Conversely, if you zip a file (or 7zip, or rar, or ...) the file system simply presents such file as is (in compressed form) to the user or tool, and consequently FFS will sync it in compressed form.

[Zenju]

As a result, the user, or any software tool reading such a file is simply presented with the uncompressed data, and unaware it comes from a compressed folder (or drive).
So, in these cases FreeFileSync (FFS) will sync such files in their uncompressed form.

That's not correct. FreeFileSync detects compressed files and copies them without uncompressing. Same for sparse, and encrypted files.

The issue here is that NTFS compressed attribute can be inherited. Most likely the parent folder has the compressed attribute on the source side but not on the target folder.
The source folder was not copied with FFS, otherwise the compressed attribute on the folder would have been copied over.

[therube]

Most likely the parent folder has the compressed attribute on the source side but not on the target folder.

I don't see that (Compression pertaining to a directory)?
On my end, NTFS, I'm seeing (source) DI, D=directory, I=not indexed.

Within said source (directory), I compressed a file (1 of 4 files).
(I did NOT use Compact to do the compression).

That said, the directory does show the file to be Compacted.
As does Salamander & Everything.

C:\out\Len2>COMPACT

 Listing C:\out\Len2\
 New files added to this directory will not be compressed.

   602438 :    602438 = 1.0 to 1   command_000 - Copy - Copy.avi
   602438 :    602438 = 1.0 to 1   command_000 - Copy.avi
   602438 :    602438 = 1.0 to 1   command_000.avi
 30140012 :  30140012 = 1.0 to 1   photo202 (with exif date take).tif

Of 4 files within 1 directories
0 are compressed and 4 are not compressed.
31,947,326 total bytes of data are stored in 31,947,326 bytes.
The compression ratio is 1.0 to 1.


Based on how I did above, the target directory (itself) did not show as compressed, nor did the "Compressed" file. Not in Salamander, nor Everything, nor Compact.

C:\out\Len2>COMPACT

 Listing C:\out\Len2\
 New files added to this directory will not be compressed.

   602438 :    602438 = 1.0 to 1   command_000 - Copy - Copy.avi
   602438 :    602438 = 1.0 to 1   command_000 - Copy.avi
   602438 :    602438 = 1.0 to 1   command_000.avi
 30140012 :  30140012 = 1.0 to 1   photo202 (with exif date take).tif

Of 4 files within 1 directories
0 are compressed and 4 are not compressed.
31,947,326 total bytes of data are stored in 31,947,326 bytes.
The compression ratio is 1.0 to 1.


> 0 are compressed and 4 are not compressed.

Now, if & "uncompress" the file, & use Compact to compress it:

> COMPACT /C *.tif

Again the source file shows as Compressed.
The source directory still does not show as Compressed (best I can tell).

And if I let FFS create the target, still the target directory/files do NOT show as compressed?

Compact itself says (on Win7):

COMPACT [/C | /U] [/S[:dir]] [/A] [/I] [/F] [/Q] [filename [...]]

  /C        Compresses the specified files.  Directories will be marked
            so that files added afterward will be compressed.


So presumably said target directory should have been "marked", oh - afterwards.

Still no good?

I don't know Compact, nor compression (like this at least).
And not getting anywhere?
All attempts, so far, have resulted in uncompressed Target.

Last attempt
emptied source directory
created a zero-byte file, zero
compact'd that - compact /c zero - which works & presumably set source directory to "compressed" - but not that i have any way to verify?

then copied a new file into said source directory - which copied uncompressed?
but given ? that the directory itself was "compressed" then (seemingly) a newly added file into that directory should have also been compressed - automatically (if i'm understanding correctly) - but it was not?

manually Compact (now) 3 files in source directory (1 existing, 1 copied in, 1 created via DIR > xxx), & they are, & then copy that directory with FFS, letting FFS create the target, again the files are uncompressed?


If I attempt the same In Salamander, telling it to Preserve the Attributes, the target files are Compressed.


So, if this is to work ? presumably have to figure out how to set C attribute for the directory itself?
Otherwise, I'm out of ideas?


Wonder if the Volume itself needs to be "compressed" in order for directories within to end up being "compressed"? (It is not on my end, & I'm not going to try.)
.

C is NOT compressed.png (9.39 KiB) Viewed 580 times

Oh, so that is how you do it - compress a folder:
.

And once a folder is compressed, newly created files within it are also automatically compressed.
And the folder itself does then have a C (compressed) attribute.
(Now how to do that from a command line?)

(Now how to do that from a command line?)

You have to use the /S option.

  /S        Performs the specified operation on files in the given
            directory and all subdirectories.  Default "dir" is the
            current directory.


So something like:
> Compat /C /S Len/LENLEN

would compress LENLEN (directory) & all files/folders within LENLEN.
(That's kind of awkard, IMO.)

Anyhow, once you have compress directories set up, a FFS made copy then does copy & maintain the compressed file/directory attributes.


So, if I'm (now) getting it, FFS will copy /directories/ that have the compressed attribute set on it, but will not copy source files that are individually compressed - if the source directory does not have the C attribute set.