Copy files that contain "~"

[basilep]

It looks that does not copy files that contain in their file name a "~"

[Zenju]

Could you give an example of a filename that cannot be copied? Are you talking
about the Windows or Linux release?

[basilep]

I am taking about Windows version, latest release of FFS, downloaded
yesterday.
When I tried to backup (copy files from one HD to another) yesterday, I
realized that there is a difference in the number of files between the source
and the destination hard drives. (In fact I tried just a folder, as a test).
When I tried again to backup, all files with "~" a message appeared as the
files existing in the destination HD and that cannot be copied, althought they
were not in the destination folder.. I believe the file names must be from old
DOS file names, when transferred as eight characters, six letters ~ and a
number.
Can you dublicate the problem from the above?

This evening, I started the copy again and indicates two days to finish the
transfer of about 160GB from one HD to another,
I can try specific case if needed.

Thanks,
B.

[Zenju]

> existing in the destination HD and that cannot be copied, althought they
were not
The message actually is correct. The names containing '~' are just aliases for
the full file names that are already existing on the other side. (In
commandline enter "dir /x" to get the 8.3 name of the files)

[Zenju]

But the question is why these files did not show up during comparison. Are you
able to see these files with explorer?

[basilep]

I understand what you say
But,
There is a difference in the number of files as reported by Windows Explorer,
and one of them was a directory with a ~ in its file name that did not existed
in the target HD..

If you cannot duplicate it I shall try something specific when finishes the
current job.

Thanks,
B.

[Zenju]

>If you cannot duplicate it I shall try something specific
Please do so.

>and one of them was a directory with a ~ in its file name that did not
existed in the target HD..
But maybe this directory did exist on target under another name?
FreeFileSync operates at very low Windows API level. This is one layer beyond
the windows shell that Explorer is using. If the message says the files are
already existing, then this is true for the file system (even if explorer
might see things differently in rare cases)

[basilep]

Hello,

I had to stop the backup.and start again.
(What I am doing is copying the files from one drive that is filled to another
bigger drive.)

The first time this error appeared (I took a photograph of the error window,
too):
--------------------------------------------------------------------------------------
Error
Error copying file
"H:\0 Temporary\CHRIST~1.DOC" ->
"F:\0 Temporary\CHRIST~1.DOC"

Target file already existing!

Ignore this error, retry, or abort
sychronization?

O Ignore subsiquent errors

Ignore Retry Abort

--------------------------------------------------------------------------

With Windows explorer, I can see the file in the source HD (H:) but is not in
the destination HD (F:)
The files before ant after the "CHRIST~1.DOC" are in both the source and
target HDs.

I selected the "Ignore subsequent errors to avoid delays. Now its is copying
and expected to finish in two days.

Thank you,
B.

[Zenju]

>With Windows explorer, I can see the file in the source HD (H:) but is not in
the destination
Explorer shows file's long names only. Please have a look at the target
directory from console and test what "dir /x" gives.

[basilep]

"...console and test what "dir /x" gives..."
Do you mean "C:\>dir /x" The DOS window?

But I can see "CHRIST~1.DOC" on the source drive with Windows explorer.

Why you say that "Explorer shows file's long names only." ?

In addition I can see the difference in the count of files reported from
Explorer for the source and target HDs.

Can you duplicate the situation?

Thanks,
B.

[Zenju]

>Do you mean "C:\>dir /x" The DOS window?
Yes

>Why you say that "Explorer shows file's long names only." ?
That's the behavior of the Explorer. All files and directories both have long
and a short path name.
Example:
Long: C:\Program Files
Short: C:\PROGRA~1

>Can you duplicate the situation?
No, but I can only make assumptions from what you describe. Right now it
seems, there are two files that have the same short name, but different long
names, which likely is the reason for the confusion.

[basilep]

I went to the computer.

Using Explorer I did a search *~*.*.
I got 10 files in the source directory and 8 in the destination directory. One
of the missing 2 is the CHRIST~1.DOC (a 1996 file).
So, I can see the files in Explorer with the ~ in their file names, the old
DOS 8.3 convention, but two.

How I can know if two files have the same short file name?

Thanks,
B.

[Zenju]

>Using Explorer I did a search *~*.*.
I fear using explorer to check whether folders are identical won't help here
but introduce confusion instead.

The best thing to do would be to go into "F:\0 Temporary" and check wheter a
file with short name "CHRIST~1.DOC" exists via console: "dir /x".

[Zenju]

Maybe some thing you could clarifiy, because that's what I think is happening:

There are two files:
"C:\Source\Filena~1" and
"C:\Target\FilenameRight" which have short names

"C:\Source\Filena~1" and
"C:\Target\Filena~1"

When comparing via explorer it seems that there is one file that is existing
on left side only, while another file is existing on right side only. But
whenever you try to copy the left file to the right side you'll get a "already
existing" error, because the file system doesn't care about their long name
(only) but also about their short names which happen to be identical.

The observable behavior is this:
When you copy a filename with a name longer than 11 characters to another
directory it will scan the target directory an assign this filename a
shortname that is unique: For example, if target already has another file with
shortname Filena~1, this filename will get "Filena~2" as it's shortname => no
collision, just as we wish

BUT (and this is the issue here) if the source filename has less-equal 11
characters, for example it's LONG name is already in format "Filena~1", then
this "trying to find a unique short name in target directory algorithm" is NOT
executed. Instead it bluntly tries to copy it and finds another (totally
unrelated) file which happens to have the same short name. -> file already
exising error

The behaviour described above is easy to reproduce (just tested on Windows 7,
x64). To me it seems to be a bug in the NTFS file system design, but who
knows, maybe this is meant to be a feature nevertheless. Anyway this should be
the explanation.

[basilep]

Hello,

I think you are right. At the target drive, F:
At the F:\> dir CHRIST~1.DOC
I get
F:\>Christians Observe Easter

which has the same 8.3 file name as the file "CHRIST~1.DOC" which is not
copied afterwards.


Does this help?

What is the proposed solution is any?
Of course, I can go back and change all the old leftover *~*.* filenames to
long filenames manualy!!!

Thanks,
B.

Which ma

[Zenju]

>Does this help?
Sure, this explains the observed behavior.

>What is the proposed solution is any?
- Generally I'd consider this a bug in the NTFS design: If a file is named "Filena~1" it cannot have a deviating shortname (e.g. Filena~2). I think this is not correct. But I may grant that there were reasons for this decision. After all retaining backwards compatibility to MS-DOS filename specifications is not an easy thing. Nevertheless, I somehow sense Microsoft won't do anything about it ;)

- Renaming the files will "fix" the issue. It's probably the only feasible solution at all.

[basilep]

Hello,

Thank you very much for your time and help.

I shall rename all these files and folders (I thing they are Windows 3.1 files
and I was using a "Long File Name" program that will create the 8.3 file name
but you will see a long file name).
I guess I can say it is time to give them meaningful names.

Thank you,
B

[Zenju]

> are Windows 3.1 files it is time to give them meaningful names
Yes, after all this time, I think they deserve it ;)

[Zenju]

FYI: I did some further tests and found out that this bug seems to be new for
Windows 7. For example I couldn't reproduce it with Windows XP XP3.
Anyway I've solved this problem for v3.7, by using a different approach when
copying this type of files.

[basilep]

Ok, I shall try 3.7 when available. Currently I am using 3.4.
Note that the computer I am using to transfer the files is Windows 2000.
Both Source and destination HD are NTFS formatted

B..

[Zenju]

If you're interested in testing this specific feature, here you go:

[404, Invalid URL: http://www.file-upload.net/download-2400676/FreeFileSync_v3.7_alpha.rar.html]

[basilep]

Hello,

I tested this specific feature and looks it works.
(Specifically, I created a file "CRIST~1.DOC" at the same place as before. The
synchronization found the one file difference and copied it in the destination
drive.)

Now, comparing the Source HD and the Destination HD, using Windows 2000
Explorer, the source shows 189 GB while the destination shows 187 GB. The
FreeFileSync shows the same number of files in both HD. Both HD are NTFS
formatted.

Thanks,
B.

[Zenju]

>I tested this specific feature and looks it works.
Good to hear!

>189 GB while the destination shows 187 GB.
That's probably due to Recycle Bin or other excluded folders.

[basilep]

Hello,

The recycling bin is empty and both Drives contain data only. (They are used
as data storage external drives).
Any idea for the difference?

Thanks,
B.

[Zenju]

>Any idea for the difference?
No, but probably the best and only way to find out is to compare both sides in
detail via Explorer: First compare sizes of top level folders. If then
differences between both sides exist navigate one level down and repeat the
steps. Sooner or later you'll know where the difference originates.

[basilep]

Hello,

Do you exclude any files. If "Yes" what is their extension. Perhaps will be
faster by try to locate them in both drives?

Thanks,
B.

[Zenju]

The only files that are excluded implicitly are the database files
"sync.ffs_db", "sync.x64.ffs_db".

Other directories that are excluded explicitly via filter:

\System Volume Information\
\RECYCLER\
\RECYCLED\
\$Recycle.Bin\

> Perhaps will be faster
If you compare both sides beginning with high-level folders, this should be
pretty fast. You can also compare sets of folders an find the difference
analog to a binary-search.

PS: There are some files that can be found with FFS but not with Explorer, for
example the "*assembly*" files in the Windows directory. So in general,
FreeFileSync should never find less than what Explorer shows (except filtered
files of course)

[basilep]

Hello,

I shall check for size difference latter.

Now I was doing another backup, using Windows XP, and FFS version 3.6.
and got:
-----------------------------------------------------------
Warning: Synchronization failed for 1 item(s):

Error: Error copying file:
"C:\Documents and Settings\Administrator\My Documents\My Documents
Lily's\Shared_Documents\STEWARDSHIP\2009_Stewardship_CD\C. Bi-Fold Appeal and
Handout\Bi-Fold and Handout_Professional Printer Versions_Stewardship
20\STEWAR~1.PDF" ->
"E:\Backup_2010-04-04\My Documents
Lily's\Shared_Documents\STEWARDSHIP\2009_Stewardship_CD\C. Bi-Fold Appeal and
Handout\Bi-Fold and Handout_Professional Printer Versions_Stewardship
20\STEWAR~1.PDF"
Target file already existing!

Warning: Synchronization completed with errors!
You may try to synchronize remaining items again (WITHOUT having to re-
compare)!
--------------------------------------------------------------------------

The three file names in both source and destination HD are:

Stewardship Handout-09-PROFF-IMPOSED.pdf
Stewardship Handout-09-PROFF-SINGLE-PAGS.pdf
STEWARDSHIP PROGRAM.PDF


Thank you,
B.

[Zenju]

>another backup, using Windows XP, and FFS version 3.6.
With FreeFileSync_v3.7_alpha this should work now.

[basilep]

Yes, but what does version 3.6 tries to copy?
Both the source and destination HD have the same number of files. Where does
it sees this "STEWAR~1.PDF" file?

Thanks,
B.