Shadow copy problems

[infonetsp]

First things first, thanks for a great application, we use it everyday.

Our problem:

When making a backup in certain computers whose users don't have administrator
rights we always get a Shadow engine error:

"The caller does not have sufficient backup privileges or is not an
administrator. (0x80070005)"

That's ok for us, since the user doesn't have enough privileges shadow copy is
not enabled for him, but why that locked file gets deleted on the
destination folder? Why not leaving it alone and showing the error on the
log file? that way the file (an Outlook .pst one) gets lost forever. Is there
an option in the FreeFileSync GUI to avoid this behaviour?.

In essence, we'd like to be notified about the file being in execution and thus Shadow Copy not being able to make the copy of the file but, we don't want it to be deleted on the destination folder.

Thanks in advance for your help, and sorry if this has been addressed before.

Best regards,

Infonet.

[Zenju]

The reason why the target file gets deleted even when the source file is
locked lies in the order FFS handles overwriting:
First the target file gets deleted, second the source file is copied to the
target location.
The second step fails, leaving the target still deleted. Main reason for using
this order is preventing disk space bottlenecks if some large file is
overwritten. The reverse order, i.e. first copying the new file, then deleting
the old one would create a peak in disk space consumption of twice the large
file's size.

[infonetsp]

Ok, thanks for the answer.

Then I assume there is no solution to this problem? Maybe executing
FreeFileSync as an admin user? any ideas?

[Zenju]

> executing FreeFileSync as an admin user
Well, this definitively would avoid the issue in first place by allowing
Shadow Copies of the locked file.

> solution to this problem? any ideas?
A workaround: If these locked files are not relevant for synchronization,
excluding them via file name filtering.
As for the current "file overwrite" approach I'm not sure if a better
algorithm exists (that doesn't introduce new drawbacks)

[infonetsp]

Thanks again, Zenju.

I've been able to advance a little more. The copy task is executed under an
administrator account and bypasses the error :

"The caller does not have sufficient backup privileges or is not an administrator."

But now I get this one:

"Error calling \"ppWriteMetaData->Wait\."

I've checked that the Shadow service is set to Automatic.

Can you provide any help with this?

[Zenju]

This seems to be an old FFS error message. What version of FFS are you using?
What is the full error message, including windows error text and code?

[infonetsp]

The version is 3.8, dated the 20th of June of 2010.

And the log is here (In Spanish):



FreeFileSync - Ejecuci├│n batch (Fecha: 05/01/2011 Hora: 11:38:39)
-----------------------------------------------------------------

Registro de mensajes:
---------------------
[11:38:39] Iniciar

[11:38:54] Info: Procesar un par de carpetas:
Izquierda: "C:\Documents and Settings\correa\Mis documentos\"
Derecha: "n:\Mis Documentos\"
[11:38:54] Info: Copiar archivo "Outlook.pst" a
"n:\Mis Documentos\Mis documentos\Correo"
[11:38:55] Error: ¡Error al copiar archivo bloqueado "C:\Documents and Settings\correa\Mis documentos\Mis documentos\Correo\Outlook.pst"!
¡Error al iniciar el servicio "Volume Shadow Copy"!
(Error calling "ppWriteMetaData->Wait". (0x80042318: IDispatch error #8472) Volume: "C:\")
[11:38:55] Info: Copiar archivo "outlook.pst" a
"n:\Mis Documentos\Mis documentos\Documentos Pepe Correa\correO"
[11:38:55] Error: ¡Error al copiar archivo bloqueado "C:\Documents and Settings\correa\Mis documentos\Mis documentos\Documentos Pepe Correa\correO\outlook.pst"!
¡Error al iniciar el servicio "Volume Shadow Copy"!
(Error calling "ppWriteMetaData->Wait". (0x80042318: IDispatch error #8472) Volume: "C:\")
[11:38:55] Atención: ¡Sincronización completada con errores!

[11:38:55] Detener (Tiempo total: 00:00:15)

[Zenju]

This error is not related to the FFS implementation of VSS but a generall VSS
issue. Googling for "0x80042318" revealed a number of forum posts.
Error code 0x80042318 has constant name VSS_E_WRITER_INFRASTRUCTURE.
Some posts suggest http://support.microsoft.com/kb/907574 were a solution.

[infonetsp]

Thanks again, will take a look at that suggestion.

[Anonymous]

> First the target file gets deleted, second the source file is copied to the
target location. The second step fails, leaving the target still deleted.



You said FFS uses CopyFileEx function (which supports cancelling). Would it be
possible to alter FFS algorithm of copying only locked files to:
1. Start copying source file with an temporary source name
2. If step 1 suceeds
2.1 cancel copying to temporary file and delete temporary file
2.2 delete destination file
2.3 start copying source file with actual destination name


> Main reason for using this order is preventing disk space bottlenecks if
some large file is overwritten.

It would be great if the "overwrite behavior" was an sync job option:
1. in many situations the free disk space on target destination is indeed small. The current behavior is preferred.

2. in many situations it is important to have at least one copy of the data at all times (Outlook .pst file). Then it would be great if new file was copied before old is deleted.

If your time permits, I would like to ask you to make the copy behavior an
option.

Thanks for the great program!

[Zenju]

I've thought about this issue and came to the conclusion, that having a file
deleted as a result of a failed overwrite operation is not really acceptable.
On the other hand, changing the order of steps is neither a solution and
exhibits the inverse problem, namely copying a temp file succeeds, but
deleting the old file fails due to some lock issue. Additionally peak disk
space consumption is worse than with the order currently implemented.
Therefore v3.14 will have the following solution: Order of steps stays like it
is right now. An additional check is included before overwiting starts that
reads a single byte from source file. Lock issues or any other reading issues
will show up and break with an error message. This check will most likely
catch 90% of issues (if not more) up front, which is sufficient.
The only possible drawback is a potential performance issue: Tests on SSD,
HDD, and USB flash show additional costs of 0ms, 43 microseconds and 1 ms per
file in that order. This is also acceptable.

[Anonymous]

Thank You very much! :)

[Zenju]

Unfortunately as turned out with v.314 checking read access before starting
"copy and delete" is flawed: If the operation backs out, it never get's the
chance to start up VSS later during file copy. This leaves VSS broken in case
of overwriting files.

v3.15 will solve this issue perfectly IMNSHO ;)

It turns out that the steps 1. "checking read access", 2. "deleting target"
and 3. "copying source to target" cannot be reordered without introducing a
problem of some kind: either VSS is broken or transactional behavior.

v3.15 solves it by applying a more granular strategy introducing the distinct
concept of "overwriting" a file (instead of emulating it by delete + copy):
1. First source file is copied to a .ffs_tmp file (this is done already, so nothing new)
2. If read access fails for some reason, e.g. source file locked, operation is restarted using VSS
If nothing went wrong, overwriting is complete except the rename operation!!
3. This (and not earlier) is the place where target will get deleted.
4. Temp file is renamed to final target

Looking at the new workflow it is almost perfectly transactional! The only
theoretical problem could occur if 3 succeeds, while 4 fails. But 4 is a very
small and fast operation and it is already clear by the success of 1. that
target is fully writable. This practically excludes this possiblitiy.

Conclusion: v3.15 will be "transactional to the bone". The last gap has been
closed, without sacrificing funcionality of any kind! and without sacrificing
performance in any kind! I'm more than happy about this result.

[infonetsp]

Glad to hear that.

It really was a problem for us that now seems to be solved.

Thanks.