FreeFileSync v6.0 crash

[bennieboj]

Hi,

I'm using FreeFileSync Portable for a long time.
I have a problem syncing my files. The program crashes whenever it has to copy the files. I can compare them, the program gives a list of differences, all is good.
When I try to sync, the program stops responding and windows throws me an error "Folder Comparison and Synchronisation doesn't work anymore". Then it shows some log files that I included.
This happens when I use the portable version, the non portable version, batch job and no batch job.

I've already had this problem once before, when v 5.22 came out.
I just got my new laptop and I thought the new program was the cause, not the laptop.
I searched the sourceforge forums and found a debug build here: viewtopic.php?t=1416.
Both builds (opt and no-opt) worked fine, so I copied the opt exe and went on using that.
Now however I accidently updated the program, and the debug builds aren't available anymore, so the problem needs to be solved :D
I think my laptop handles some things differently than my older one, which makes the program crash.

Can you perhaps give me a debug build?

I added the log files (1 shows garbage in notepad++, I didn't post it) and a DXDIAG:
part of DXDIAG: http://pastebin.com/PASvmcaE
Log 1: http://pastebin.com/7eMtV2hR
Log 2: http://pastebin.com/6f1wsKCJ

[Zenju]

Hi Bennieboj,

this could indeed be a similar problem like discussed in the link above. Can you give some info about the CPU of your laptop? It be a great help if you could post a screenshot of the first tab page that the CPU-Z tool displays. In particular I'm interested if the Cpu supports AVX2.

I've also created a first debug build:
[404, Invalid URL: http://freefilesync.sourceforge.net/FFS_6.1_test.rar]

If it crashes again, I'm interested in the crash details like in your "Log 2". Additionally, don't close the crash dialog yet, but try to create a minidump (start Process Explorer, right-click on the crashed FreeFileSync_x64.exe and select context menu option "create dump", full dump provides for better debugging, it the file's too large a "mini dump" may be fine), then send me this file via email.
This should give me a good idea about what's triggering the crash.

Regards, Zenju

[bennieboj]

Hi Zenju,

I tested syncing again with the debug build, using test data.
It crashed again, here are all the logs and the CPU-Z screenshot:

So my Intel Core i7 4700 MQ supports AVX2 :)
WER8AA3.tmp.appcompat.txt: http://pastebin.com/ehqwXx7G
WER737A.tmp.WERInternalMetadata.xml: http://pastebin.com/5BtRJseH
I zipped the mini dump + the one windows provided and added them to the post, some characters in the file aren't visible on pastebin (mostly NULL values, but still now you have the whole file).
The full dump was kinda big (145 MB), so I couldn't add it to the post.
I zipped it and uploaded it here (48.3 MB): [404, Invalid URL: http://www.4shared.com/zip/fByh_wDX/FreeFileSync_x64_full.html]

Thanks for the fast response!
I love devs like this :)

Bennieboj

[Attachment was removed! File name: logs_2013_12_07.zip (794692 bytes)]

[Zenju]

Very nice, thanks for the detailed infos! All three dump files point the the same location within a system math function, but I'm not yet sure what is happening. The last debug test was with all optimization enabled, so I can't fully trust the stack data contained. Therefore I have created a new version without optimization. If you can still reproduce the crash with the following version, this should greatly simplify analysis.

[404, Invalid URL: http://freefilesync.sourceforge.net/FFS_6.1_test2.rar]

It also contains code to automatically write a .dmp file in case of certain app crashes. *If* it works, please send me the dump file, too. In any case just to be sure send again a "full" dump file, since dmp files are not 100% reliable and can easily get corrupted. So the more, the better...

[bennieboj]

I still got a crash.
I didn't find a *.dmp file using windows search in the freefilesync directory, so I assume the program didn't produce one.
Full dump is here (37.3 MB): [404, Invalid URL: http://www.4shared.com/zip/NVVEE8cj/FreeFileSync_x64_full__1_.html]
The other dumps and log files are in the attachment.

[Zenju]

Thanks again! This is a tough one. I've closely analyzed the crash location, but everything looks perfectly fine. I'm even able to reproduce the very same math call (std::log) with the bit-wise identical 64-bit input on my machine, which is identical to yours except that it's an Intel Haswell Core i5, without any crash or sign of problems.

The exception code c000041d which presumely complains about an unhandled exception in a user callback is also puzzling, since there are no known user callbacks during the call to std::log. Except maybe... I have a far-fetched hunch, lacking a better idea right now, let's see if the _matherr callback gets hit.

[404, Invalid URL: http://freefilesync.sourceforge.net/FFS_6.1_test3.rar]

Ideally, this will write a "exception.dmp" file when it sees problems before the actual crash. However make sure you run FreeFileSync with admin rights, since the dmp file will be written in the working directory, which may be located in "C:\program files\FreeFileSync" and that's why it needs admin access to write a file there.
Again I'm also interested in the full dump in any case.

[Zenju]

PS: another far-fetched idea: Some sources on Google indicate that STATUS_FATAL_USER_CALLBACK_EXCEPTION (0xc000041d) could be related to broken system files. So a "sfc.exe /scannow" probably won't hurt...

[bennieboj]

FreeFileSync is installed in %USERPROFILE%\Desktop\FFS\FreeFileSyncPortable\App\FreeFileSync so no admin rights are needed, ran once with and once without, no exception.dmp or any *.dmp found anywhere.
Full dump is here (37.38 MB): [404, Invalid URL: http://www.4shared.com/zip/rflgDK8i/FreeFileSync_x64_full__2_.html]
The other dumps and log files are in the attachment, like before.

I'll try "sfc.exe /scannow" soon.
EDIT: ran the command:
Windows resource protection did not find any integrity violations.
Or something like that in dutch, sounds good to me.

[Attachment was removed! File name: logs_2013_12_07_3.zip (714741 bytes)]

[Zenju]

It seems you've attached the old dmp and log files: they've the time stamp of 19:18pm.

[bennieboj]

Oops, I'll add them in a few hours, I'm not on my laptop at the moment.

[peanutlasko]

I too am also seeing crashing with v6.0 where I didn't in the past - not sure if this is related or not.

Attached are all crash dumps I've gotten so far. Sorry if these are not related.

[Attachment was removed! File name: CrashDumps.zip (2913532 bytes)]

[peanutlasko]

Here is a SS of CPU-Z

[bennieboj]

I added the correct files to the post above.
I also updated the dump file in the link, same link different zip file.

[Zenju]

I too am also seeing crashing with v6.0 where I didn't in the past - not sure if this is related or not.

Attached are all crash dumps I've gotten so far. Sorry if these are not related.

Are you able to reproduce the crash with the FFS_6.1_test5.rar version below? If so please attach the corresponding dmp file. The FreeFileSync executable and the dmp files need to match, otherwise it's debugging in assembler and that's "a little" difficult.
Also let me know how you created the dump file: 1. created by WER, 2. automatically by the test3 version into the working directory 3. manually via Process Explorer

Technically your crash is unrelated to the crash at the beginning of the topic. Its crash code is EXCEPTION_STACK_OVERFLOW (0xc00000fd). I'm also seeing a lot of threads. What are you doing while the crash occurs? E.g. are you scanning a lot of large and deep directory hierarchies?

[Zenju]

I added the correct files to the post above.
I also updated the dump file in the link, same link different zip file.

Thanks, but the "FreeFileSync_x64_full__2_.html" link is broken.

PS: I see that you are using Avast. I've seen app crashes in FFS bevore that were caused by a bug in the Avast drivers. Are you somehow able to test FreeFileSync without avast and see if the crash still occurs?

[bennieboj]

Ok, third time's the charm: [404, Invalid URL: http://www.4shared.com/zip/mv0LC_I4/FreeFileSync_x64_full__2_.html]

A bug in avast may be possible, I'll try to disable avast and report back to you.
EDIT:
I disabled avast. I think that's about the same as uninstalling it completely, I'm not going through the process of uninstalling my antivirus right now if it's not necessary.
Freefilesync still crashes when avast is disabled, FFS ran with admin rights and all, I added the logs.
[404, Invalid URL: http://www.4shared.com/rar/Zn1PcgdI/FreeFileSync_x64_no_avast_full.html]

[Attachment was removed! File name: logs_2013_12_08_no_avast.rar (552040 bytes)]

[Zenju]

Ok, third time's the charm: [404, Invalid URL: http://www.4shared.com/zip/mv0LC_I4/FreeFileSync_x64_full__2_.html]

A bug in avast may be possible, I'll try to disable avast and report back to you.
EDIT:
I disabled avast. I think that's about the same as uninstalling it completely, I'm not going through the process of uninstalling my antivirus right now if it's not necessary.
Freefilesync still crashes when avast is disabled, FFS ran with admin rights and all, I added the logs.
[404, Invalid URL: http://www.4shared.com/rar/Zn1PcgdI/FreeFileSync_x64_no_avast_full.html]

Great, the dmp file works! There's another thing you can test: The crash happens while executing AVX instructions and it seems there are a few bugs in VS for x64 and AVX. The command below and a reboot will turn off AVX support.

bcdedit /set xsavedisable 1

Let me know if the crash still happens with AVX disabled. After this you can enable it again with

bcdedit /set xsavedisable 0

and a reboot.

[bennieboj]

I just ran with AVX disabled, same result.
[404, Invalid URL: http://www.4shared.com/rar/y2QtXD9D/FreeFileSync_x64_no_avx_full.html]

[Attachment was removed! File name: logs_2013_12_08_no_avx.rar (542458 bytes)]

[Zenju]

I don't understand how an "An unhandled exception was encountered during a user callback" exception is possible while at the same time the unhandled exception filter is set and supposed to automatically write a .dmp file. Unless... writing the dmp file fails for some reason. I've created yet another version for testing. This time it will block showing message boxes upon seeing an unhandled exception, and report success or failure when writing the .dmp file also via a messagebox. So, you should either get a .dmp file created automatically, or should see an error message in a popup before the crash:

[404, Invalid URL: http://freefilesync.sourceforge.net/FFS_6.1_test4.rar]

[bennieboj]

I don't see any dump, I searched my whole SSD for "*.dmp" and also "*.*dmp*", the only dump files that are created in the past few days are the ones I copied from the temp folder (the ones windows generates) and the ones I made myself (using process explorer).
What is the path of the dmp file, relative to the exes? Is it next to the FreeFileSync.exe or next to the FreeFileSync_x64.exe? Either way, no dmp files are found on my SSD.
I also don't see any message boxes...

No dump file means there is an unhandled exception but I don't see any error boxes.
So I assume there are 2 things that could be wrong:
1) the dmp file cannot be written, which is weird, because all the files are inside a FFS folder on my desktop and all the other files (like a .lock file) can be written.
2) there is an unhandled exception but the program crashes before it can show the error dialog.

Log: [404, Invalid URL: http://www.4shared.com/rar/zQTsGVS5/FreeFileSync_x64_4_full.html]

[Attachment was removed! File name: logs_2013_12_08_4.rar (544660 bytes)]

[Zenju]

If you did not see any message box which tells where the .dmp file was written to, this means that the unhandled exception filter was not called, so there hasn't been any uncaught exception. This however is at odds with the exception code 0xc000041d.

A second error that happened after whatever it is that is causing the original error could be an explanation. However this is all Windows internal code that is frequently used, so it's unlikely to be a Windows bug. Except there were some general data corruption happening in the program to trigger abnormal behavior... however the .dmp file status looks all perfectly valid. If the program did not visibly crash, I'd say the dmp file indicates normal execution.

The other remaining possibility is that there is no uncaught exception in first place. This fits with all other observable behavior, so it is the more likely hypothesis. However it fails to explain what else caused "0xc000041d", which then would be at least a bug of giving misleading error codes in Windows error reporting - but that's not entirely unlikely.

I'll continue analysis. If nothing else helps, we can try brute force: Create a minimal sample that still exhibits the crash behavior by "binary search".

[Zenju]

Hi Bennieboj,

I tested "bcdedit /set xsavedisable 1" on my machine and lo and behold: I'm able to reproduce your crash when running the FFS sync!!!

We have two bugs here:

1. The AVX2 instruction "vpsrlq" crashes on your machine. It's not clear why this is the case since the CPU is marked as having AVX2 support, but it looks like it is disabled for yet unknown reason. But anyway, the VC x64 compiler should implement a fallback if a CPU feature is not present, and that is not happening. So we have a compiler code generation bug. Therfore I've opened the following connect ticket:

https://connect.microsoft.com/VisualStudio/feedback/details/811093/visual-studio-2013-rtm-c-x64-code-generation-bug-for-avx2-instructions

2. Instead of "0xC000001D: Illegal Instruction" we're seeing "0xc000041d - An unhandled exception was encountered during a user callback". The latter is somehow triggered by the former. I have yet to investigate why this is the case. This bug severly complicated troubleshooting so far by leading into wrong directions.

[bennieboj]

I think I found the "unknown" reason:
http://www.drdobbs.com/tools/intel-avx2-will-bring-integer-instructio/231000372
On this link I found that AVX and AVX2 require Windows 7 SP1, which didn't have installed yet, because I'm on low disk space atm, so that's why AVX/AVX2 is "disabled".
I'm going to do the updates now, just to see if it fixes the problem, I'll let you know if it works.
EDIT: it all works now!
Thanks a lot for your time and efforts! If you need me to do anything, just let me know :)

The MSVS2013 compiler does something wrong in the backend, not producing a fall back I mean. I know that's not something you can fix, you can only submit the ticket and hope things get fixed. So, you discovered a bug because I did something stupid.

Somehow the "Illegal instruction" isn't shown, but we see "An unhandled exception was encountered during a user callback", if this expected behaviour there should be a way to handle the exception. I hope you can find out how to handle it, or find out why it's not showing the first exception.

[Zenju]

That's excellent! :) Also big thanks to you for not giving up after the initial bug submission and having the patience of testing and sending all these crash dumps!

Updating the SP1 isn't a real fix, but at least a feasible workaround until Microsoft decides to fix the compiler bug - which they might not do if they decide that having Windows 7 without SP1 is not a supported scenario.

The only remaining question is why the unhandled exception filter was seemingly not called before the crash dialog was shown with exception code 0xc000041d. In my tests I was able to reproduce a number of combinations of 0xC000001D followed by 0xc000041d, or only 0xc000041d, but in all cases the crash dmp file was written.
But all my tests were on Win 7 SP1, this alone could explain difference in behavior considering that SP1 brings certain improvements for Windows crash behavior.
Anyway, my unhandled exception filter code became much more robust during the cause of these tests, so next time I'll likely require less testing since all known corner cases of potential abnormal behavior during structured exception handling should be closed.

Thanks again!