MTP Sync Error - "Unexpected size of data stream"

[cpearless]

Using FFS 7.6, under Win 10 Pro x64, whenever I attempt to run a sync from an MTP source, to any desitination, I get the following error (screenshot attached):


> Cannot read file "mtp:\XXXXXXXXXX\Internal storage\Pictures\XXXXXXXX\XXXXXXXXXXX.jpg".
>
> Unexpected size of data stream.
> Expected: 843831 bytes
> Actual: 844085 bytes


This error occurs for every new file, I can copy the file manually using explorer no problem at all.

Any help or pointers would be greatly appreciated.

Cheers

[Zenju]

That's an issue of the MTP device: the WPD_OBJECT_SIZE property of the jpg file is set to "843831", but when actually opening the file stream it returns with "844085" bytes.

[cpearless]

That's an issue of the MTP device: the WPD_OBJECT_SIZE property of the jpg file is set to "843831", but when actually opening the file stream it returns with "844085" bytes.

Cheers for the response. Copying the file to the machine manually using explorer works fine though? If I then look at the file in detail on the device (via MTP), locally on the device and in Windows the sizes all report correct, the error only occurs in FFS? Even more bizarrely the difference is always eaxctly 5 bytes. For each file that causes errors, FFS always thinks it's 5 bytes larger than it is.

Cheers

[Zenju]

FFS deliberately checks expected and actual size and shows an error if both disagree. Explorer obviously does not do this check.

> Even more bizarrely the difference is always eaxctly 5 bytes. For each file that causes errors, FFS always thinks it's 5 bytes larger than it is.

Let's keep to your first example: What does Explorer show for the size of the jpg file while it is on the device, and what after it was copied manually to a local drive?

[cpearless]

FFS deliberately checks expected and actual size and shows an error if both disagree. Explorer obviously does not do this check.

> Even more bizarrely the difference is always eaxctly 5 bytes. For each file that causes errors, FFS always thinks it's 5 bytes larger than it is.

Let's keep to your first example: What does Explorer show for the size of the jpg file while it is on the device, and what after it was copied manually to a local drive?

I just realised that after making a comment about it being exactly a 5 byte descrepency, my example isn't! Apologies for that! Nonetheless, the reported size from explorer while the file is on the device is 843,831.00 bytes and the size when it's copied to disk is 843,831 bytes.

Screenshots attached.

Cheers for looking at this.

[Zenju]

I've created a version of FFS with the size check disabled. Can you sync the above jpg file to disk with this version and then binary-compare it against the Explorer copy? (or alternatively send me both files for comparison)

[404, Invalid URL: http://www.mediafire.com/download/28yedyje3fsr4rr/FreeFileSync_7.7_no_mtp_size_check_Setup.exe]

[cpearless]

I've created a version of FFS with the size check disabled. Can you sync the above jpg file to disk with this version and then binary-compare it against the Explorer copy? (or alternatively send me both files for comparison)

[404, Invalid URL: http://www.mediafire.com/download/28yedyje3fsr4rr/FreeFileSync_7.7_no_mtp_size_check_Setup.exe]

Just attempted to sync the file again, got a new error:

> Cannot write file "C:\Users\cpearless\Desktop\Instagram\IMG_20151102_131616.jpg.ffs_tmp".
>
> Unexpected size of data stream.
> Expected: 843831 bytes
> Actual: 844085 bytes

I presume it's also performing the same check on write, as well as read?

[Zenju]

Just attempted to sync the file again, got a new error:

> Cannot write file "C:\Users\cpearless\Desktop\Instagram\IMG_20151102_131616.jpg.ffs_tmp".
>
> Unexpected size of data stream.
> Expected: 843831 bytes
> Actual: 844085 bytes

I presume it's also performing the same check on write, as well as read?

Forgot to remove a second check. Just updated the test version above.

[cpearless]

Great, that copied the files just fine. Bizarrely, the files when copied with FFS actually increase in size. The slightly larger files still work just fine though, so there's no major corruption going on. Screenshots and files attached.

Cheers

[Attachment was removed! File name: MTP_Copy-FFS_7-vs-Explorer.zip (1688079 bytes)]

[Zenju]

Thanks for the screenshot of the WPD Info tool. From all the data gathered so far this looks like bug or data corruption of the MTP device:

- Both WPD_OBJECT_SIZE and WPD_RESOURCE_ATTRIBUTE_TOTAL_SIZE of WPD_RESOURCE_DEFAULT report an expected file size of 843831

- However FFS reads 844085 bytes until reaching end of stream.

FreeFileSync's version seems to be the correct one, the file really is 844085 bytes long:

I ran a consistency check on both jpg files and the Explorer version is cut short missing data:

jpeginfo -c *.jpg
Copied with Explorer.jpg 1944 x 1944 24bit Exif N 843831 Premature end of JPEG file [WARNING]
Copied with FFS.jpg 1944 x 1944 24bit Exif N 844085 [OK]

And indeed, visually comparing both images, the Explorer version is missing pixels at the lower right.

Unfortunatly FreeFileSync doesn't have good options to remedy this situation: It could behave like Explorer and only copy WPD_OBJECT_SIZE bytes, but that's deliberately copying corrupt data. Or it could ignore the mismatch and copy the full stream, no matter how long it is. But this may mask other cases of data corruption where WPD_OBJECT_SIZE is correct, but the stream is too short/long. Essentially this is a bug that must be fixed by the manufacturer of the MTP device.

[cpearless]

Thanks for the screenshot of the WPD Info tool. From all the data gathered so far this looks like bug or data corruption of the MTP device:

- Both WPD_OBJECT_SIZE and WPD_RESOURCE_ATTRIBUTE_TOTAL_SIZE of WPD_RESOURCE_DEFAULT report an expected file size of 843831

- However FFS reads 844085 bytes until reaching end of stream.

FreeFileSync's version seems to be the correct one, the file really is 844085 bytes long:

I ran a consistency check on both jpg files and the Explorer version is cut short missing data:

jpeginfo -c *.jpg
Copied with Explorer.jpg 1944 x 1944 24bit Exif N 843831 Premature end of JPEG file [WARNING]
Copied with FFS.jpg 1944 x 1944 24bit Exif N 844085 [OK]

And indeed, visually comparing both images, the Explorer version is missing pixels at the lower right.

Unfortunatly FreeFileSync doesn't have good options to remedy this situation: It could behave like Explorer and only copy WPD_OBJECT_SIZE bytes, but that's deliberately copying corrupt data. Or it could ignore the mismatch and copy the full stream, no matter how long it is. But this may mask other cases of data corruption where WPD_OBJECT_SIZE is correct, but the stream is too short/long. Essentially this is a bug that must be fixed by the manufacturer of the MTP device.

Thanks for looking into this and pointing me in the right direction, it's greatly appreciated.

I'm going to bury myself in the OS of the device and try to find out what's going on.

As always, FFS kicking ass.

Cheers

[Giangi]

Unfortunately MTP is a "crappy protocol", look at this my old comment about MTP implemented on Android: viewtopic.php?t=2166&p=9246#p9246

[asdhhjsdka]

Shit, I run into the same issue with my Nexus 4 with CM12.1. Any chance, this can be fixed by Cyanogenmod Team if I open a bug report?

[Giangi]

That's a very bad news... Tired of all the MTP's *stupidity* I was seriously thinking about rooting my MotoG 2014 and use a CM build... but I guess they didn't touch the sources so the MTP errors *implemented* in the official Google sources are present into CM builds too...

[clausmeister]

Hello,

I'm having the same issue when trying to sync my iPhone's photos and videos to my computer. It works well for pictures, but for videos (*.mp4 and *.mov) I almost always get "unexpected size of data stream" / [notifyUnbufferedRead] errors. These also happen when using explorer. I'm on windows 10, FFS 11.0 (donation edition).

Is there a cmd-line switch or a setting in the GUI to disable the file size checks you mentioned above or is there a chance you could add something like that? I'm also ready to donate for it to happen!!!

The different file sizes are no problem for me as long as the videos work after they've been transferred. Also the risk of having corrupt/incorrect data is something I can live with.

Thanks a lot in advance, cheers

[Snoopy354]

Hello clausmeister,

Have you got an answer, I have the same Problem with Apple IPhone.

[ArtM7]

I am having the same error problem - Size of Data Stream,
updating my Windows 7 PC from my ZTE AXON 7 Android 8.0.0 phone camera pictures
but only for 'LivePhotos'.
Using the 'Transfer Files' option on USB cable connection. Will not get upon retry either. Consistent.

Cannot read file "mtp:\ZTE A2017U\A\DCIM\Camera\LivePhoto_20210321_153249.jpg".

Unexpected size of data stream.
Expected: 7224123 bytes
Actual: 7224129 bytes [notifyUnbufferedRead]

[Snoopy354]

Hy everybody

on IPhone if you disable convert to PC format in Settings it works for me.