MTP Sync Error - "Unexpected size of data stream"

Get help for specific problems
cpearless
Posts: 6
Joined: 2 Nov 2015

Post by cpearless

Using FFS 7.6, under Win 10 Pro x64, whenever I attempt to run a sync from an MTP source, to any desitination, I get the following error (screenshot attached):


> Cannot read file "mtp:\XXXXXXXXXX\Internal storage\Pictures\XXXXXXXX\XXXXXXXXXXX.jpg".
>
> Unexpected size of data stream.
> Expected: 843831 bytes
> Actual: 844085 bytes


This error occurs for every new file, I can copy the file manually using explorer no problem at all.

Any help or pointers would be greatly appreciated.

Cheers
You do not have the required permissions to view the files attached to this post.
User avatar
Zenju
Site Admin
Posts: 5528
Joined: 9 Dec 2007

Post by Zenju

That's an issue of the MTP device: the WPD_OBJECT_SIZE property of the jpg file is set to "843831", but when actually opening the file stream it returns with "844085" bytes.
You do not have the required permissions to view the files attached to this post.
cpearless
Posts: 6
Joined: 2 Nov 2015

Post by cpearless

That's an issue of the MTP device: the WPD_OBJECT_SIZE property of the jpg file is set to "843831", but when actually opening the file stream it returns with "844085" bytes.Zenju
Cheers for the response. Copying the file to the machine manually using explorer works fine though? If I then look at the file in detail on the device (via MTP), locally on the device and in Windows the sizes all report correct, the error only occurs in FFS? Even more bizarrely the difference is always eaxctly 5 bytes. For each file that causes errors, FFS always thinks it's 5 bytes larger than it is.

Cheers
You do not have the required permissions to view the files attached to this post.
User avatar
Zenju
Site Admin
Posts: 5528
Joined: 9 Dec 2007

Post by Zenju

FFS deliberately checks expected and actual size and shows an error if both disagree. Explorer obviously does not do this check.

> Even more bizarrely the difference is always eaxctly 5 bytes. For each file that causes errors, FFS always thinks it's 5 bytes larger than it is.

Let's keep to your first example: What does Explorer show for the size of the jpg file while it is on the device, and what after it was copied manually to a local drive?
You do not have the required permissions to view the files attached to this post.
cpearless
Posts: 6
Joined: 2 Nov 2015

Post by cpearless

FFS deliberately checks expected and actual size and shows an error if both disagree. Explorer obviously does not do this check.

> Even more bizarrely the difference is always eaxctly 5 bytes. For each file that causes errors, FFS always thinks it's 5 bytes larger than it is.

Let's keep to your first example: What does Explorer show for the size of the jpg file while it is on the device, and what after it was copied manually to a local drive?Zenju
I just realised that after making a comment about it being exactly a 5 byte descrepency, my example isn't! Apologies for that! Nonetheless, the reported size from explorer while the file is on the device is 843,831.00 bytes and the size when it's copied to disk is 843,831 bytes.

Screenshots attached.

Cheers for looking at this.
You do not have the required permissions to view the files attached to this post.
User avatar
Zenju
Site Admin
Posts: 5528
Joined: 9 Dec 2007

Post by Zenju

I've created a version of FFS with the size check disabled. Can you sync the above jpg file to disk with this version and then binary-compare it against the Explorer copy? (or alternatively send me both files for comparison)

[404, Invalid URL: http://www.mediafire.com/download/28yedyje3fsr4rr/FreeFileSync_7.7_no_mtp_size_check_Setup.exe]
You do not have the required permissions to view the files attached to this post.
cpearless
Posts: 6
Joined: 2 Nov 2015

Post by cpearless

I've created a version of FFS with the size check disabled. Can you sync the above jpg file to disk with this version and then binary-compare it against the Explorer copy? (or alternatively send me both files for comparison)

[404, Invalid URL: http://www.mediafire.com/download/28yedyje3fsr4rr/FreeFileSync_7.7_no_mtp_size_check_Setup.exe]Zenju
Just attempted to sync the file again, got a new error:

> Cannot write file "C:\Users\cpearless\Desktop\Instagram\IMG_20151102_131616.jpg.ffs_tmp".
>
> Unexpected size of data stream.
> Expected: 843831 bytes
> Actual: 844085 bytes

I presume it's also performing the same check on write, as well as read?
You do not have the required permissions to view the files attached to this post.
User avatar
Zenju
Site Admin
Posts: 5528
Joined: 9 Dec 2007

Post by Zenju

Just attempted to sync the file again, got a new error:

> Cannot write file "C:\Users\cpearless\Desktop\Instagram\IMG_20151102_131616.jpg.ffs_tmp".
>
> Unexpected size of data stream.
> Expected: 843831 bytes
> Actual: 844085 bytes

I presume it's also performing the same check on write, as well as read?cpearless
Forgot to remove a second check. Just updated the test version above.
You do not have the required permissions to view the files attached to this post.
cpearless
Posts: 6
Joined: 2 Nov 2015

Post by cpearless

Great, that copied the files just fine. Bizarrely, the files when copied with FFS actually increase in size. The slightly larger files still work just fine though, so there's no major corruption going on. Screenshots and files attached.

Cheers

[Attachment was removed! File name: MTP_Copy-FFS_7-vs-Explorer.zip (1688079 bytes)]
You do not have the required permissions to view the files attached to this post.
User avatar
Zenju
Site Admin
Posts: 5528
Joined: 9 Dec 2007

Post by Zenju

Thanks for the screenshot of the WPD Info tool. From all the data gathered so far this looks like bug or data corruption of the MTP device:

- Both WPD_OBJECT_SIZE and WPD_RESOURCE_ATTRIBUTE_TOTAL_SIZE of WPD_RESOURCE_DEFAULT report an expected file size of 843831

- However FFS reads 844085 bytes until reaching end of stream.

FreeFileSync's version seems to be the correct one, the file really is 844085 bytes long:

I ran a consistency check on both jpg files and the Explorer version is cut short missing data:

jpeginfo -c *.jpg
Copied with Explorer.jpg 1944 x 1944 24bit Exif N 843831 Premature end of JPEG file [WARNING]
Copied with FFS.jpg 1944 x 1944 24bit Exif N 844085 [OK]

And indeed, visually comparing both images, the Explorer version is missing pixels at the lower right.

Unfortunatly FreeFileSync doesn't have good options to remedy this situation: It could behave like Explorer and only copy WPD_OBJECT_SIZE bytes, but that's deliberately copying corrupt data. Or it could ignore the mismatch and copy the full stream, no matter how long it is. But this may mask other cases of data corruption where WPD_OBJECT_SIZE is correct, but the stream is too short/long. Essentially this is a bug that must be fixed by the manufacturer of the MTP device.
You do not have the required permissions to view the files attached to this post.
cpearless
Posts: 6
Joined: 2 Nov 2015

Post by cpearless

Thanks for the screenshot of the WPD Info tool. From all the data gathered so far this looks like bug or data corruption of the MTP device:

- Both WPD_OBJECT_SIZE and WPD_RESOURCE_ATTRIBUTE_TOTAL_SIZE of WPD_RESOURCE_DEFAULT report an expected file size of 843831

- However FFS reads 844085 bytes until reaching end of stream.

FreeFileSync's version seems to be the correct one, the file really is 844085 bytes long:

I ran a consistency check on both jpg files and the Explorer version is cut short missing data:

jpeginfo -c *.jpg
Copied with Explorer.jpg 1944 x 1944 24bit Exif N 843831 Premature end of JPEG file [WARNING]
Copied with FFS.jpg 1944 x 1944 24bit Exif N 844085 [OK]

And indeed, visually comparing both images, the Explorer version is missing pixels at the lower right.

Unfortunatly FreeFileSync doesn't have good options to remedy this situation: It could behave like Explorer and only copy WPD_OBJECT_SIZE bytes, but that's deliberately copying corrupt data. Or it could ignore the mismatch and copy the full stream, no matter how long it is. But this may mask other cases of data corruption where WPD_OBJECT_SIZE is correct, but the stream is too short/long. Essentially this is a bug that must be fixed by the manufacturer of the MTP device.Zenju
Thanks for looking into this and pointing me in the right direction, it's greatly appreciated.

I'm going to bury myself in the OS of the device and try to find out what's going on.

As always, FFS kicking ass.

Cheers
You do not have the required permissions to view the files attached to this post.
User avatar
Giangi
Posts: 69
Joined: 22 May 2006

Post by Giangi

Unfortunately MTP is a "crappy protocol", look at this my old comment about MTP implemented on Android: viewtopic.php?t=2166&p=9246#p9246
You do not have the required permissions to view the files attached to this post.
asdhhjsdka
Posts: 1
Joined: 2 Dec 2011

Post by asdhhjsdka

Shit, I run into the same issue with my Nexus 4 with CM12.1. Any chance, this can be fixed by Cyanogenmod Team if I open a bug report?
You do not have the required permissions to view the files attached to this post.
User avatar
Giangi
Posts: 69
Joined: 22 May 2006

Post by Giangi

That's a very bad news... Tired of all the MTP's *stupidity* I was seriously thinking about rooting my MotoG 2014 and use a CM build... but I guess they didn't touch the sources so the MTP errors *implemented* in the official Google sources are present into CM builds too...
You do not have the required permissions to view the files attached to this post.
clausmeister
Posts: 1
Joined: 3 Aug 2020

Post by clausmeister

Hello,

I'm having the same issue when trying to sync my iPhone's photos and videos to my computer. It works well for pictures, but for videos (*.mp4 and *.mov) I almost always get "unexpected size of data stream" / [notifyUnbufferedRead] errors. These also happen when using explorer. I'm on windows 10, FFS 11.0 (donation edition).

Is there a cmd-line switch or a setting in the GUI to disable the file size checks you mentioned above or is there a chance you could add something like that? I'm also ready to donate for it to happen!!!

The different file sizes are no problem for me as long as the videos work after they've been transferred. Also the risk of having corrupt/incorrect data is something I can live with.

Thanks a lot in advance, cheers