Possible problems using FreeFileSync with Sharepoint and Box.com

[gudjkrist]

I understand that this may be an edge case but I just wanted too see if anyone knew what my problems are.

The scenario: I'm trying to mirror a folder on Sharepoint to a folder on my local computer that is synced to the cloud using Box.com. This folder is then synced using Box.com to a Mac. I'm using version 6.2 since I don't have access to a Windows computer with admin rights.

The problem: After initial sync when I run compare again FFS wants to re-sync most of the files.

What I've tested so far:
- Confirmed that the timestamps and sizes of the files FFS wants to re-sync are the same.
- Confirmed in Box.com manual that it does not change the timestamps of files it syncs.
- Tried to sync by "File content" only. Did not help.
- Changed FileTimeTolerance config to both -1 and 31540000. No help.

Anyone got an idea of what I could be doing wrong or what the next steps to try are?

[gudjkrist]

As an update I've tested the same scenario with FFS 8.2 and the problem remains. And to clarify, syncing from Sharepoint to my local computer works fine but syncing from either Sharepoint or local computer to the Box.com folder is still causing FFS to want to replace most files. There is something going on in the Box.com folder, but damned if I know what it is since the files are the same size and with the same timestamp.

[Zenju]

You should check how the file pairs have been categorized in the middle column of FFS's main window.

[gudjkrist]

They are categorised as "Item exists on left side only" and "Item exists on right side only". Very strange. An example right in front of me is:

Mirroring from left to right side:
- Left side: "Relative path": "Documents\Technical documents\Costs.xlsx"
- Right side side: "Relative path": "Documents\Technical documents\Costs.xlsx"
- Middle column shows that it wants to delete item on right side due to "Item exists on right side only" and copy item on left side to right side due to "Item exists on left side only".

[Zenju]

Can you post a screenshot and your configurations file?

[gudjkrist]

What exactly do you need?

- GlobalSettings.xml?
- The front screen with the comparison results?
- Synchronization Settings?

[gudjkrist]

The configuration file is pasted below. (GlobalSettings.xml) — note that I removed information about my folders and files. Attached is a screenshot with irrelevant blurred out. You can see an example of three files in the same folder with the same size that haven't been touched and show the same timestamp in Windows. Still FFS wants to delete the one on the right and copy the left one over (it's set to mirror from left to right).

<?xml version="1.0" encoding="UTF-8"?>
<FreeFileSync XmlFormat="2" XmlType="GLOBAL">
<General>
<Language Name="English (U.S.)"/>
<FailSafeFileCopy Enabled="true"/>
<CopyLockedFiles Enabled="false"/>
<CopyFilePermissions Enabled="false"/>
<AutomaticRetry Count="0" Delay="5"/>
<FileTimeTolerance Seconds="2"/>
<FolderAccessTimeout Seconds="20"/>
<RunWithBackgroundPriority Enabled="false"/>
<LockDirectoriesDuringSync Enabled="true"/>
<VerifyCopiedFiles Enabled="false"/>
<LastSyncsLogSizeMax Bytes="100000"/>
<NotificationSound CompareFinished="" SyncFinished="gong.wav"/>
<OptionalDialogs>
<WarnUnresolvedConflicts Enabled="true"/>
<WarnNotEnoughDiskSpace Enabled="true"/>
<WarnSignificantDifference Enabled="true"/>
<WarnRecycleBinNotAvailable Enabled="true"/>
<WarnInputFieldEmpty Enabled="true"/>
<WarnDatabaseError Enabled="true"/>
<WarnDependentFolders Enabled="true"/>
<WarnFolderPairRaceCondition Enabled="true"/>
<WarnDirectoryLockFailed Enabled="true"/>
<ConfirmSaveConfig Enabled="true"/>
<ConfirmStartSync Enabled="false"/>
<ConfirmExternalCommandMassInvoke Enabled="true"/>
</OptionalDialogs>
</General>
<Gui>
<MainDialog Height="1398" Maximized="true" PosX="-8" PosY="-8" Width="2560">
<ManualCopyTo KeepRelativePaths="false" OverwriteIfExists="false">
<FolderHistory LastUsedPath="" MaxSize="15"/>
</ManualCopyTo>
<ManualDeletion UseRecycler="true"/>
<CaseSensitiveSearch Enabled="false"/>
<FolderPairsVisible Max="6"/>
<OverviewPanel ShowPercentage="true" SortAscending="false" SortByColumn="Bytes">
<Columns>
<Item Stretch="1" Type="Tree" Visible="true" Width="-120"/>
<Item Stretch="0" Type="Count" Visible="true" Width="60"/>
<Item Stretch="0" Type="Bytes" Visible="true" Width="60"/>
</Columns>
</OverviewPanel>
<CenterPanel IconSize="Small" SashOffset="0" ShowIcons="true">
<ColumnsLeft PathFormat="Relative">
<Item Stretch="1" Type="Path" Visible="true" Width="-80"/>
<Item Stretch="0" Type="Date" Visible="false" Width="112"/>
<Item Stretch="0" Type="Size" Visible="true" Width="80"/>
<Item Stretch="0" Type="Ext" Visible="false" Width="60"/>
</ColumnsLeft>
<ColumnsRight PathFormat="Relative">
<Item Stretch="1" Type="Path" Visible="true" Width="-80"/>
<Item Stretch="0" Type="Date" Visible="false" Width="112"/>
<Item Stretch="0" Type="Size" Visible="true" Width="80"/>
<Item Stretch="0" Type="Ext" Visible="false" Width="60"/>
</ColumnsRight>
</CenterPanel>
<DefaultViewFilter>
<Shared Conflict="true" Equal="false" Excluded="false"/>
<CategoryView Different="true" LeftNewer="true" LeftOnly="true" RightNewer="true" RightOnly="true"/>
<ActionView CreateLeft="true" CreateRight="true" DeleteLeft="true" DeleteRight="true" DoNothing="true" UpdateLeft="true" UpdateRight="true"/>
</DefaultViewFilter>
<Perspective5>layout2|name=CenterPanel;caption=;state=256;dir=5;layer=0;row=0;pos=0;prop=100000;bestw=884;besth=171;minw=-1;minh=-1;maxw=-1;maxh=-1;floatx=-1;floaty=-1;floatw=-1;floath=-1|name=TopPanel;caption=Main Bar;state=2099708;dir=1;layer=2;row=1;pos=0;prop=100000;bestw=401;besth=50;minw=180;minh=50;maxw=-1;maxh=-1;floatx=-1;floaty=-1;floatw=-1;floath=-1|name=ProgressPanel;caption=;state=2097662;dir=1;layer=2;row=2;pos=0;prop=100000;bestw=256;besth=81;minw=200;minh=81;maxw=-1;maxh=-1;floatx=-1;floaty=-1;floatw=-1;floath=-1|name=FoldersPanel;caption=Folder Pairs;state=2099708;dir=1;layer=2;row=3;pos=0;prop=100000;bestw=882;besth=45;minw=-1;minh=86;maxw=-1;maxh=-1;floatx=-1;floaty=-1;floatw=-1;floath=-1|name=SearchPanel;caption=Find;state=2099710;dir=3;layer=2;row=2;pos=0;prop=100000;bestw=378;besth=25;minw=200;minh=25;maxw=-1;maxh=-1;floatx=-1;floaty=-1;floatw=-1;floath=-1|name=ViewFilterPanel;caption=View Settings;state=2099708;dir=3;layer=2;row=1;pos=0;prop=100000;bestw=884;besth=45;minw=82;minh=45;maxw=-1;maxh=-1;floatx=-1;floaty=-1;floatw=-1;floath=-1|name=ConfigPanel;caption=Configuration;state=2099196;dir=4;layer=3;row=0;pos=0;prop=100000;bestw=352;besth=57;minw=142;minh=57;maxw=-1;maxh=-1;floatx=-1;floaty=-1;floatw=-1;floath=-1|name=OverviewPanel;caption=Overview;state=2099196;dir=4;layer=3;row=0;pos=1;prop=100000;bestw=884;besth=171;minw=-1;minh=-1;maxw=-1;maxh=-1;floatx=-1;floaty=-1;floatw=-1;floath=-1|dock_size(5,0,0)=265|dock_size(1,2,1)=50|dock_size(1,2,3)=195|dock_size(3,2,1)=45|dock_size(4,3,0)=302|</Perspective5>
</MainDialog>
<DefaultExclusionFilter>
<Item>\System Volume Information\</Item>
<Item>\$Recycle.Bin\</Item>
<Item>\RECYCLER\</Item>
<Item>\RECYCLED\</Item>
<Item>*\desktop.ini</Item>
<Item>*\thumbs.db</Item>
</DefaultExclusionFilter>
<LastUsedConfig>
<Item>ffs_drive:REMOVED</Item>
</LastUsedConfig>
<ConfigHistory MaxSize="50">
<Item>ffs_drive:REMOVED</Item>
<Item>ffs_drive:REMOVED</Item>
</ConfigHistory>
<FolderHistoryLeft MaxSize="15">
<Item>REMOVED</Item>
<Item>REMOVED</Item>
<Item>REMOVED</Item>
</FolderHistoryLeft>
<FolderHistoryRight>
<Item>REMOVED</Item>
<Item>REMOVED</Item>
<Item>REMOVED</Item>
</FolderHistoryRight>
<OnCompletionHistory MaxSize="8"/>
<ExternalApps>
<Item>
<one>Show in Explorer</one>
<two>explorer /select, "%item_path%"</two>
</Item>
<Item>
<one>Open with default application</one>
<two>"%item_path%"</two>
</Item>
</ExternalApps>
<LastOnlineCheck>1466770852</LastOnlineCheck>
<LastOnlineVersion>8.2</LastOnlineVersion>
</Gui>
</FreeFileSync>

[Zenju]

When seemingly equal file paths on both sides are not matched on a single row each, then there must be some difference in their path: Maybe there is some additional blank character or some other small difference on one side, but not the other.

[gudjkrist]

Thanks. Since this happens with all the files in all directories it's surely something in how Sharepoint and Box.com handle the files? I'm guessing this is something out of your (or my) control?

[Zenju]

First I'd install FFS 8.2. Then compare the "relative path" column. There probably is some minor difference that is easy to miss. (If not, then this is somthing worth of further investigation).

[gudjkrist]

Sorry, forgot to let you know that I already upgraded to FFS 8.2 (I was able to borrow another Windows computer to put 8.2 on a flash drive and copy to this computer).

I don't think it's a minor difference as it's on all my folders and files in all projects that I'm mirroring from Sharepoint to Box.com and not touching at all. Also, when I compare right after it reports that no changes should be made but when I compare a day later FFS wants to replace most of the files.

I suspect this is some strange issue between Sharepoint and Box.com.

[Zenju]

Pick a specific file pair that should be matched, but isn't, then CTRL+C on the FFS main dialog on the left file and paste into a text editor, ideally Notepad++, do the same for the second file on the right, then compare if the two paths are equal. There might be hidden Unicode control characters, but this should be visible in a Hex Editor.

[gudjkrist]

I did this and found no difference. Then I started trying all kinds of different files and folders and found a pattern. It seems to only happen on Box.com folders (that are synced to the cloud) and not on local folders and only with folders and files that have non-standard ASCII characters, such as í, á, ó, ð.

[Zenju]

You may be on to something. Can you do a "menu->tools->export file list" for such a specific non-matching file pair and attach the .csv file here?

[gudjkrist]

Here you go. Picked one folder with few files.

[Zenju]

The paths are indeed different, although they are rendered identically on the GUI:

E.g. "Financial - Greining fyrir áhættumat.vsdx":

On "C:\Users\smith\Box Sync\Sharepoint\Div", the "á" character is returned in decomposed UTF as "61 cc 81" hex.

\\project.star.is\svid\comp\throunardeild\Shared Documents OTOH returns the character in precomposed from as "c3 a1" hex.

The problem seems to be the box.com device, since on Windows precomposed UTF is the norm.

[Zenju]

I believe the following is happening:

1. You change some file on "\\project.star.is\svid\comp\throunardeild\Shared Documents" => it is precomposed UTF according to Windows default
2. You sync "\\project.star.is\svid\comp\throunardeild\Shared Documents" to "C:\Users\smith\Box Sync\Sharepoint\Div" using FreeFileSync
3. FFS compares both sides as equal (both sides are precomposed UTF)
4. Box.com automatically uploads the changed files on "C:\Users\smith\Box Sync\Sharepoint\Div"
5. The Mac automatically converts precomposed to decomposed, latter is the default of Mac's HFS file system
6. This is seen again as a change by Box.com, so it propagates the conversion back to "C:\Users\smith\Box Sync\Sharepoint\Div"
7. Now FreeFileSync finds the UTF difference and the files do not match anymore.

[gudjkrist]

Ah. Makes sense. And there is nothing I can do about that, I guess?

By the way, thanks a bunch for the support.

Edit: Well, except using only ASCII characters perhaps?

[Zenju]

Generally speaking, this is a bug for box.com to solve, since their implementation should consider the standards of both Windows and Mac and internally do the UTF conversion.

Alternatively, FreeFileSync could be made more tolerant and accept the decomposed UTF form on Windows, too, even though it's an edge case. However this might impose a (slight?) performance degradation at the expense of 99% of the normal usage. I'll do a few tests and see if there is a quick way to detect this scenario. However even this might not work if, after a UTF conversion, the file system does not recognize the file path anymore, because technically the two UTF forms are just two different paths.

And yes, using ASCII-only avoids the precomposed/decomposed UTF problem altogether.

[gudjkrist]

It's such an edge case that I don't even feel comfortable asking for it. Thanks a bunch for giving your time getting to the root of the problem. For now, I'll try to keep files ASCII-only.