I hope this is the correct place to raise this issue. I've done a quick search on the problem but I couldn't find anything obviously relevant.
Situation is; I'm using Lubuntu to sync (mirror) two external NTFS 250GB USB drives using RealtimeSync.
The drives are ~80% full and I've split the task into 3 sync jobs (syncing 3 different folders).
One of the jobs reports the error "Error Code 28: No space left on device (inotify_add_watch)"
The other two are fine.
FreeFileSync has no problem with the drives (full binary compare and sync had no problems), it's just RealtimeSync that seems to have problems with monitoring.
I moved the offending folder out of the folder that's being synced and RealtimeSync failed on the next folder (alphabetically).
I'm guessing there are too many files and or folders for RealtimeSync. Lubuntu reports 64,791 files which is 58,098 files and 6,692 folders by Windows (after removing the folder mentioned above from the sync folder).
Completely wild stab in the dark but could change unsigned int to long?
Error Code 28: No space left on device (inotify_add_watch)
- Posts: 11
- Joined: 13 Oct 2014
-
- Site Admin
- Posts: 7211
- Joined: 9 Dec 2007
The system-generated error message is misleadingly generic: In the context of inotify, the error code should be read as:
ENOSPC -
"The user limit on the total number of inotify watches was reached or the kernel failed to allocate a needed resource."
There is one inotify watch per folder, so too many folders being watched is the problem. Since the error message is talking about a user limit, this probably can be tweaked somewhere.
ENOSPC -
"The user limit on the total number of inotify watches was reached or the kernel failed to allocate a needed resource."
There is one inotify watch per folder, so too many folders being watched is the problem. Since the error message is talking about a user limit, this probably can be tweaked somewhere.
- Posts: 11
- Joined: 13 Oct 2014
Quick update;
The issue doesn't appear to be file/folder count, maybe total data size allowed for paths (again a complete stab in the dark).
I've got monitoring to work by removing folders until the error goes away but this is a lower file/folder count than one of the other syncs (I would prefer to monitor larger folders if at all possible).
The issue doesn't appear to be file/folder count, maybe total data size allowed for paths (again a complete stab in the dark).
I've got monitoring to work by removing folders until the error goes away but this is a lower file/folder count than one of the other syncs (I would prefer to monitor larger folders if at all possible).
- Posts: 11
- Joined: 13 Oct 2014
Thanks for the update Zenju. Any suggestions regarding where to start looking?
-
- Site Admin
- Posts: 7211
- Joined: 9 Dec 2007
Google for "inotify" and "user limit"Thanks for the update Zenju. Any suggestions regarding where to start looking?sfuser0000
PS: I've updated the RealtimeSync code and replaced the system-generated error message with the proper one from the Posix documentation.
- Posts: 11
- Joined: 13 Oct 2014
Sorry, didn't see that one coming, it's painful having to be asked to Google (i.e. I should have done this already).
Common problem, temporary solution, increase the watches limit (to, say, 512K):
sudo sysctl fs.inotify.max_user_watches = 524288
(worked for me, but I'll probably go permanent and maybe larger, info below)
To make the change permanent, edit the file /etc/sysctl.conf and add this line to the end of the file:
fs.inotify.max_user_watches = 524288
Size info below:
"Can you increase the watches? By how much? Any tradeoffs?
Short answer: Sure, no sweat. Go to straight to a half-million (524288) if you want...the additional memory used should be negligible on a modern system with 4GB+ of memory.
Each used inotify watch takes up 540 bytes (32-bit system), or 1 kB (double - on 64-bit)
This comes out of kernel memory, which is unswappable.
So, assuming you set the max at 524288, and all were used (improbable), you'd be using approx. 256MB/512MB of 32-bit/64-bit kernel memory
Note that your application will also use additional memory to keep track of the inotify handles, file/directory paths, etc. -- how much depends on its design."
Common problem, temporary solution, increase the watches limit (to, say, 512K):
sudo sysctl fs.inotify.max_user_watches = 524288
(worked for me, but I'll probably go permanent and maybe larger, info below)
To make the change permanent, edit the file /etc/sysctl.conf and add this line to the end of the file:
fs.inotify.max_user_watches = 524288
Size info below:
"Can you increase the watches? By how much? Any tradeoffs?
Short answer: Sure, no sweat. Go to straight to a half-million (524288) if you want...the additional memory used should be negligible on a modern system with 4GB+ of memory.
Each used inotify watch takes up 540 bytes (32-bit system), or 1 kB (double - on 64-bit)
This comes out of kernel memory, which is unswappable.
So, assuming you set the max at 524288, and all were used (improbable), you'd be using approx. 256MB/512MB of 32-bit/64-bit kernel memory
Note that your application will also use additional memory to keep track of the inotify handles, file/directory paths, etc. -- how much depends on its design."
