Reactive File / Directory Synchronization
In *nix land, rsync is the tool of choice for file/directory synchronization. It can be used to sync source and target destinations with extreme ease/speed based on file deltas.
Many system administrators use rsync in scripts to automate sync jobs which are usually kicked off via cron. The issue with doing something like is rsync can be very resource intensive especially on memory and disk IO.
Sometimes it is best to have these rsync jobs kicked off in a reactive manner based on certain conditions which are not dependent on cron or a schedule.
Since Linux kernel 2.6.13 a new subsystem was introduced which has the ability to monitor changes on a Linux filesystem – inode notify or Inotify.
Various wrappers/tools have been written to consume this inotify subsystem. One such tool is inotify-tools which is a C library and set of commands to interface with inotify.
Inotify can monitor various events on the filesystem such as metadata changes, file access, closing of a file etc. A more detailed list of events will be enclosed below. Using inotify-tools we can monitor for a single target file, or recursively monitor a directory as well for various events.
Recently we were asked to kick of an automated synchronization job based on a file drop via FTP/SFTP into a particular home directory.
To start with if you are using Ubuntu/Debian, please install the inotify-tools package. Two scripts are required to automate this functionality –
push-files.sh (the script which actually sets up the inotify watches and handles the rsync job)
$RSYNC -av –delete –log-file=/var/log/rsync.log –remove-source-files -e “ssh -i $KEY” $SOURCE $TARGETUID@$TARGETHOST:$TARGET
start-inotify.sh (this script is used to kick of the monitoring process and issue an rsync based on a filesystem event)
while true; do
# various events can be used such as close_write, modify, acccess etc
inotifywait -e close_write /source_dir && \
sleep 15 && /usr/local/bin/push-files.sh
More information on inotify-tools can be found here. Some of the filesystem events inotify can monitor are listed below.
IN_ACCESS File was accessed (read) (*). IN_ATTRIB Metadata changed, e.g., permissions, timestamps, extended attributes, link count (since Linux 2.6.25), UID, GID, etc. (*). IN_CLOSE_WRITE File opened for writing was closed (*). IN_CLOSE_NOWRITE File not opened for writing was closed (*). IN_CREATE File/directory created in watched directory (*). IN_DELETE File/directory deleted from watched directory (*). IN_DELETE_SELF Watched file/directory was itself deleted. IN_MODIFY File was modified (*). IN_MOVE_SELF Watched file/directory was itself moved. IN_MOVED_FROM File moved out of watched directory (*). IN_MOVED_TO File moved into watched directory (*). IN_OPEN File was opened (*).
If you require any information on this article/scripts please drop me an email – Sujay D’Souza / email@example.com/ Win2ix Systems