Reactive File / Directory Synchronization

Reactive File / Directory Synchronization

In *nix land, rsync is the tool of choice for file/directory synchronization. It can be used to sync source and target destinations with extreme ease/speed based on file deltas.

Many system administrators use rsync in scripts to automate sync jobs which are usually kicked off via cron. The issue with doing something like is rsync can be very resource intensive especially on memory and disk IO.

Sometimes it is best to have these rsync jobs kicked off in a reactive manner based on certain conditions which are not dependent on cron or a schedule.

Since Linux kernel 2.6.13 a new subsystem was introduced which has the ability to monitor changes on a Linux filesystem – inode notify or Inotify.

Various wrappers/tools have been written to consume this inotify subsystem. One such tool is inotify-tools which is a C library and set of commands to interface with inotify.

Inotify can monitor various events on the filesystem such as metadata changes, file access, closing of a file etc. A more detailed list of events will be enclosed below. Using inotify-tools we can monitor for a single target file, or recursively monitor a directory as well for various events.

Recently we were asked to kick of an automated synchronization job based on a file drop via FTP/SFTP into a particular home directory.

To start with if you are using Ubuntu/Debian, please install the inotify-tools package. Two scripts are required to automate this functionality –

push-files.sh (the script which actually sets up the inotify watches and handles the rsync job)

#!/bin/bash
SOURCE=<>
TARGET=<>
TARGETUID=<<TARGET_UID_IF_REQUIRED>>
TARGETHOST=<<TARGET_HOST>>
KEY=<<SOURCE_SSH_PRIVATE_KEY-IF_USING_AUTO_SSH>>
RSYNC=/usr/bin/rsync

$RSYNC -av –delete –log-file=/var/log/rsync.log –remove-source-files -e “ssh -i $KEY” $SOURCE $TARGETUID@$TARGETHOST:$TARGET

exit 0

 

 

and

start-inotify.sh (this script is used to kick of the monitoring process and issue an rsync based on a filesystem event)

 

#!/bin/bash
while true; do

#
# various events can be used such as close_write, modify, acccess etc
#
inotifywait -e close_write /source_dir  && \
sleep 15 && /usr/local/bin/push-files.sh
done

More information on inotify-tools can be found here. Some of the filesystem events inotify can monitor are listed below.

IN_ACCESS
File was accessed (read) (*).

IN_ATTRIB

Metadata changed, e.g., permissions, timestamps, extended attributes, link count (since Linux 2.6.25), UID, GID, etc. (*).

IN_CLOSE_WRITE

File opened for writing was closed (*).

IN_CLOSE_NOWRITE

File not opened for writing was closed (*).

IN_CREATE

File/directory created in watched directory (*).

IN_DELETE

File/directory deleted from watched directory (*).

IN_DELETE_SELF

Watched file/directory was itself deleted.

IN_MODIFY

File was modified (*).

IN_MOVE_SELF

Watched file/directory was itself moved.

IN_MOVED_FROM

File moved out of watched directory (*).

IN_MOVED_TO

File moved into watched directory (*).

IN_OPEN

File was opened (*).

If you require any information on this article/scripts please drop me an email – Sujay D’Souza / sadsouza@win2ix.ca/ Win2ix Systems