Keeping Abreast with Filesystem Changes

Originally published December 13, 2018 @ 11:48 am

The inotify is a Linux kernel sybsystem for notifying user-space applications of filesystem changes. I always thought this exceptionally handy utility was under-appreciated or at least underutilized.

I am not planning on going through the many features of inotify, but instead will provide a couple of practical examples of how it can be used in your daily sysadmin work.

Recently I’ve ran into an issue when an application configuration file was being modified by people who can’t avoid typos to save their lives. Since they were unlikely to get better at typing and I couldn’t watch them around the clock, inotify was the solution.

If you don’t have it installed already, the process is quick:

yum -y install inotify-tools

The script below will use inotifywait to keep an eye on /etc/smb/smb.conf file. If that file changes, the script will undo the changes by restoring /etc/samba/.smb.conf_lastgood and then it reload Samba.

#!/bin/bash

while true

do
  
  inotifywait -e modify,attrib,close_write,move,create,delete /etc/samba/smb.conf && \
  
  /bin/cp -p /etc/samba/.smb.conf_lastgood /etc/samba/smb.conf && \
  /sbin/service smb reload

done


As you can see, just about any change to the file would trigger a response. You can run this script with nohup or & disown. Alternatively, you can have it started with the system (say, by adding it to Samba startup, which would make perfect sense here). Here’s an example of how to start this script with nohup:

nohup /var/adm/bin/inotifywait_smb.conf.sh </dev/null >/dev/null 2>&1 &

This will continue to run until the next reboot. To start this process at boot time (but after Samba daemon), you can create an init configuration file in /etc/init/inotifywait.conf that would look something like this:

description "inotifywait monitor for smb.conf"
author      "Igor"

# Start after samba
start on started smb

# Autostart this service if it stops
respawn

# If this service stops 10 times within half an hour, don't start it again
respawn limit 10 1800

# This service should run on run levels 3-5
start on runlevel [345]

# In case the fails fails to shut down gracefully, kill it
kill timeout 5

# We use this instead of stop on shutdown in order to make sure we're stopped by initctl stop (which means no respawn will happen)
stop on starting rc RUNLEVEL=[0126]

pre-start script
    echo "[$(date)] inotifywait starting"
end script

script
    echo $$ > /var/run/inotifywait.pid
    exec /var/adm/bin/inotifywait_smb.conf.sh
end script

pre-stop script
    /bin/rm /var/run/inotifywait.pid
    echo "[`date`] inotifywait stopping"
    mkdir -p /etc/samba/inotifywait/stopped
end script

Then you can start it once manually and check the status to make sure everything worked as expected:

initctl reload-configuration
initctl list | grep inotifywait
initctl start inotifywait
initctl status inotifywait

   inotifywait start/running, process 1318

ps -ef | grep [1]318

   root      1318     1  0 12:49 ?        00:00:00 /bin/bash /var/adm/bin/inotifywait_smb.conf.sh
   root      1331  1318  0 12:49 ?        00:00:00 inotifywait -r -e modify,attrib,close_write,move,create,delete /etc/samba/smb.conf

Similar to the previous example, the script below would monitor changes to SSHd configuration and recover the specified version of sshd_config file from etckeeper (you can read more about etckeeper here).

#!/bin/bash

while true

do
  
  inotifywait -e modify,attrib,close_write,move,create,delete /etc/ssh/sshd_config && \
  
  etckeeper vcs checkout 2afabe41081bab570c987160b89e42506dd1f3c0 /etc/ssh/sshd_config && \
  
  /sbin/service sshd restart

done


I commonly use inotify to trigger rsync when source filesystem content changes. What folks don’t always realize is that inotify can also be used to monitor special system files, such as those in /proc and /dev and this can be tremendously useful.

Here’s a basic example of being notified when a specified process (sleep 30 in this case) has exited:

sleep 30 &

[1] 469

pid=$!

inotifywait /proc/${pid}/exe && \

echo "PID ${pid} is done" | mailx -s "PID ${pid} is done" root@domain.com


Setting up watches.

Watches established.

/proc/469/exe CLOSE_NOWRITE,CLOSE