NFS - Automate attaching / detaching of shares

Prerequisites and what this article is NOT about

Prerequisites:

• Administrative privileges to the system
• Profound knowledge of
  • user administration,
  • bash editing,
  • nfs administration

This article does not cover the nfs administration of a linux/unix system.

The solution shown here is in principle scalable to any system. But specially the way client mounting via nfs is handled is only a quick hack, not threaded and therefore only useful for small networks with about a couple of PCs. Each nfs client is first ping and then handles mount/umount commands. Further development is possible.

This solution requires a new unpriviledged user, that is in the sudoers list, allowed to do the mount and unmount command. Additionally the instantiation of passwordless ssh connections between the machines is required for this user.

For some systems, this might be an security risc.

This solution requires that both, nfs client and server are permanently connected through the network.

The problem: add or removing shares

The intention for this HowTo was the problem, that in case you are removing an exported - and mounted - nfs file system from a network this will have severe side effects on the clients:

• All applications depending on the file system will fail/hang
• The Desktop Manager which will be left in an unusable state for minutes
• Drives will not be synced, data loss can occur

The solution: Publisher/Subscriber Pattern

If a nfs exported filesystem is removed from the nfs server, it must also be removed on all computers that are connected to the nfs server.

The server therefore has to have a list of these computers. In speach of the publisher/subscriber pattern he is the publisher.

The nfs clients are the subscribers. They subscribe to the server. The administrator is doing this in our approach by writing them into the servers clientlist.

To provide mechanisms to add or remove files on the clients, the server must be able to connect somehow to the clients - and that's the problem of most of the current nfs servers implementations: Mostly, just the clients connect in certain time intervals to the nfs servers in the network and then wait fore a certain amount of time for the drive to reappears.

The publisher/subscriber pattern however wants the subscribers (clients) to be reported immediately, when the system state changed, e.g. a drive is unmounted.

And remember the following: any PC can be publisher and/or subscriber. So a mixed setup with more than one nfs server is no problem. Each server cares for it's own clients, no matter what other servers do.

On my home's setup, my server PC exports shares covering a backup drive via USB and a home directory. The second one is a laptop and exports his desktop. The third one is the working horse and exports music and video data. Each PC cares for the others, no matter which on will boot or shutdown - he takes care for unmounting his exported shared on its addicted clients.

First open an editor

Login as root open the following files with your favorite editor.

gedit /etc/fstab \
      /etc/exports \
      /root/bin/nfs_clientlist.sh \
      /root/bin/mount_nfs_exports_on_attached_clients \
      /root/bin/udevmount.sh

These are the files we will alter/write. The last two have to be made executable via chmod 755.

All informations in the /etc/exports

The one and only file on the nfs server to handle

• nfs share's mountpoints
• possibly connected clients

is the /etc/exports.

Shares

It is easy to extract all exported shares of the nfs server with a little bash and awk charm and store it in a bash variable called EXPORTS

EXPORTS=`cat /etc/exports | grep -v ^# | grep " " \
| cut -f 1 | awk '{system("echo -n " $1 "----")}' \
| sed s/----/"  "/g`

It filters out all commentary lines, all empty lines, and writes the extracted directory names - devided by spaces - into the string $EXPORTS.

Clients

It could be possible, to extract these information for each share of this file with a little bash magic. Feel free to do this. It also might be possible to write all this in perl ;) Drop me a mail, if you made something new.

In this solution I quickly source the variables of the nfs server in a configuration file, called nfs_clientlist.sh. You can use the $HOSTNAME line, if you just have one network card and correctly configured your /etc/hosts:

############################################################
### PLEASE EDIT ############################################
############################################################

# All CLIENTLIST that use the nfs server on this machine
CLIENTLIST=" client1 client2 client3 "

# The hostname of this machine.
# Might be different with more than on network card.
#THISHOST=$HOSTNAME
THISHOST=myServerName

############################################################

Trigger the remote mount/unmount of shares

For the nfs servers machine to be able trigger the mount/unmount on connected clients there might be two way's:

• Writing a client/server program that communicates on a special port.
• Issuing a bash mount/umount via ssh on the remote machine

The first approach might be possible in e.g. perl/pyhton, but for my purposes connection/reconnecting via ssh fits my needs.

Prepare every PC in the network

To be able to trigger the mount/unmount on each client we need a user has the following abilities:

• Part of sudoers for mount/umount
• Passwordless reachable via ssh

Create an unpriviledged user with ssh

Log in as root and create an unpriviledged user with a unique user id (in my case 130) and give it the password udevmount. Because we enable passwordless login via ssh a secure password makes no sin. Afterwards we create the ssh keys. Please press just ENTER until the key generation is finished. We don not use a password!

useradd -g 0 -m -u 130 -s /bin/bash udevmount
passwd udevmount
su udevmount -c /bin/bash
ssh-keygen -t dsa

As root, add the user to the sudoers file:

visudo

Add the two lines

udevmount ALL=(ALL) NOPASSWD: /bin/mount
udevmount ALL=(ALL) NOPASSWD: /bin/umount

Export the ssh keys from the nfs server to each client

On each nfs server you have to attach the generated public key to the authorized_keys2 file of the client you want to connect to.

Login as user udevmount:

su udevmount -c /bin/bash

Execute the following script on your nfs server as user udevmount

#!/bin/sh
#
# /root/bin/ssh_helper.sh

source /root/bin/nfs_clientlist.sh

for client in $CLIENTLIST
do
 ssh $client "mkdir -p ~/.ssh; chmod 700 ~/.ssh"
 scp id_dsa.pub $client:~/.ssh/id_dsa.pub_${HOSTNAME}
 ssh $client "cat ~/.ssh/id_dsa.pub_${HOSTNAME} \
   >> ~/.ssh/authorized_keys2; \
   rm ~/.ssh/id_dsa.pub_${HOSTNAME}"
 chmod 600 id_dsa.pub
done

Here the version for a single client in one line ;)

CLIENT=mySingleClient;ssh $CLIENT "mkdir -p ~/.ssh; chmod 700 ~/.ssh";scp id_dsa.pub $CLIENT:~/.ssh/id_dsa.pub_${HOSTNAME}; ssh $NFS_SERVER "cat ~/.ssh/id_dsa.pub_${HOSTNAME} >> ~/.ssh/authorized_keys2; rm ~/.ssh/id_dsa.pub_${HOSTNAME}"

If you now try to connect to your client via ssh you should not be prompted for a password.

Handle mount/umount on the client

A script will trigger the mount/umount on each client. It is called with the parameter [mount|umount]:

#!/bin/sh
#
# /root/bin/umount_nfs_exports_on_attached_clients.

# Umount all exported nfs mount on attached clients.
# Do an appropriate fstab entry on these clients.
#
# The used mount directories AND client names MUST 
# match the udev and fstab entries 
# on thes machines - which should be the same - exactly.

export HOME=/root
export USER=root
export PATH=/usr/local/bin:/usr/bin:/usr/X11R6/bin:/bin:/opt/gnome/bin
. /root/bin/nfs_clientlist.sh

# Add some extra space to end and begin
CLIENTLIST=" ${CLIENTLIST} "

# Next line grabs all nfs exports
EXPORTS=`cat /etc/exports | grep -v ^# | grep " " | \
   cut -f 1 | awk '{system("echo -n " $1 "----")}' | \
   sed s/----/"  "/g`
EXPORTS=" $EXPORTS "
MOUNT=$1
ME=$0

echo $0 on $THISHOST.
echo CLIENTLIST: $CLIENTLIST 
echo exports: $EXPORTS

# force remount on network CLIENTLIST
for client in $CLIENTLIST
do
 if ping -c2 $client
 then
  logger -t $ME "Trying to do $MOUNT on $client - `date`"
  for export in $EXPORTS
  do
   if [ "$MOUNT" == "mount" ]
   then
     # For removable devices check, if they are really mounted
     if mount | grep ${export}
     then
       su udevmount -c "/usr/bin/ssh udevmount@${client} \
       'sudo /bin/$MOUNT ${THISHOST}:${export}'" \
       | logger -t $ME 
     fi
   else
     su udevmount -c "/usr/bin/ssh udevmount@${client} \
     'sudo /bin/$MOUNT ${THISHOST}:${export}'" \
     | logger -t $ME
   fi
  done
 fi
done

Call the script on shutdown/boot

To trigger all attached clients of the system, you should call the script in your systems shutdown or boot scripts. On my ubuntu system this works like this:

Boot

Just call the script in the file /etc/init.d/rmnologin. It is the last script called on the system before the login prompt appears:

vim /etc/init.d/rmnologin

Add the line

# Mount nfs shares on connected pcs
sh /root/bin/mount_nfs_exports_on_attached_clients mount

Pay attention to the parameter mount to call the script. This mounts all shares on this PC according to fstab. The script /etc/rc.local didn't work for me, because network connections where not aware when calling the script.

Edit a shutdown runlevel script

Unfortunatly there is not runlevel script for this on ubuntu. So I devired the rc.local accordingly:

vim /etc/init.d/rc.shutdown

#! /bin/sh

PATH=/sbin:/bin:/usr/sbin:/usr/bin
[ -f /etc/default/rcS ] && . /etc/default/rcS
. /lib/lsb/init-functions

do_stop() {
       if [ -x /etc/rc.shutdown ]; then
               log_begin_msg "Running local shutdown scripts (/etc/rc.shutdown)"
               /etc/rc.shutdown
               log_end_msg $?
       fi
}

case "$1" in
   start)
       # DO nothing
       ;;
   restart|reload|force-reload)
       echo "Error: argument '$1' not supported" >&2
       exit 3
       ;;
   stop)
       do_stop
       ;;
   *)
       echo "Usage: $0 stop" >&2
       exit 3
       ;;
esac

Now edit the script file, which will be called from the above init script when shutdown occurs:

vim /etc/rc.shutdown

#!/bin/sh -e
#
# rc.local
#
# This script is executed when machine is shutting down or entering an runlevel 
# without network support.
#
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

# Unmount nfs shares on attached clients
/bin/sh /root/bin/mount_nfs_exports_on_attached_clients umount

exit 0

Update the runlevels

And add it to the runlevels

update-rc.d -f rc.shutdown defaults

A more restrictive way to do this is

update-rc.d rc.shutdown start 01 2 3 4 5 . stop 01 0 1 6 .

The last approach assures that the script starts before others.

This works out of the box. If not see section troubleshooting at the end of this article.

Using desktop manager login/logout

Please see in the in the troubleshoot section at the end of this article.

Using udev to mount/umount drives

This chapter has additional prerequisites:

• Editing special udev rules for the actions add/remove
• Unique naming conventions for udev-rules and fstab

If you don't use removable devices, like an USB backup drive or hot swappable drives, than you can skip the next paragraphs.

Udev Rule for an external USB drive

The udev rule we intend to write has to take care of

• Giving the device to be mounted a unique name
• Triggering our mount_nfs_shares... script according to a remove or adding of the device
• Rejecting a buggy udev behaviour, that unpredictably fires on remove actions

The following udev rule suffices this:

vim /etc/udev/rules.d/10-local.rules

Unfortunately there is no variable handling in udev rules. So we have to specify the name of our device two or three times. This name must exactly match the entry in the /etc/fstab !!!

# LOKAL UDEV-RULES
#
# You can get infos about devices with
#
#   e.g. udevinfo  -a -p $(udevinfo -q path -n /dev/sda)

# External USB 3,5 MySpecialHarddisk
ACTION=="add",   SUBSYSTEMS=="usb", KERNEL=="sd?1", \
ATTRS{serial}=="200501CAF55", NAME="MySpecialHarddisk", \
SYMLINK:="disk/MySpecialHarddisk", GROUP="users", MODE="660", \
RUN+="/root/bin/udev_mount.sh $env{ACTION} 'MySpecialHarddisk' &"

ACTION=="remove", ENV{PRODUCT}=="67b/3507/1", \
RUN+="/root/bin/udev_mount.sh $env{ACTION} 'MySpecialHarddisk' &", \
OPTIONS=="ignore_remove"

The add action is no problem, but on remove action, the ATTRS{} key for the device is not avaiable anymore.

But with

udevmonitor --env

you will get the PRODUCT_ID - comes to ENV{PRODUCT} in the udev rule - that matches the disconnected device. It took me over a year to figure this trick out!

Reload the rules

 udevcontrol reload_rules

On the clients the /etc/fstab should look like this

# on client
vim /etc/fstab

[...]
nfs_server:/media/MySpecialHarddisk  /media/MySpecialHarddisk  nfs  defaults        0 0

The /etc/fstab on the server accordingly should mount the device the same way to /media/MySpecialHarddisk with noauto

# on nfs server
vim /etc/fstab

/dev/disk/MySpecialHarddisk  /media/MySpecialHarddisk  ext3 \
                rw,noauto,sync,noexec,users  0 0

Triggering the clients

As you can see, this rule triggers a script called udev_mount.sh. It is responsible for

• mount/umount local devices according to the above udev rule
• mount/umount shares on remote clients

The following script does all this. Since udev/hal fires unpredictable on removing a device, I blocked the call of the script for 15 seconds (BLOCK_FOR_SECONDS). This should be enough on any system but is not a save solution for the problem. I hope further versions of udev/hal will fix this.

This script takes two parameters, that are triggerd with the udev rule:

• the action that occurs: remove->umount, add->mount
• the name of the device: mount/umount according to /etc/fstab

#!/bin/sh
#
# /root/bin/udev_mount.sh

# mount tevion according to /etc/udev/rules.d/10-local.rules
# Do an appropriate fstab entry, so mount -a
# will do the rest

export HOME=/root
export USER=root
export PATH="$PATH:/usr/local/bin:/sbin/:/usr/bin:/usr/sbin/:/usr/X11R6/bin:/bin:/opt/gnome/bin"
. /root/bin/nfs_clientlist.sh

# Add some extra space to end and begin
CLIENTLIST=" ${CLIENTLIST} "

ACTION=$1
NAME=$2
ME=$0
LOCKFILE=/tmp/udev_mount.lock
TIMESTAMP=`cat $LOCKFILE`
myTime=`date +%s`
BLOCK_FOR_SECONDS=15

# Troubleshooting of fireing hal - udev events
test -f $LOCKFILE || echo $myTime > $LOCKFILE
if [ $myTime -lt $TIMESTAMP ]
then
  #logger -t udev_mount.sh "exiting ..."
  exit 1
else
  #logger -t udev_mount.sh "setting new timestamp ..."
  expr $myTime + $BLOCK_FOR_SECONDS > $LOCKFILE
fi


###################################################
# NOW LETS go

logger -t $ME "${NAME} ${ACTION}ed" 

# --------------------------------------------------
# mount local devices
if [ "$ACTION" == "add" ]
then
 if mount | grep /media/$NAME 
 then
  echo already mounted ...
 else
  mkdir -p /media/$NAME
  mount /media/$NAME

  # force remount on network clients
  for client in $CLIENTLIST
  do
    if ping -c2 $client
    then
      logger -t $ME "Trying to mount $NAME on ${client}."
      su udevmount -c "/usr/bin/ssh udevmount@${client} \
                      'sudo /bin/mount  ${THISHOST}:/media/${NAME}'" \
      | logger -t $ME
    fi
  done

 fi
fi

if [ "$ACTION" == "remove" ]
then
  
  # force remount on network clients
  for client in $CLIENTLIST 
  do
    if ping -c2 $client
    then
     logger -t $ME "Trying to do unmount ${NAME} on ${client}."
     su udevmount -c "/usr/bin/ssh udevmount@${client} \
                      'sudo /bin/umount ${THISHOST}:/media/${NAME}'" \
     | logger -t $ME
    fi
  done

  #The following is a bad idea, cause udev handles this
  #umount -f /media/${NAME}

fi

logger -t $ME "$ACTION of $NAME finished."

Test it

If you now add or remove a device, shutdown or boot your nfs server, on all your clients should seamlessly appear/disappear the devices. No desktop manager should hang anymore.

Discussion

I know that some aspects of this implementation could cause problems.

The scripts are a little bit time critical. It assumes, that all clients/servers are running and the network connections are o.k. There are no further checks.

One is the fact, that ping is used detect running clients. This might be a problem in secure networks, where ping is not enabled, or firewalls are blocking it.

Another is, that the udevmount user could be a security whole. The only way to decouple this, is to write a real wrapper or client/server application, that runs's as root and tunnels this aspect.

For the system administrator it might be a little bit complicated to handle all the files, but with just one nfs server it boiles down to

• Add a custom udev rule
• Unify the fstab mountpints and udev names for the devices
• edit the nfs_clients.sh with clientlist and local hostname.

The naming convention for equal names in fstab, exports and udev could lead to less flexible mounts. Dealing with symlinks could fix the problem.

Someone could bother about the file location in /root/bin. Everything could be moved to the home of udevmount's home and be protected speacially, so just root is able to read and execute the files.

Troubleshooting

System hangs nevertheless

Probably you disconnected the network. A permannt network connection is one of the prerequisites for a nfs network (see intro of this article).

First reconnect or restart the nfs server that causes the problem. Then wait a little bit - the lost share should reappear, the hung system work.

Then shutdown the system normally and your scripts should work.

Path

On some systems, your environment will not match the requirements.

• Is your shell /bin/sh
• Is your mount command in /bin/mount, same for umount

General

Open all this files in one editor on your nfs server as root:

gedit /etc/udev/rules.d/10-local.rules \
      /etc/fstab \
      /etc/exports \
      /etc/hosts \
      /root/bin/nfs_clientlist.sh \
      /root/bin/mount_nfs_exports_on_attached_clients \
      /root/bin/udevmount.sh

On your nfs client open

gedit /etc/fstab

• The mountpoints under /media, the device names for /etc/udev/rules.d/10-local.rules and /etc/fstab are unique
• Are all nfs impors named correct in /etc/fstab according to the naming conventions on the server. Same unique device name, same mountpoint.
• When you are using more than on network card (think about the wlan card on your laptop), check if you are using the correct IP/name mapping.

PC too fast

Use gdm logout/login

My laptop is shutting down too fast, so the network connection is lost, before the nfs clients could be contacted. The problem also occurs, when using NetworkManager for network connections (e.g. wireless card).

You can then try to trigger the mount/umount script via the gnome login/logout scripts which you find in the following directories

# Run something ...

# ...before the desktop mangager session starts:
/etc/gdm/PreSession/Default

# ... after the login:
/etc/gdm/PostLogin/Default

# ... after the gnome session ends
# e.g. pressing the logout button
/etc/gdm/PostSession/Default

Put the following command at the beginning of these files and change use mount and umount for whatever is needed:

/bin/sh /root/bin/mount_nfs_exports_on_attached_clients [mount|umount]

Alter the runlevels

Here another way for initialisation of the rc.shutdown script:

Remove the old links: $ update-rc.d -f rc.shutdown remove

Removing any system startup links for /etc/init.d/rc.shutdown ...
  /etc/rc0.d/K20rc.shutdown
  /etc/rc1.d/K20rc.shutdown
  /etc/rc2.d/S20rc.shutdown
  /etc/rc3.d/S20rc.shutdown
  /etc/rc4.d/S20rc.shutdown
  /etc/rc5.d/S20rc.shutdown
  /etc/rc6.d/K20rc.shutdown

Add rc.shutdown to be started at first:

$ update-rc.d rc.shutdown start 01 2 3 4 5 . stop 01 0 1 6 .
Adding system startup for /etc/init.d/rc.shutdown ...
  /etc/rc0.d/K01rc.shutdown -> ../init.d/rc.shutdown
  /etc/rc1.d/K01rc.shutdown -> ../init.d/rc.shutdown
  /etc/rc6.d/K01rc.shutdown -> ../init.d/rc.shutdown
  /etc/rc2.d/S01rc.shutdown -> ../init.d/rc.shutdown
  /etc/rc3.d/S01rc.shutdown -> ../init.d/rc.shutdown
  /etc/rc4.d/S01rc.shutdown -> ../init.d/rc.shutdown
  /etc/rc5.d/S01rc.shutdown -> ../init.d/rc.shutdown