[Bioclusters] Apple NFS + Automounter file loss behavior

[Christopher Dwan]

Over the past several years, I've had several customers mysteriously  
lose the entire contents of an NFS shared volume on OS X.  I finally  
isolated the behavior, and I want to share the recipe in the hopes  
that it might prevent future data loss.

I've reported this to Apple through their "Radar" system (ID  
4890612).  It's also been reported under another number:  4666168.   
The general response is that this will be resolved in the next  
version of OS X (Leopard), and may or may not be addressed as a patch  
for existing systems.

In order to completely destroy the contents of an NFS export by this  
method, one must do four questionable, but not totally unreasonable,  
things all at the same time:

1) The directory must be NFS exported with root=root, rather than  
with root "squashed"
2) The client machine must use the automounter to access the directory.
3) The client machine must use the NFS SystemStarter script to stop  
and start NFS (note, not AutoFS ... but NFS on the client).
4) The client machine must have had the NFS lockfile manually removed.

If you do all of those things, then the following code from the NFS  
SystemStarter script will run (/System/Library/StartupItems/NFS/NFS):

================================
AUTOMOUNTDIR=/private/var/automount

StartService ()
{
     CheckForNetwork
     if [ "${NETWORKUP}" = "-NO-" ]; then exit; fi
     lockfile -r 0 /var/run/NFS.StartupItem || exit 0

     ##
     # Set up NFS client.
     ##
     echo "Starting network file system"

         if [ -d ${AUTOMOUNTDIR} ]; then
                 chflags -R nouchg ${AUTOMOUNTDIR}
                 rm -rf ${AUTOMOUNTDIR}
         fi
==================================

If the lock file is present, you get this message:

node001:~ root# SystemStarter start NFS
lockfile: Sorry, giving up on "/var/run/NFS.StartupItem"

If the lockfile has been removed but the NFS export directory is  
still mounted under AUTMOUNTDIR then when the SystemStarter script  
runs again the "rm -rf" as root will systematically destroy all the  
data in the mounted directory.

Hopefully this will be useful.  I've been chasing it for a long time,  
and had to tell someone.

-Chris Dwan