S/390 NFS Post Box for remote hosts (postbox)

Introduction

Often, you need to update files or execute commands on some or all of your Linux hosts. For example, you need to shutdown all your hosts for routine hardware or software maintenance. Or you may want to update some file, such as /etc/motd, on all or some of hosts. However, this can be time consuming and error prone if you have many of these hosts when you are using telnet or ftp.Using an NFS server, you can set up a simple postbox system to issue the necessary commands automatically on all or only certain hosts. The postbox can be scanned periodically using a crontab on the remote hosts to look for commands to execute.

This technique consists of the following parts:

back to top

Remote Host Setup

On each remote host you want to participate, you set up a remote postbox, insert a script to check the postbox, and then a crontab entry to periodically check the NFS postbox.

  1. mkdir /home2/postbox (Our convention is to put all user directories on a separate volume from the root and mount it as /home2 so we can ipl various root volumes without having to recreate the user files).
  2. mkdir /home2/postbox/bin
  3. Create the script to check the NFS postbox
    /home2/postbox/bin/admin.check.postbox
    #..............................................................................
    # check $adminBOX file for bash scripts to execute
    # - all hosts execute file adminall on the NFS server
    # - $HOSTNAME executes file $HOSTNAME on the NFS server, then removes the file
    #..............................................................................
    PATH=$PATH:/sbin:/usr/sbin
    adminBOX=/mnt/admin/postbox
    HOSTBOX=/home2/admin/postbox
    #
    test ! -d "$adminBOX" -a "$HOSTNAME" != "syssbld" && \
         mkdir -p /mnt/admin && \
         mount -t NFS -o rw,timeo=300,retrans=5,wsize=8192,rsize=8192,hard,intr,bg,suid,nolock
                     NFSserverIP:/mnt/admin /mnt/admin
    #
    test -f "$adminBOX/adminall" && \
    bash $adminBOX/adminall
    #
    test -f "$adminBOX/$HOSTNAME" && \
         bash $adminBOX/$HOSTNAME && \
         rm $adminBOX/$HOSTNAME
    #
    sleep 1
    case $HOSTNAME in
         syssbld
         # do not dismount postbox on NFS server host
         ;;
         sysstst)
         # do not dismount postbox on certain other hosts
         ;;
         *)
         umount -f /mnt/admin
         ;;
    esac
  4. Create the crontab entry - generally, we check once a day, during third shift. Each host has a slightly different time to check to minimize clustering on activity on the net and NFS server.
    # check at 1:36 a.m. every day for this remote host
    36 1 * * * /bin/bash   /home2/admin/postbox/bin/admin.check.postbox

back to top

NFS Postbox Server Setup

  1. Set up your /etc/exports file to allow read/write to your postbox directory. If you do not allow write, the remote host will execute the file each time the cron interval is executed. To avoid this, allow the file to be removed from the NFS server by the remote host. I've used the "net ip/subnet mask" format.
    /mnt/admin/postbox 10.1.1.64/255.255.255.192(rw,no_root_squash)
  2. mkdir /mnt/admin/postbox
  3. mkdir /mnt/admin/postbox/sample - files to help generate remote host scripts

back to top

Sample NFS scripts and files to be sent to/ execute on remote host

These are some of the things that you might want to do on the remote host. They are kept in "/mnt/admin/postbox/sample".

Note that "#!/bin/bash" is not present in the scripts to be executed by the remote hosts because they are executed using "/bin/bash scriptname" on the remote hosts.They are set up so that they can be concatenated to other scripts in the postbox for a remote host, preserving any pending scripts.

back to top

Copy files to remote host


/mnt/admin/postbox/sample/remote.cp
#.......... remote.cp
PATH=$PATH:/sbin:/usr/sbin
adminBOX=/mnt/admin/postbox
adminSAMP=$adminBOX/sample
HOSTBOX=/home2/admin/postbox
HOSTETC=/etc
#.......... remote copy
cp -p $adminSAMP/motd $HOSTETC/motd
chmod 644 $HOSTETC/motd
back to remote scripts

Execute commands at remote host

/mnt/admin/postbox/sample/remote.ex
#.......... remote.ex
PATH=$PATH:/sbin:/usr/sbin
adminBOX=/mnt/admin/postbox
HOSTBOX=/home2/admin/postbox
#.......... commands to be issued
logger -t admin "$(date) $adminBOX/$HOSTNAME executed"
back to remote scripts

Execute command at remote host at a later time

/mnt/admin/postbox/sample/remote.at
#.......... remote.at
PATH=$PATH:/sbin:/usr/sbin
adminBOX=/mnt/admin/postbox
HOSTBOX=/home2/admin/postbox
SYNC=$HOSTBOX/at.0600.sync
#..... generate contents of SYNC script on remote host
echo "PATH=$PATH:/sbin:/usr/sbin" > $SYNC
echo "sync" >> $SYNC
echo "rm $SYNC" >> $SYNC
#.......... at certain times, do a sync
at 0600 Monday -f $SYNC
back to remote scripts

Message to be issued on remote host

/mnt/admin/postbox/sample/remote.shut.msg
*****
***** host shutting down at 2:00 p.m. today
***** for Scheduled USER Standalone time
*****
back to remote scripts

Shutdown on Sunday on remote host

/mnt/admin/postbox/sample/remote.shut.sunday
#.......... remote.shut.sunday
PATH=$PATH:/sbin:/usr/sbin
adminBOX=/mnt/admin/postbox
HOSTBOX=/home2/admin/postbox
#.......... name script and text files on the remote host
SHUTDAY=$HOSTBOX/at.Sunday.0600.shutdown
WALLDAY=$HOSTBOX/at.Sunday.0600.wall
SYNCDAY=$HOSTBOX/at.Sunday.0600.sync
WALLMSG=$HOSTBOX/at.Sunday.0600.wallmsg
#
#..........log this in /var/log/messages when this script executes
logger -t admin "$(date) $adminBOX/$HOSTNAME executed"

#.......... you can copy files to the remote postbox
#
#..... generate contents of WALLMSG message file on remote host
cp -p $adminBOX/sample/remote.shutmsg$WALLMSG

#
#.......... or you can generate your files inline on your remote host
#
#..... generate contents of WALLDAY script on remote host
echo "PATH=$PATH:/sbin:/usr/sbin" > $WALLDAY
echo "wall $WALLMSG" >> $WALLDAY
#
#..... generate contents of SYNCDAY script on remote host
echo "PATH=$PATH:/sbin:/usr/sbin" > $SYNCDAY
echo "sync" >> $SYNCDAY
#
#.......... generate contents of SHUTDAY script on remote host
#.......... to keep all code together
echo "PATH=$PATH:/sbin:/usr/sbin" > $SHUTDAY
echo "#..... remove some traces" >> $SHUTDAY
echo "rm $WALLDAY" >> $SHUTDAY
echo "rm $SYNCDAY" >> $SHUTDAY
echo "rm $WALLMSG" >> $SHUTDAY
echo "#..... issue shutdown command" >> $SHUTDAY
echo "shutdown -h now" >> $SHUTDAY
echo "#..... remove last trace" >> $SHUTDAY
echo "rm $SHUTDAY" >> $SHUTDAY
#
#.......... or you can execute various commands on the remote host
#
#.......... issue at commands at appropriate times
#           the commented out at commands allow additional times
#           (see admin command "sed2nd")
#at 1000 Sunday -f $WALLDAY
#at 1300 Sunday -f $WALLDAY
at 0545 Sunday -f $SYNCDAY
at 0550 Sunday -f $WALLDAY
at 0550 Sunday -f $SYNCDAY
at 0555 Sunday -f $SYNCDAY
at 0550 Sunday -f $SYNCDAY
at 0600 Sunday -f $SHUTDAY
#
#.......... list the at commands in the remote host /var/log/messages
logger -t admin ".... 0600 Sunday $SHUTDAY $(/usr/bin/at -l) ...."
back to remote scripts
back to top

Sample NFS Server - administrative scripts

back to top

Generate scripts in postbox for each remote host

This examples takes scripts that have been built and propagates them to the /mnt/admin/postbox directory, one file per host. To preserve any previous pending scripts that have not been read, the results of this execution are appended to the pending files.

/mnt/admin/postbox/sample/admin.gen
. admin.gen remote.shut.sunday
#!/bin/bash
#.......... admin.gen
PATH=$PATH:/sbin:/usr/sbin
adminBOX=/mnt/admin/postbox
HOSTBOX=/home2/admin/postbox
#
for D in lnx1 lnx2 lnx3
   do
   cat $1 >> $adminBOX/sys$D
   done
back to admin scripts

Modify a script for date and time

This example takes the /mnt/admin/postbox/sample/remote.shut.sunday file and modifies for a 2nd shift shutdown (2:00 p.m.). The results are stored in /mnt/admin/postbox/sample/temp.shut.2nd.xxxx where xxxx is the date argument given to the script.

/mnt/admin/postbox/sample/sed2nd
. sed2nd 073101 # results in temp.shut.2nd.070301
#!/bin/bash
#.......... sed2nd
sed -e s/#at/at/g \
      -e s/Sunday/$1/g \
      -e s/0600/1400/g \
      -e s/05/13/g \
      -e s/Maintenance/Standalone/g \
      remote.shut.sunday > temp.shut.2nd.$1   done
back to admin scripts
back to top

Delete "at" commands

Sometimes you may have to delete some of the "at" commands already issued. For safety sake, you probably will have to logon to each host you need to delete and check their pending "at" commands ("atq'). Then you can use the admin.atqdel script to delete a range of "at" commands.

/mnt/admin/postbox/sample/admin.atqdel
atq
. admin.atqdel 5 10
#!/bin/bash
#.......... admin.atqdel
echo "----- start: $1 stop: $2"
let N=$1
while test $N -le $2
    do
    at -d $N
    let N=$N+1
    done
back to admin scripts
back to top