Last updated on:
Sunday, July 06, 2008

Software

• Linux/390 Distributions
• Linux/390 Patches
• Hercules - An S/390 Hardware Emulator

Information

• Large numbers of links
• Linux/390 HOWTOs
• IBM's Linux/390 pages
• Distribution Contents
• IBM VM and Linux Resources
• Linux man pages
• Notes & Observations
• Top 10 reasons to run Linux on VM
• Linux/390 at Princeton University
• i/370 Port, aka Bigfoot
• linuxvm.org Archives

Community

• Community Members
• Linux-390 Listserv
• Linux/390 Presentations
• Linux/390 Subscribers
• Linux/390 Survey Results

News

• Enterprise Linux Today
• Linux Journal
• Linux Today
• Linux World
• Kernel Traffic

Fun

• Ubersoft
• User Friendly

Credits

• Cast of characters

Using BIND Mounts to Create A Simplified BaseVol/GuestVol Linux Server

The information in this document is based on the concept provided in the IBM Redbook Linux on IBM eServer zSeries and S/390: Large Scale Linux Deployment, SG24-6824-00. This is a simpler procedure to follow to create a base-volume/guest-volume Linux for zSeries server which exploits the bind command. The procedure here was developed on a SUSE SLES7 2.4 kernel and the specific implementation here has been tested with the SUSE SLES8 2.4 kernel.

This document last updated on 28 January 2004 by William Scully, Senior Systems Programmer, Computer Associates.

Product, company, and service names referenced in this document may be copyright, trademark, or service marks of others.

The information provided herein is provided as is and has not been subjected to testing by Computer Associates. Use this information at your own risk and only after you test in your environment.

Planning

Overview

The plan is to create a two-pack system for each Linux system.

• The first pack is the so-called base volume, or basevol. Among these materials are those which are rarely changed and can be shared read-only among several servers simultaneously. This device is owned by the so-called base-server, or, in the examples below, userid LINXBASE.
• The second volume is the so-called guest volume or guestvol. These are the materials, which are unique to each Linux server and in directories which must be read/write. The userid used in the examples below is LNXGUEST.

The goal is to place as many programs (both the kernel code and vendor-supplied packages) as possible in the base volume so that it is leveraged over and over again. The remaining guest volume read/write disk space given to a typical Linux server can be far smaller as a result. Also, because fixes applied to packages installed onto the read-only disk are used by all servers, the maintenance burden is substantially decreased.

To implement this scheme requires only a modest change in the way a Linux server is created. Overall, the approach is to prevent the operating system from mounting the base volume materials in read/write mode during the boot process. This is easily accomplished because:

• When the Linux boot process starts all directories are in read-only mode.
• As the boot process continues, vendor-supplied scripts remount Linux directories in read/write mode, as needed.

It is easy to change the vendor-supplied boot-time scripts to avoid the read/write mounting of the Linux materials. It is also easy to create new boot scripts which mount in read/write mode only those directories which must be read/write. Thus we can easily pick and choose which directories (and subdirectories) are read-only and which are read/write. It is the use of bind-type mount commands which allow specific (sub-)directories to be R/O or R/W. The Redbook reference above includes a useful description of how bind mounts work. In particular you may want to read Sections 8.1 through 8.8.

Note that an objective is to hide, as much as possible, the fact that the server is using bind mounts. Using this implementation users who issue mount -l or cat /etc/mtab will see the same results as if the R/O server was booted with R/W directories. Only if the user issues a cat /proc/mounts will they see the details of what directories are mounted using the bind option.

Read-only and Read/Write Directories

The directories to be designated as read-only or read/write are selected based on commonly followed standards used by Linux. Please refer to Filesystem Hierarchy Standard for additional information.

The directories which will be included in the read-only base volume are:

Of special note is the directory in which is kept the Red Hat Package Manager (RPM) data base. This is typically located in /var/rpm. This directory is kept in R/O mode to prevent any attempt at using the RPM command from working. It is to be understood that all package or kernel maintenance which requires the use of the RPM command be performed by MIS. However, since any user can create an RPM data base and use the RPM command, we cannot stop someone from trying to circumvent this policy. But since the production RPM data base is on a R/O device, there's no chance MIS's data will be adversely affected.

The directories which will be included in the read/write guest volume are:

Note that the directory sizes quoted above are from a typical install of SUSE Linux Enterprise Server 8, Kernel 2.4, including kernel source.

There are a few other special-case or locally-defined directories as well:

Requirements

The Linux servers which are going to share materials were tested using the following standards:

• The swap space must be V-Disk and be located at virtual address 1F0 and be known as /dev/dasda1. The following attributes are recommended for the initial swap space sizing:

  * dsk 1F0 Swap          409600 512-byte blocks = 200M
  MDISK 1F0 FB-512 V-DISK 409600 MR
  

• The directory /tmp must be V-Disk and be located at virtual address 1F1 and be known as /dev/dasdb1. The following attributes are recommended for the initial temporary space sizing:

  * dsk 1F1 /tmp          40960 512-byte blocks = 20M
  MDISK 1F1 FB-512 V-DISK 40960 MR
  

• Disk address 200 and /dev/dasdq1 are reserved.
• The read-only base-volume root file system ("/" not "/root") must be located at virtual address 201 and be known as /dev/dasdr1. The device label must be BASVOL. The format of the base-volume is EXT3. The size of the disk should be 3338 3390 cylinders.
• The read/write guest-volume file systems must be located at virtual address 202 and be known as /dev/dasds1. The device label must be GSTVOL. The format of the guest-volume is EXT3. The size of the disk should be at least 150 3390 cylinders. Since end-users home directories are located on this disk you may wish to substantially increase the size of this disk from the suggested minimum.
• Additional space added to the server should be found in subdirectories located off /spacenn. These disks must be in virtual address range 203-21F and be known as /dev/dasdt1 through /dev/dasdav1.
• Networking is supported as QDIO VLAN at C00-C02 with portname VMLAN2. The device is known as ETH0. Therefore, a typical /etc/modules.conf might include (among others) the following records:

  # Aliases - specify your hardware
  alias eth0 qeth
  

  And a typical /etc/chandev.conf might include the following records:

  noauto
  qeth0,0xc00,0xc01,0xc02,0,0;add_parms,0x10,0xc00,0xc02,portname:VMLAN2
  

• A typical directory entry for the Linux base-volume server should be similar to:

  USER LINXBASE password 64M 256M G
  * 256M of memory is required to install SLES8,
  * but 64M is acceptable to run SLES8.
  ACCOUNT acntcode distcode
  MACHINE ESA
  IPL CMS
  CONSOLE 009 3215 T OPERATOR
  SPOOL 00C 2540 READER *
  SPOOL 00D 2540 PUNCH A
  SPOOL 00E 1403 A
  LINK MAINT 190 190 RR
  LINK MAINT 19E 19E RR
  LINK MAINT 19D 19D RR
  SPECIAL C00 QDIO 3 SYSTEM VMLAN2
  MDISK 191 3390 start-cyl 1 MR
  * dsk 1F0 Swap          409600 512-byte blocks = 200M
  MDISK 1F0 FB-512 V-DISK 409600  MR
  * dsk 1F1 /tmp          40960 512-byte blocks = 20M
  MDISK 1F1 FB-512 V-DISK 40960   MR
  MDISK 201 3390 start-cyl 3338 MR
  MDISK 202 3390 start-cyl  150 MR
  

• On the 191 A-disk, create file PROFILE EXEC to IPL the Linux 201 boot-disk.

Installation of a Base Server

Keeping the above requirements in mind, you must now conduct a SLES8 installation. At this point you're creating the "model" server, LINXBASE, which will hold the materials to be shared by several Linux servers. The disks owned by this model server are called the golden base volumes.

Conduct a normal installation, with the following additions:

• The SLES8 "Default System" installation can be used as the "Software Selection". If there are packages which you do or do not want installed ensure they are selected here. You may find it more convenient to install all the packages that any of your Linux servers may use. Remember, installing a package doesn't necessarily mean that any particular Linux server must be configured to utilize it.
• Additionally, the following vendor-supplied packages must be installed:
  • The cpint package, to allow Linux to communicate with the VM Control Program and determine the read-only or read/write status of disks. This package is found in on the support CD image in /suse/s390x.
  • The Regina package adds the Rexx scripting language to Linux. The basevol/guestvol tools are written in Rexx.
  • The Hessling Editor (the) package, to supply an Xedit-like editor.
• As always, remember the logon password for root. This password will be needed each time a guest server is create using the base server's materials.

Keep in mind that it is important to be careful not to make any unnecessary changes to the default installation. This is because the so-called "golden base volumes" will be replicated and used by several Linux servers. You should make only those changes which are to be used by all Linux servers. Any customization changes needed by a single Linux server can (and will) be made later, on that server's private (that is, non-shared) disk(s).

Post-Installation Modifications

After the standard SLES8 installation is complete, a few minor local modifications are needed and are discussed below.

• Since FBA disk is used for swap space and /tmp you will need to issue the following commands:

  insmod dasd_fba_mod
  mkinitrd
  zipl
  

  This step is performed only once on the basevol system after the initial install of SLES8.
• The server should be enabled to handle a CP SIGNAL SHUTDOWN. This requires the following statement to be added to /etc/inittab:

  ca:12345:ctrlaltdel:/sbin/shutdown -t1 -h now
  

• Ensure file /etc/fstab includes records similar to:

  /dev/dasda1     swap                      swap            defaults   0   0
  /dev/dasdb1     /tmp                      ext2            defaults   0   0
  #    dasdq1     reserved for future use
  /dev/dasdr1     /                         ext3            defaults   1   1
  #    dasds1     basevol/guestvol use
  #    dasdt1     is the first available device....
  

  Note that swap will be formatted and activated via /etc/init.d/boot.local. A filesystem check is never performed on /tmp since it is formatted and mounted in boot.local.
• Ensure file /etc/zipl.conf appears similar to:

  # Generated by YaST2 for SLES8
  [defaultboot]
  default=ipl
  
  [ipl]
  target=/boot/zipl
  image=/boot/kernel/image
  ramdisk=/boot/initrd
  parameters="dasd=1F0-21F root=/dev/dasdr1"
  
  [dumpdasd]
  target=/boot/zipl
  dumpto=/dev/dasd??
  
  [dumptape]
  target=/boot/zipl
  dumpto=/dev/rtibm0
  

  You may be tempted to configure the dasd= record as follows:

  dasd=1F0-200,201(ro),202-21F
  

  However note that this cannot be done from the guestvol system, since the boot device is R/O. And if this is done from the basevol system you will no longer be able to write to the boot device to maintain it in the future.

  As always, if you do change /etc/zipl.conf please ensure you rerun the zipl command.

• A typical /etc/init.d/boot.local should include the following statements:

  #! /bin/sh
  
  # Enable the so-called "timer patch"
  sysctl -w kernel/hz_timer=0
  
  # Create and activate swap space for the server
  mkswap /dev/dasda1
  swapon -a
  
  # Enable the "cpint" package, providing the "hcp" command
  /sbin/cpint_load
  /sbin/cpint-post-insmod
  
  # Create and mount the disk space used for /tmp
  mke2fs /dev/dasdb1
  mount /dev/dasdb1 /tmp
  chmod o=t /tmp
  chmod a+rwx /tmp
  

• Create file /etc/init.d/boot.bind-mount-etc. This is the first of two locally-written scripts which enable a basevol/guestvol Linux server. The source for the script is listed below:

  #!/usr/bin/regina
  /*---------------------------------------------------------------------+
     Check if we're running on the so-called "model" Linux server or
     if we're running on a Linux server using the model's disks R/O.
     If running on the model then we completely skip this tool so all
     the "standard" scripts run unmodified.  If we're running on a
     Linux server which is using the R/O root and other directories
     then this tool is utilized to set up the server correctly.
  +---------------------------------------------------------------------*/
  Trace Normal
  
  Say 'Computer Associates Bind-Mount (etc) script begins... .'
  
  /*---------------------------------------------------------------------+
    Enable the cpint "hcp" command, to allow VM's CP command to be issued
    from within userid root.  Note that cpint requires the Linux guest's
    storage be less than 2G in size, or the DIAGNOSE may barf.  Note that
    that these two commands will display error messages at boot-time since
    the root filesystem (and hence "dev") is read-only.  This is not a
    problem.
  +---------------------------------------------------------------------*/
  '/sbin/cpint_load'
  '/sbin/cpint-post-insmod'
  
  /*---------------------------------------------------------------------+
     Is the 201 disk R/W?  If so then this must be the systems programmer
     logging onto the Linux server which manages the "model" materials.
     Otherwise this must be one of the many Linux servers who are using
     these shared materials in R/O mode.  Bring up the server accordingly
  +---------------------------------------------------------------------*/
  save_rc = Popen( 'hcp QUERY VIRTUAL 0201', rec. )
  Select
  When Word( rec.1, 5 ) = 'R/W'
  Then Do
       Say 'Welcome to the Linux disk model userid!'
       Say 'All directories will be in R/W mode.'
       End
  When Word( rec.1, 5 ) = 'R/O'
  Then Do
       Say 'This server is running with shared disk support.'
       Say 'Many directories will be in R/O mode.'
       Call Enable_ETC
       End
  Otherwise Do
            Say 'Unexpected results checking disk status.'
            Say 'By default all directories will be in R/W mode.'
            End
  End
  
  /*---------------------------------------------------------------------+
     We're done, so inform and exit
  +---------------------------------------------------------------------*/
  Say 'Computer Associates Bind-Mount (etc) script ends.'
  Exit 0
  
  
  Enable_ETC: /*---------------------------------------------------------+
    Do what's needed to prepare and mount the various file systems in R/O
    or R/W mode
  +---------------------------------------------------------------------*/
  Procedure
  Say 'Forcing a filesystem check on R/W space before mount... .'
  'e2fsck /dev/dasds1'
  Say 'Attempting to mount R/W guest volume... .'
  'mount -n /dev/dasds1 /guestvol'
  Say 'Temporarily enabling directory /etc... .'
  'mount --bind /guestvol/etc  /etc'
  Return
  

• Create file /etc/init.d/boot.bind-mount-other. This is the second of the two locally-written scripts for a basevol/guestvol Linux server. The source for the script is listed below:

  #!/usr/bin/regina
  /*---------------------------------------------------------------------+
     Check if we're running on the so-called "model" Linux server or
     if we're running on a Linux server using the model's disks R/O.
     If running on the model then we completely skip this tool so all
     the "standard" scripts run unmodified.  If we're running on a
     Linux server which is using the R/O root and other directories
     then this tool is utilized to set up the server correctly.
  +---------------------------------------------------------------------*/
  Trace Normal
  
  Say 'Computer Associates Bind-Mount (other) script begins... .'
  
  /*---------------------------------------------------------------------+
     Is the 201 disk R/W?  If so then this must be the systems programmer
     logging onto the Linux server which manages the "model" materials.
     Otherwise this must be one of the many Linux servers who are using
     these shared materials in R/O mode.  Bring up the server accordingly
  +---------------------------------------------------------------------*/
  save_rc = Popen( 'hcp QUERY VIRTUAL 0201', rec. )
  
  Select
  When Word( rec.1, 5 ) = 'R/W'
  Then Do
       Say 'All directories will be in R/W mode.'
       End
  When Word( rec.1, 5 ) = 'R/O'
  Then Do
       Say 'Many directories will be in R/O mode.'
       Call Enable_Other
       End
  Otherwise Do
            Say 'Unexpected results checking disk status.'
            Say 'By default all directories will be in R/W mode.'
            End
  End
  
  /*---------------------------------------------------------------------+
     Disable the cpint "hcp" command.  If needed it will be enabled in
     the vendor-supplied scripts and/or boot.local.
  +---------------------------------------------------------------------*/
  '/sbin/cpint-post-rmmod'
  '/sbin/cpint_unload'
  
  /*---------------------------------------------------------------------+
     We're done, so inform and exit
  +---------------------------------------------------------------------*/
  Say 'Computer Associates Bind-Mount (other) script ends.'
  Exit 0
  
  
  Enable_Other: /*-------------------------------------------------------+
    Do what's needed to prepare and mount the various file systems in R/O
    or R/W mode
  +---------------------------------------------------------------------*/
  Procedure
  Say 'Creating temporary mount-table entries... .'
  'cat /proc/mounts > /etc/mtab'
  Say 'Overlapping R/O mount points with R/W materials... .'
  
  'mount -t ext3 --bind /guestvol/dev  /dev'
  'mount -t ext3 --bind /guestvol/home /home'
  'mount -t ext3 --bind /guestvol/root /root'
  'mount -t ext3 --bind /guestvol/opt/local /opt/local'
  'mount -t ext3 --bind /guestvol/usr/local /usr/local'
  
  /* the following required for force /var/lib/rpm to be R/O */ 
  'mount -t ext3 --bind /var /basevol/var'
  'mount -t ext3 --bind /guestvol/var  /var'
  'mount -t ext3 --bind /basevol/var/lib/rpm   /var/lib/rpm'
  Return
  

• The two locall-written boot scripts boot.bind-mount-etc and boot.bind-mount-other must be made executable to their owner, root, alone.

  chmod u+x boot.bind-mount-etc
  chmod u+x boot.bind-mount-other
  

• The two locally-written boot scripts must be run at start-up. Issue the following commands to create symbolic links in /etc/init.d/boot.d pointing to the scripts:

  ln -s ../boot.bind-mount-etc S04Boot.bind-mount-etc
  ln -s ../boot.bind-mount-other S05Boot.bind-mount-other
  

  Note the mixed-case use of the word "Boot". This ensures that these scripts are run before the vendor-supplied S04boot.lvm or S05boot.localfs scripts.
• Create one more local tool, /etc/init.d/make-guestvol:

  #!/usr/bin/regina
  Trace Normal
  
  Say 'Computer Associates Make-GuestVol begins... .'
  
  'mkdir /guestvol'
  'mkdir /basevol'
  'mkdir /basevol/var'
  
  'umount /guestvol'
  'mke2fs -j -b 4096 /dev/dasds1'
  If RC <> 0 Then Exit RC
  'mount /dev/dasds1 /guestvol'
  If RC <> 0 Then Exit RC
  
  dirs = '/dev /etc /home /var /root /srv /opt/local /usr/local'
  
  Do i = 1 To Words( dirs )
     dir = Subword( dirs, i, 1 )
     Say 'Copying 'dir' to guest volume... .'
     'tar -clpSf - 'dir' | (cd /guestvol ; tar -xpSf - )'
     If RC <> 0 Then Exit RC
  End i
  
  Say 'You must now update the boot.lvm and boot.localfs scripts'
  Say 'in /guest/etc/init.d so they do not mount root in R/W mode.'
  Say 'For boot.lvm, the change is at or near line 88.'
  Say 'Just one moment... .'
  'sleep 30 s'
  'the /guestvol/etc/init.d/boot.lvm'
  Say 'For boot.localfs the change is at or near lines 101 & 121.'
  Say 'Just one moment... .'
  'sleep 30 s' 
  'the /guestvol/etc/init.d/boot.localfs'
  
  Say 'Computer Associates Make-GuestVol ends.'
  
  Exit 0
  

  Ensure that only root can see and execute this tool:

  chmod u+rwx make-guestvol
  chmod go-rwx make-guestvol
  

• Make any final changes which you want reflected on all your Linux servers. For example, you may want to display a standard greeting to be shown to all users when they log onto Linux. In this case put that message in file /etc/motd now. You may also want to create a personal userid for yourself (or your peers or the staff in Operations) to allow you to gain access to the server at a future date.
• The last step is to copy the read/write directories off the boot device. Recall that the objective is to keep the boot device in read-only mode with a small subset of all the directories read/write, located on a disk separate from the boot device. To accomplish this issue the command:

  /etc/init.d/make-guestvol
  

  If you review the logic of the Rexx tool, shown above, you'll see that the tool:
  • Creates two new locally-defined subdirectories (/basevol and /guestvol)
  • Mounts and formats the guest volume (minidisk 202, also known as /dev/dasds1)
  • Copies the R/W directories from the base volume (minidisk 201, also known as /dev/dasdr1) to the guest volume
  • Prompts you to make three changes to two SUSE-supplied scripts: boot.lvm & boot.localfs
  The needed change is to replace:

  mount -n -o remount,rw /
  

  With:

  mount -n -o remount,ro /
  

  That is, do not remount the root device in read/write mode but rather leave it read-only. This same change is required once in boot.lvm and twice in boot.localfs at the line numbers 88 and 101 & 121. (Note that service from SUSE may change these exact line numbers in these vendor-supplied scripts.)

At this point you have completed the installation of Linux on the base server, LINXBASE, which owns the golden base volume. The guest server(s) can now be created and can utilize the DASD in shared R/O mode.

Implementation of a Guest Server

Now that the base volume has been created it is possible to exploit it. To do so create a Linux server which will utilize the read-only boot device, as well a its own copy of the necessary read/write materials. The steps to follow are detailed below. You must perform each of these steps every time you create a new Linux server:

• Create or update the server so that the VM directory entry matches the supported configuration:

  USER LNXGUEST password 64M 256M G
  * 64M is acceptable to run SLES8.
  ACCOUNT acntcode distcode
  MACHINE ESA
  IPL CMS
  CONSOLE 009 3215 T OPERATOR
  SPOOL 00C 2540 READER *
  SPOOL 00D 2540 PUNCH A
  SPOOL 00E 1403 A
  LINK MAINT 190 190 RR
  LINK MAINT 19E 19E RR
  LINK MAINT 19D 19D RR
  SPECIAL C00 QDIO 3 SYSTEM VMLAN2
  * dsk 1F0 Swap          409600 512-byte blocks = 200M
  MDISK 1F0 FB-512 V-DISK 409600  MR
  * dsk 1F1 /tmp          40960 512-byte blocks = 20M
  MDISK 1F1 FB-512 V-DISK 40960   MR
  MDISK 191 3390 start-cyl   1 MR
  LINK LINXBASE 201 201 RR
  MDISK 202 3390 start-cyl 150 MR
  

  The server will find the kernel and the majority of its applications on the read-only 201 disk. Since these materials are never updated by the guest server a large number of Linux servers can share the materials on this disk, saving DASD for more valuable uses.

  The size of the 202 disk must match the size of the 202 disk which was used to create the model server. In the example shown here the 202 disk is the minimum size, 150 cylinders. Note that is enough space for the basic SLES8 materials. Any additional space for user home directories is above and beyond this value. You should also understand that the size of this device can be increased, at a later date, if needed, by following procedures documented on the LinuxVM.org web site.

• Log onto the Linux guest server. On the 191 A-disk, create file PROFILE EXEC to IPL the Linux 201 boot-disk.
• Recall that during the post-installation step the tool make-guestvol copied several directories to the base-server's 202 disk. A copy of this disk will be created on the guest server's 202 disk now. Use the CP LINK command to gain access to the model's 202 disk in R/O mode:

  CP LINK LINXBASE 202 1202 RR
  

  Use DDR to copy model's 202 to the new server's 202 disk.

  DDR
  SYSPRINT CONS
  INPUT 1202 DASD
  OUTPUT 202 DASD
  COPY ALL
  

  Detach the model's 202 disk when done.

  CP DETACH 1202
  

• Run the PROFILE EXEC which will start the new Linux server.

At this point you're booting an exact copy of the model base-volume server. All the attributes of the model server are still in effect. For example, the IP address and host-name used by the base server are still as configured when Linux was first installed. For this reason the base-volume server (LINXBASE) cannot run simultaneously with the guest-volume server (LNXGUEST).

Final Customizations

The files which Linux uses to uniquely identify this server are located on R/W disk space, and in particular in directory /etc. In the next few steps you'll perform the final customization for the guest server, by changing these files.

• Allocate an IP address for the new server. This address must be on the same Guest LAN as the original base-server. (After the server is completely operational you can change the IP address, or even the Guest LAN used, if you see fit.)
• Telnet to the IP address or DNS name of the base server. For example: linxbase.ca.com
• Logon as root. Specify the logon password for root which you used when you first installed Linux onto LINXBASE.
• Start YaST. Configure the IP address and host-name for the new Linux server. Exit from YaST.
• Shutdown Linux using the reboot option:

  shutdown -r now
  

• Telnet to the guest Linux server using its proper name. For example: lnxguest.ca.com
• Logon as root once more. Change root's password:

  passwd
  

  This password you will give to the owner of the server.

Above are the minimal final customizations. However since directories such as /etc are read/write and since virtually all the customization of Linux is controlled by files in /etc, you can, if need be, make each server completely unique.

Note: Should you find a configuration file (or virtually any other file) which resides in a directory on the R/O shared disk space, you can use a Linux symbolic link to redirect that file from its current location to another R/W directory, such at /etc.

Servicing the Base Server

Please check back later on this web site for an updated version of this document which describes the service procedure to follow when using this DASD-sharing technique.