Enterprise Jumpstart

(C) Copyright Scott Howard, 2001-02
scott@doc.net.au

This document describes how to setup an Enterprise-wide Jumpstart setup.

The goals of this setup are :

  1. To allow jumpstart to work well in a large-scale, multiple subnet (possibly distributed) environment. In particular, to remove the need to have a rootserver on each network.
  2. To make some functionality improvements to Jumpstart.
  3. The create a post-install script which is more intuitive than normal (by working around the need to prefix all operations with /a/)

The environment described in this document consists of the following :

Generally you would have one Jumpstart server per site. eg, one server for Sydney, one for London, one for NY, etc.

Setting up the Jumpstart Server

You will need a partition with at least 1Gb free. I'd suggest using a new partition, and making it at least 2Gb in size. Although this partition/directory can be located anywhere, this document presumes that it is either mounted on /jumpstart, or is symlinked to that directory.

It is also presumed you are setting up a Jumpstart server for Solaris 8. If you are using a different version of Solaris then some minor details will be different, but the general process will remain the same.

First we need to copy the files from the Solaris distribution CDs. This is handled by scripts which are supplied with Solaris, and which are located in the Solaris_8/Tools directory on slice 0 of the Software 1 of 2 CD-ROM.

# cd /cdrom/cdrom0/s0/Solaris_8/Tools
# mkdir /jumpstart/sol8
# ./setup_install_server /jumpstart/sol8
This will copy all of the relevant files from the Software 1 of 2 CD into the directory /jumpstart/sol8. The time this takes to complete will depend on the speed of your CD-ROM and host, but will probably be somewhere between 10 minutes and a few hours.

Once this has completed, the files from the Software 2 or 2 CD need to be added to the image. Again this is done using a script which is located on the CD-ROM.

# cd /cdrom/cdrom0/Solaris_8/Tools
# ./add_to_install_server /jumpstart/sol8/
Again this will take quite some time to copy.

Once the files have been copied you need to share the directory. Although you only need to share a smaller part of it, I find it easier just to share the entire /jumpstart directory. This directory must be shared as read-only.

This is done by adding the following to /etc/dfs/dfstab :
share -F nfs -o ro,anon=0 -d "Jumpstart" /jumpstart

If your machine is already running as an NFS server (ie, nfsd is running), run "shareall" for this change to take effect.
If your machine is not already an NFS server, start the NFS server processes by running "/etc/init.d/nfs.server start"

Check tftp is enabled in inetd.conf by making sure that the line below exists and is uncommented :

tftp    dgram   udp6    wait    root    /usr/sbin/in.tftpd      in.tftpd -s /tftpboot
Now that we've got our Jumpstart server setup, lets test it! We'll do this by attempting to boot a client from our Jumpstart server. At this stage the client will need to be on the same network as our server, although this will change later.

Setup the new clients name in whatever database you use for naming (NIS/NIS+/hosts, etc). If you are using DNS it's a good idea to add the client details to /etc/hosts as well as (or instead of) DNS.

Setup the client details on the Jumpstart server :

# cd /jumpstart/sol8/Solaris_8/Tools
# ./add_install_client -e 8:0:20:22:33:44 clientname sun4m
The -e option is the Ethernet (MAC) Address of the client. This can be obtained by using the "banner" command at the OK prompt. The next option is the hostname of the client, followed by the architecture, which is either sun4m (SS4/5/10/20), sun4d (SS1000/2000), or sun4u (any Ultra machines)

At this point you should be able to boot your client, using the "boot net" command!

<#0> ok boot net
Boot device: /iommu/sbus/ledma@f,400010/le@f,c00000  File and args:
20800
SunOS Release 5.8 Version Generic_108528-07 32-bit
Copyright 1983-2001 Sun Microsystems, Inc.  All rights reserved.
[..etc..]
We now have a Jumpstart server setup and working, although it's use is limited. Installing using this server has gained little over installing from a CD-ROM, other than removing the need to actually put the CD-ROM in the drive!

The next step is to start automating the process...

Jumpstart Server Configuration

The first thing to do is to setup a sysidcfg file so that Jumpstart won't need to ask us some of the mundane questions each time.

This file must be called "sysidcfg", and for reasons which will become clearer later, I choose to put this file in the directory /jumpstart/sol8/conf/server/sysidcfgXX where XX refers to the netmask of the network we're on. eg, if the network we're on has a netmask of 255.255.255.0 (also known as a /24 network), then XX will be 24.
Other common values are :
255.255.0.016
255.255.224.019
255.255.252.022
255.255.255.024
255.255.255.22427
255.255.255.240   28
In my case, I'm on a 255.255.255.224 network, so the file is /jumpstart/sol8/conf/server/sysidcfg27/sysidcfg

The contents of this file will vary depending on your setup, but will be something like :

/jumpstart/sol8/conf/server/sysidcfgXX/sysidcfg 

timezone=Australia/NSW
root_password=xxxxxxxxxxxxx
system_locale=en_AU
timeserver=localhost
name_service=NIS {domain_name=xxxxxx name_server=nisserv (yyy.yyy.yyy.yyy)}
network_interface=primary {protocol_ipv6=no netmask=xxx.xxx.xxx.xxx}
security_policy=none
The "root_password" entry is an encrypted form of what you want the root password set to. The easiest way to get this is to copy it from the /etc/shadow file on an existing machine.

You can find details of all of the other settings you can put into this file at http://docs.sun.com/ab2/coll.214.7/SPARCINSTALL/@Ab2PageView/1261

If you've got multiple network masks across your enterprise you'll need to setup multiple copies of this file in different directories - one for each netmask.

Next we need to re-setup our client to use this file, so :

# cd /jumpstart/sol8/Solaris_8/Tools
# ./add_install_client -e 8:0:20:22:33:44 -p servername:/jumpstart/sol8/conf/server/sysidcfg27 clientname sun4m
This is similar to what we used before, only we've added a pointer to our sysidcfg file (replace servername with the name of your Jumpstart server)

If we again boot out client with "boot net" you should find that it will ask you a lot less questions that last time. Rather than asking you questions about your environment, it should now just ask you questions about the server you are wanting to install.

So obviously the next step is to setup a profile for how we want our server built!

To do this we need to create a few files, all in the /jumpstart/sol8/conf/server directory

The first of these is our actual "profile" file. This file can be called anything you want, however I recommend a nameing scheme such as server_prof_4gb which designated that this is a "server" build for a "4gb" boot disk.

In this file we have something along the lines of : /jumpstart/sol8/conf/server/server_prof_4gb 

install_type initial_install
system_type server
cluster SUNWCXall
partitioning explicit
filesys rootdisk.s0 2000 /
filesys rootdisk.s1 1000 swap
filesys rootdisk.s3 3 unnamed
filesys rootdisk.s4 1000 /var
This profile tells Jumpstart that we wish to install a "Full Software + OEM" install (SUNWCXall), with a 2000Mb root partition on slice 0, 1000Mb swap on slice 1, an unused 3Mb partition on slice 3 (eg, for an SDS state database), and a 1000Mb partition on slice 4 for /var.

Full details of all options are available at http://docs.sun.com/ab2/coll.214.7/SPARCINSTALL/@Ab2PageView/8327

The next file we need to create (in the same directory) is called "rules", and is used to select which profile should be used for the install of each server.

For now this file only needs a single entry, listing the name of the client to install, and the profile to use.

/jumpstart/sol8/conf/server/rules 

hostname clientname			-	server_prof_4g		-
Further on this file can be made more generic so that you do not need to edit it for each new client. Full details can be found at http://docs.sun.com/ab2/coll.214.7/SPARCINSTALL/@Ab2PageView/7328 but a typical entry might be something like :
disksize rootdisk 3500-4500     	-	server_prof_4g		-
Jumpstart itself doesn't actually use the "rules" file, but instead a processed version of this file called "rules.ok". In order to create this file we need the "check" script supplied with Jumpstart. Copy this file into this directory with :

# cp /jumpstart/sol8/Solaris_8/Misc/jumpstart_sample/check /jumpstart/sol8/conf/server/
We can then use this script to confirm our profile is setup correctly, and create the rules file :
# ./check
With the profile setup on the server, we need to point the client config to this file by running add_install_client with yet another option :

# cd /jumpstart/sol8/Solaris_8/Tools
# ./add_install_client -e 8:0:20:21:45:e0  \
	-c servername:/jumpstart/sol8/conf/server  \
	-p servername:/jumpstart/sol8/conf/server/sysidcfg27  \
	clientname sun4m
This time a "boot net" should only ask at most one question - the terminal type you are using (and then only if you're on a serial console)! The rest of the install should complete without any manual intervention! During the install you should see output similar to the following :

Searching for JumpStart directory...
Using rules.ok from xx.xx.xx.xx:/jumpstart/sol8/conf/server.
Checking rules.ok file...
Using profile: server_prof_4g
This will confirm that your install is using the correct setup files.

We've now managed to completely automate our install!

Of course, there's usually more to setting up a machine than just installing the operating system. We usually need to do a fair amount of customisations, and thats where post install scripts come into it!

Pre-Install Scripts

Before we actually get to a Post-Install script, we'll do some work to get around a problem which can often occur when our network port is running at 100Mbps.

If your host is connected at 100Mbps you will often have problems with the duplex setting (ie, full or half duplex) getting confused between the host and the switch. If this happens, the performance of your network will drop off significantly.

The normal symptom of this problem is that your 100Mbps port on your host will be running at Half Duplex. We can check for this before we start the actual install by using a "pre-install" script.

This script lives in the same directory as the profile itself. The only limitations are that it must be a shell script, and it must return an error code of 0 (any other error and the install process will abort)

A sample pre-install script, called "generic.begin" is listed below. This will check each of your 100Mbps ports, and if any are running at half duplex it will disable auto-negotiation (which will in effect change them to full duplex).

/jumpstart/sol8/conf/server/generic.begin 

#!/bin/sh

for port in hme eri; do
        ndd -set /dev/${port} instance 0 2>&1 > /dev/null
        if [ $? -eq 0 ]; then
                PORT_STATUS=`ndd -get /dev/${port} link_status 2>/dev/null`
                PORT_SPEED=`ndd -get /dev/${port} link_speed 2>/dev/null`
                PORT_MODE=`ndd -get /dev/${port} link_mode 2>/dev/null`
                if [ "$PORT_STATUS" = "1" ] && [ "$PORT_SPEED" = "1" ] && [ "$PORT_MODE" = "0" ]; then
                        ndd -set /dev/${port} adv_autoneg_cap 0
                        echo Setting ${port}0 to 100FDX
                        sleep 1
                fi
        fi
done

exit 0
Copy this file into the /jumpstart/sol8/conf/server directory, and chmod it to 755.

In order for our install process to actually use this file, we need to edit the "rules" file and change the first dash for our entry to the name of our script. ie :
/jumpstart/sol8/conf/server/rules 

hostname clientname			generic.begin	server_prof_4g		-
As we've made a change to the rules file, we need to run "check" to update the "rules.ok" file.

If we reboot the server now, we'll see a few new lines in the output :

Checking rules.ok file...
Using begin script: generic.begin
Using profile: server_prof_4g
Using finish script: generic.finish
Executing JumpStart preinstall phase...
Executing begin script "generic.begin"...
Begin script generic.begin execution completed.
Searching for SolStart directory...
Post-Install Script

The post install script is where all of the customisations to the standard Solaris build take place. It can be as simple or as complex as you like, depending on your environment.

The default Jumpstart setup means that your post install script runs with the miniroot mounted on /, and your new machine mounted under the /a directory. Unfortunately this makes things more complicated as you need to prefix everything you do with /a/. This ranges from annoying (Having to remember to modify /a/etc/passwd rather than /etc/passwd) to impossible (3rd party scripts which do not support running on different root directory).

However, there is another option. By shuffling mounts a little, and using chroot, we are able to end up with our new machine mounted on /, and our jumpstart directory mounted under /mnt.

The following post-install script will do exactly that (for Solaris 8. It's a little different in earlier versions of Solaris) :

/jumpstart/sol8/conf/server/generic.finish 

#!/bin/sh

if [ -d /a/etc ]; then
  BASEDIR=/a
else
  BASEDIR=/
fi

mount -F lofs /cdrom $BASEDIR/mnt

if [ $BASEDIR != / ]; then
	mount -F lofs /proc $BASEDIR/proc
fi

# Unmount mnttab so we can remount it in the chroot. What we get there will
# be wrong, but at least it's something.

umount /etc/mnttab

chroot $BASEDIR /mnt/doc/`basename $0`

exit 0
This script will look after the mounting as discussed, and then run the script under /mnt/doc/ (ie, /jumpstart/sol8/doc on our Jumpstart server) with the same name.

Save this script as "generic.finish" in the profile directory, add it to the rules file, and then run ./check

Your rules file should now look like this :

hostname clientname		generic.begin	server_prof_4g	generic.finish
The format and contents of the "real" finish script, /jumpstart/sol8/doc/generic.finish is entirely up to you. The format I use for this script is :
/jumpstart/sol8/doc/generic.finish 
#!/bin/sh

#########################################################
# Work out what type of machine we are
#
MACHINETYPE=`echo $SI_CLASS | cut -b 1-6`
if [ "$MACHINETYPE" = "server" ]; then
        echo "##########################################"
        echo "### Starting Server Post-Install Script ###"
        echo "##########################################"
elif [ "$MACHINETYPE" = "workst" ]; then
        echo "###############################################"
        echo "### Starting Workstation Post-Install Scipt ###"
        echo "###############################################"
else
        echo "Unknown machine type (SI_CLASS is $SI_CLASS) - not running post in
stall script"
        exit 0
fi

#
# Code common to all platforms goes here
#

#########################################################
#########################################################
# Customizations specific to classes of machines
# (ie, server v's workstation)
#
[ "$MACHINETYPE" = "server" ] && /mnt/doc/config_server
[ "$MACHINETYPE" = "workst" ] && /mnt/doc/config_workst


#
# Code common to all platforms goes here
#

#########################################################
echo "### Done ###"

exit 0
This allows us to use the same post-install script as a basis for all type of machines (servers, workstations, etc), and just branch for things specific to each platform.

So what do you want in your post-install script? The answer is largely going to depend on what modifications you make to the default Solaris build in your environment. However, there is one thing which you should always want to do - apply the latest Recommended Patch Cluster from Sun.

Attempting to do this results in a few unexpected problems - the patch install processes requires that we have /etc/mnttab available. Unfortunately, the fact that we are working within a chroot environment causes problems for mnttab

/etc/mnttab
Under earlier version of Solaris, /etc/mnttab was a standard file, and thus it was easy to create a file was which enough to keep patchadd happy. Under Solaris 8 /etc/mnttab changed to be a virtual file - basically something which looks like a file, but is actually a link into the kernel. Replacing it with a plain file is not enough for patchadd to work correctly.
The easiest way to get around this is to unmount mnttab before we chroot, and then re-mount it after we've chroot'ed. If you look back to our original "generic.finish" script you'll see that we unmounted /etc/mnttab there, which just leaves us to remount it near the top of our new finish script (the one which lives in /jumpstart/sol8/doc/) :

# Remount mnttab
/sbin/mount -F mntfs mnttab /etc/mnttab
Having done this workaround, applying the patches is as easy a changing to the directory where they are mounted (which must be under the /jumpstart/sol8/doc/ directory). eg :
########################################################
echo "### Installing Recommended Patches"
cd /mnt/doc/patches
./apply_patches -q -nosave
A more complete sample post-install script is available here. It consists of 3 files, generic.finish, config_server and config_workst. generic.finish is called for all clients, and then spawns one of config_server or config_workst, depending on the type of host being installed.

Enterprise Wide Jumpstart

The biggest problem with attempting to use Jumpstart in a medium to large network is the requirement to install a Jumpstart server on each subnet. As well as requiring a reasonable amount of disk space on a machine, this also results in high administrative overhead in installing and maintaining Jumpstart servers.

Fortunately it's possible to get away with only a single,centralized Jumpstart server, and a very small boot server on each subnet - it just needs a few modifications to our setup. The easiest way to set this up is to first do it on the same server as our Jumpstart server, and then just copy the config over to as many other servers as you want.

Firstly, create a new directory to house the boot server. As this is only small (around 1Mb) and will exist on a number of machines, it shouldn't be on it's own partition - personally I use /opt/jumpstart. Within the directory create a directory for each operating system you want this to be a boot server for.

# mkdir /opt/jumpstart
# cd /opt/jumpstart
# mkdir sol8
# cd sol8
Next we need to copy the standard Jumpstart add_install_client and rm_install_client scripts from the Jumpstart server.

# cp /jumpstart/sol8/Solaris_8/Tools/{add,rm}_install_client /opt/jumpstart/sol8/
Unfortunately the default add_install_client script does not allow for a client to mount it's miniroot from a server other than the server it is run from. This patch will add a -r (root server/patch) option too allow this. This patch can be applied by changing to the directory containing add_install_client and running :
# cd /opt/jumpstart/sol8
# patch add_install_client < add_install_client.patch
In order for add_install_client to work it needs a few additional files. Some of these are just to trick the script into thinking you've got a full miniroot installed on this machine (otherwise it will return an error and fail), the others are the kernel files that your jumpstart client will need to tftp in order to boot. These can be copied over with the following commands :
# cd /opt/jumpstart/sol8
# mkdir Boot
# cd Boot
# mkdir -p .tmp_proto platform/sun4u platform/sun4m platform/sun4d
# (cd /jumpstart/sol8/Solaris_8/Tools/Boot && tar cf - usr/platform/sun4{d,m,u,us}/lib/fs/nfs/inetboot ) | tar xfp -
We're now in a position to run the add_install_client with the right options (including -r), and it will set things up correctly. However, as there's a number of options to pass, we can setup a wrapper script to make things easier.

The jumpstart script, which should be saved in /opt/jumpstart, gives us a very easy command-line to setup a new server :

Usage: /opt/jumpstart/jumpstart -e ethernetid -t host_type -v os_ver -s js_server client_name platform_grp
  eg : /opt/jumpstart/jumpstart -e 0:8:20:12:34:56 -t server -s servername clientname sun4u

       host_type is one of  server, workst
       os_ver is one of  2.6, 7 or 8
       js_server is the server to use as a root/install server
       platform_grp is one of sun4c, sun4d, sun4m or sun4u
As the add_install_client script (and thus the jumpstart script) put details into a number of files/directories, it's good to have an automated process to clean them up. The "cleanup_all" script achieves this.

This uses rm_install_client, so we need to link in a copy of this from one of the versions we have. In this case we only have one, so :

# ln -s sol8/rm_install_client /opt/jumpstart
We then need a cronjob to run the cleanup_all script (say) once a week. As this needs to be run as root, it needs to go into root's crontab :
# Cleanup jumpstart files once a week
0 0 * * 1       /opt/jumpstart/cleanup_all
Once all this has been done the /opt/jumpstart directory can be tar'ed up and copied to servers on other subnets (or rdist'ed, or rsync'ed, or possibly even NFS/auto mounted)

Multiple Jumpstart Servers

If you've got a centralised environment, with everything linked with a reasonable speed network, one Jumpstart server is probably all you need. However, if you've sites at the other end of a WAN, Jumpstarting over the slower WAN links will be slow, and will have a serious impact on the performance of the link.

For these reasons you might want to setup local jumpstart servers at these remote sites, and have the jumpstart script do a lookup to try and find the correct server to use for the network it's being run on.

In order to do this, create a file "Jumpstart_Servers" in the /opt/jumpstart directory with entries in the format "NETWORK-OSVER SERVERNAME" where NETWORK is the network address listed in the "Net/Dest" column of a "netstat -in" and OSVER is the Operating system version in the form "sol2.6", "sol7", or "sol8". eg :

10.10.0.0-sol8		jsserver1
10.10.0.0-sol2.6 	jsserver1
10.20.0.0-sol8		jsserver2
This means that jsserver1 should act as the jumpstart server for Solaris 2.6 and Solaris 8 on the network 10.10.0.0, whilst jsserver2 is the server for Solaris 8 only on the 10.20.0.0 network.

The details above can also be put into a NIS map called "jsservers" which will normally make distributing changes easier than a flat file.

Modifications to Jumpstart Itself

There's a number of modifications we can make to the Jumpstart process itself to improve it. Which of these you choose to do will depend on you!

Telnet Access to a Building Host

If for some reason you feel the need to be able to telnet (ftp, etc) into your machine whilst it's being Jumpstarted, it can be arranged!
However, before you do this, it's worth putting a little thought into the consequences. It someone is able to gain un-authorised access to the machine whilst it is being built it's exactly the same as if they gain access to it once it's built. For this reason, you need to make sure that the security measures you take for the Jumpstart miniroot are the same you take for a fully-built machine (ie, secure passwords, etc)

So if I haven't talked you out of it yet, you can enable everything which runs from inetd by editing /jumpstart/sol8/Solaris_8/Tools/Boot/sbin/sysconfig and uncomment the line :

/usr/sbin/inetd -s
If you're using NIS, anyone with a NIS account will be able to log into the machine whilst it is being built. If you're not using NIS, you'll need to add a new account. To do so you'll need to copy the entries from your passwd and shadow files into the miniroot files in /jumpstart/sol8/Solaris_8/Tools/Boot/etc/ You'll also need to give root a password in miniroot/etc/shadow if you want to be able to su to root.

Confirmation Prompt

We worked hard above to get the entire Jumpstart process to work in a completely hands-off manner - but is this what we really want?
What happens if you accidentally boot a machine off the network? This can be a real problem if for example you've got "diag-device" set to "net", and you turn on the diag-switch (the machine will then boot from the network rather than disk like it normally would)

By default Jumpstart will act the same wether you put "boot net" or "boot net - install". We can disable this by editing /jumpstart/sol8/Solaris_8/Tools/Boot/sbin/sysconfig The last line of the file is "exec /sbin/suninstall", but we can replace this line with :

if [ -f $INSTALLBOOT ]; then
        echo
        echo "This will install a new Solaris 8 build on this machine, deleting"
        echo "the current install".
        echo
        OK=""
        while [ -z "$OK" ]; do
                /usr/bin/echo 'Do you wish to continue? (y/n) \c'
                read resp
                case $resp in
                        y*|Y*)  OK=1;;
                        n*|N*)  echo Shuttingg down. Please wait..."
                                halt;;
                esac
        done

        exec /sbin/suninstall
fi

echo ''
echo ''
exec /bin/csh
Now if we run "boot net" we'll get a shell, but if we run "boot net - install" we'll get a prompt asking us to confirm that we want to install a new machine.

Of course, we may not want to always want to be prompted. We can get around this by allowing another option to be passed on the "boot" line - lets say the option "y" implying that we want to answer yes to this question.

We need to edit two files to make this work - sysconfig again, plus rcS in the same directory.

In rcS we need to look for the section which processes the boot options. To do this, search for "getbootargs" and you'll find a case statement which checks the boot opts and creates empty flag files for each that exists. eg, for "w" (do not start openwin) we have :

                w)
                        cat < /dev/null > /tmp/.nowin
                        shift
                        ;;
Add a new entry below this one with the following :
                y)
                        cat < /dev/null > /tmp/.noprompt
                        shift
                        ;;
We then need to make changes in sysconfig to read this. Near the top we have :
INSTALLBOOT=/tmp/.install_boot
Add an entry below this one for our new flag file :
NOOKPROMPT=/tmp/.noprompt
And further down in our new code from above we already have :
	OK=""
	while [ -z "$OK" ]; do
		/usr/bin/echo 'Do you wish to continue? (y/n) \c'
We need to change this to :
        OK=""
        if [ -f $NOOKPROMPT ]; then
                OK="y"
        fi
        while [ -z "$OK" ]; do
		/usr/bin/echo 'Do you wish to continue? (y/n) \c'
Screen

One thing which is nice about the OpenWindows-based install is that you can open additional windows whilst the install is happening. This isn't quite as simple from a console-based install, but it can be done using GNU Screen (http://www.gnu.org/software/screen/).

The modification to rcS for this is :

                screen)
                        cat < /dev/null > /tmp/.screen
                        shift
                        ;;
and for sysconfig :
USESCREEN=/tmp/.screen
and above :
        exec /sbin/suninstall
put :
       if [ -f $USESCREEN ]; then
                TERM=`tail -1 /etc/.sysIDtool.state`
		export TERM
                exec /usr/bin/screen -s /bin/csh /sbin/suninstall
        fi
Of course, you'll also need to get yourself a copy of screen, and put it into /jumpstart/sol8/Solaris_8/Tools/Boot/usr/bin/screen