 AIX
Network Install Manager
Bill Wood
When we replaced our old hardware platform with IBM AIX RS/6000
systems, a main concern was controlling how our new systems would
be installed. With the old platform, we ran into several instances
where the software (moving from our development group to QA and
into production) was not the same, which caused numerous build breaks.
We chose the AIX Network Installation Management (NIM) setup, because
we needed to ensure consistent, standard OS installations between
systems within our four separate data centers. Our company also
has a security rule that doesn't allow any one machine access
to all networks within our organization. Our four separate data
centers have four separate networks with no connection between them,
and each network is segmented into either inside or outside the
firewall.
NIM Resources
NIM allows you to customize installations and maintain clients
on the network from a centralized location (the NIM master) or the
NIM client itself. The master contains the NIM database and can
serve resources. Resources in NIM are files or directories containing
data that NIM will use to install, customize, and maintain NIM clients.
A NIM client is any machine configured and defined in the NIM database.
Some key NIM resources used in our setup are:
- Licensed Program Product Source Directory (lpp_source):
This directory contains backup file format (BFF) images, which
AIX installp uses to load software. One way to understand the
role of the lpp_source directory in a BOS installation
is to compare it to all the installation images needed to support
any configuration (specifically different device configurations)
along with a base core set of software (called simages) that are
on the BASE installation CDs. We created a base 433 lpp_source,
multiple lpp_sources containing different maintenance levels,
and separate lpp_sources for our 32-bit and 64-bit third-party
application software.
- Shared Product Object Tree (SPOT): This directory is created
from an lpp_source and is equivalent in content to a
/usr file-system on AIX. The purpose of a SPOT in a NIM installation
is similar to the boot images and BOS installation scripts (bi_main,
rc.boot, and rc.bosinst) on volume 1 of the BASE
install CD. The SPOT must contain support for all boot environments
(platform, network type, kernel type). We created several different
SPOTs for the different data centers and maintenance levels we
use to support our systems.
- bosinst_data: This data file contains information that
drives the BOS install (e.g., prompt vs. no-prompt, which disk
to install the OS on, and the type of installation (Overwrite,
Preservation, or Migration) to name a few). First, we created
separate bosinst_data resources for each machine type (S80,
H70, B50, M80, P680, and 43P). Then, by specifying two disks to
target in our bosinst_data resource and specifying copies
in the image_data resource, we could set up mirroring during
the initial load.
- image_data: This data file contains information about
the characteristics of the OS being installed. For example, it
includes the size of file systems, whether or not to mirror, and
whether or not to disk stripe. We created separate image_data
resources for each machine type (S80, H70, B50, M80, P680 and
43P).
- Script: This is a customization file the systems administrator
can create, which executes near the end of the BOS installation.
You can execute multiple NIM script resources during a BOS installation.
If you need to control the order of execution, I recommend putting
it all in one large script. We created one NIM script to set up
the OS environment and hardware parameters, configure certain
system files, set up security, install third-party software, create
and set up dump devices, customize paging space to correspond
to the system RAM, and help create a complete inventory by machine
type and serial number. I've included a copy of this script
at the end of this article (Listing 1). All listings for this
article are available from the Sys Admin Web site: http://www.sysadminmag.com
- Installp_bundles: This data file contains a customized
list of additional software to install after the base AIX software
is loaded. If you have different configurations that you need
to duplicate on a repeatable basis, this resource is very useful.
In our environment, we have different OS software requirements
for development, QA, and production above the minimal AIX software
needed to support different hardware systems. The easiest way
to facilitate and maintain these different requirements, which
need to be consistent, is to use installp_bundles.
- mksysb: This is a backup archive file that contains
a system image of rootvg. Because of our network security
restrictions (no one machine could be connected to all the networks
within our organization), we used mksysb and savevg
tapes to replicate the NIM master to the other data centers. If
we had one machine connected to the different data centers, we
could have used NIM to replicate and update the NIM masters in
the different data centers by BOS-installing a NIM mksysb
resource and using a NIM script to restore the other volume group
data.
- mac_group: This is a logical grouping of machine types
(standalone, diskless, or dataless) that enables the systems administrator
to target one or more machines with a single command or NIM operation.
We did not use this feature, but we could have taken advantage
of this by grouping all like systems and like configurations to
install to more than one machine at a time.
Working with NIM
We used the 43P systems as our NIM masters for each data center
because they could complete remote installations of machines or
be moved and directly connected to a server for OS installations.
These NIM masters were also designated as the resource servers in
our environment. To ensure consistency and standardization of each
NIM master (for the different data centers), we created a standard
NIM master machine, which we cloned. We made a stacked tape containing
a mksysb image and a savevg image of the standard
NIM master to sync up and update the other NIM masters. Here are
the commands we ran on the standard NIM master to create this stacked
single tape:
1. mksysb -i /dev/rmt0
2. tctl -f/dev/rmt0.1 fsf4
3. savevg -i -m {volume_group_name} -f/dev/rmt0.1
4. mt -f/dev/rmt0 rewind
To restore the tape to the other NIM masters, we did the following:
1. Booted and restored the mksysb image from the stacked
tape
2. tctl -f/dev/rmt0.1 fsf4
3. restvg volume_group_name
These NIM servers were placed throughout our environment and are
used by multiple operators with different training levels. Thus,
we developed a strict directory tree for storage of resources on
the NIM server, but this was our own restriction, not a restriction
created by NIM. I recommend creating either one large separate filesystem
or a few file systems that will contain your NIM resource files.
Also, use naming conventions that make sense to you in terms of
the resource name and directory structure. The following is a breakdown
of our directory structure and a description of what is contained
in the directory:
Master directory of bundle lists:
/export/installp_bundles
Master directory of all NIM resources:
/export/nim
lpp sources for use by the NIM server during SPOT generation
and system installation:
Installations/export/nim/aix433
32-bit and 64-bit compiled third-party applications:
/export/nim/aix433/32bit-lpp and
/export/nim/aix433/64bit-lpp
Master bosinst.data files for each server type:
/export/nim/bosinst
Master image.data files for each server type:
/export/nim/imagedata
In-house customization scripts:
/export/nim/scripts
Master OS system files:
/export/nim/system
Flat file inventories by serial number of each system installed
by the NIM server:
/export/nim/system/{system type}/{serial number}
Master directories of additional special filesets:
/export/nim/packages
Any mksysb file images of servers:
/export/nim/mksysb
Master installable SPOT images:
/export/nim/spot
NIM custom scripts:
/export/nim/custom
Manual backups of the NIM configurations:
/export/nim/nim-backup
We used SPOT copies (customized to contain only the minimal
OS software) to complete the initial loads of our NIM client
systems to support our six different hardware platforms. To
build a SPOT, we first loaded all AIX 4.3.3.0 filesets into
a directory called raw-lpp_source. This directory essentially
contained all volumes of the BASE install CDs, bonus pack CDs,
and base and extended documentation CDs for 4.3.3. We also had
an equivalent raw-lpp_source for each maintenance level
we needed to support. We made sure that every fix for that maintenance
level was present. Once completed, we generated a listing of
lpp's into a bundle format (an installp_bundle
resource that contained our customized minimal OS software)
to create the AIX433-base lpp_source. We then created
an MLXX lpp_source containing only the maintenance level
we approved for our standard. This was customized similar to
the AIX433-base lpp_source except it contained only the
ML fixes. After our customized SPOTs were created, we were ready
to install a system from this base lpp_source with any
ML level.
During the installation process, we used the bosinst.data
and image.data files to control the layout of the rootvg
onto the hard disks of the server. Using NIM has been facilitated
by establishing strict hardware configuration standards for
each each IBM server type at our site. This provides the ability
to move machines around without conducting a hardware upgrade.
It also makes installation easy, because we always know what
kind of hardware we are using.
Customizing
After the installation of the OS was complete, we had the
NIM server automatically run a customization script to enforce
more unique standards within our computing environment. This
customization script (Listing 1) was written, in-house, to facilitate
turning on and off selected functions. It uses these functions
to control the flow of each operational section. Each command
used was echoed to the NIM console log along with a description
to aid in diagnostics in case something went wrong during the
NIM installation. By defining this script as a NIM resource,
NIM is able to perform the administrative tasks discussed below.
Note that this script is designed to meet the objective of a
system being 95% pre-configured for our operating environment
after NIM has completed its operations. The other 5% is what
our operations staff performs on the server to turn it into
a Web server, ftp server, gateway, or even a database server.
Also note that this script is good for teaching new AIX systems
administrators about essential system commands in AIX. Each
command executed is echoed to the screen with a description
of what it's doing, it's command syntax, and it's
output.
Set Up OS Environment Parameters
During this phase of the script, we adjust AIX to meet our
standards for items such as:
- Timezone
- Remote boot facility
- Network Options
- Starting certain demons on restart
Setup Hardware Parameters
AIX will set the 10/100 NIC cards to "autonegotiate".
We found this caused a network bottleneck in our infrastructure.
We utilize these commands to force the adapter to 100/full duplex
on startup of the server.
Configure System Files
We have created a master directory of certain system files
that are copied onto each system when a NIM installation takes
place. Utilizing this repository allows control of what the
initial system files will conform to. This is also the first
time in the script that you see special programming for a particular
platform. Note at the end of this function that we do not want
the "httplite docsearch" utility running on
anything other than a 43P workstation.
Setup Security
Here we utilize the NIM server to create another script that
will only run once on bootup and then remove itself, the /etc/firstboot.
By placing commands in the /etc/firstboot script, the
NIM-installed client will execute these commands and then remove
the script after it gets an OS installed. This allows us to
lock a system down to our security standards immediately. We
do this after NIM is finished with the server, because you will
lose connectivity to the NIM server and your install will hang
if you run these commands before NIM has completed.
Install Third-Party Applications
Here we have created our own "BFF"-formatted install
packages for 32-bit and 64-bit compiled software. We keep these
in separate directories and use this script to determine whether
we are installing a 32-bit or 64-bit machine, and then to install
the correct versions.
Create and Set Dump Devices
Depending on the configuration of an AIX server, the initial
dump space defined may not be big enough. The two main parts
of this function are to calculate whether the dump space is
big enough and to determine whether a mirror drive exists. If
the function determines the dump space is too small, it will
increase it automatically and set up a secondary dump file on
the mirror drive if available.
Adjust Paging Space to Complement RAM
Because AIX has changed its swap algorithms, it is no longer
necessary to have swap space equal to size of RAM. (This really
helps on a 64-Gig RAM system.) We use this function to calculate
the size of RAM and then adjust the swap space to the values
noted in the script.
Create a Hardware Inventory -- By Machine Type
or by Serial Number
This function calls another script that will create an exhaustive
inventory of a NIM installed server (Listing 2). We use this
to verify a server install or for disaster recovery of a down
server. We could literally rebuild a like server from this listing.
Create a Complete Inventory -- By Machine Type
or by Serial Number
The purpose of this function is to call a smaller system inventory
script (Listing 3) that will only give the amount of information
about each NIM-installed server for our inventory-tracking database.
Conclusion
In summary, we needed to ensure consistent standard OS installations
between four separate data centers. By using AIX 43P systems
and NIM system software, we succeeded in our efforts. NIM allowed
for remote installation and management of our networked systems.
Specifically, the NIM script resource handled tedious and time-consuming
tasks that cause systems administrators to dread post-install
configuration. From setting up hardware parameters to creating
inventory lists, NIM allowed complete customization while leaving
us free to handle other vexing tasks...like resetting passwords.
Bill Wood is a co-author of three UNIX books and now works
as a systems team lead in a UNIX shop. His primary function
is the mentoring and coaching of the other UNIX administrators
and engineers. He enjoys working with large IBM AIX SP Frames
within supercomputing data centers with large server farms.
The largest IBM AIX SP Frame he has worked on to date was 612
nodes with 912 Terabytes of disk storage, and he's looking
forward to working with his first Petabyte. This also explains
why he is so adamant about standards -- without them
in large data centers, you will quickly lose control, which
is why he has chosen NIM to ensure these standards.
|
