 Replacing
rdist and ftp with scp and Associated Utilities
Michael Watson
At some point, manually logging into every server to repetitively
make changes gets rather tiresome. Therefore, my team decided to
automate these procedures. With thirty-plus Solaris systems, we
began by utilizing a set of simple ftp scripts (see the Listings
at www.sysadminmag.com) to get, display (in the case of text
files), or put files to some or all of our servers.
Although our servers are not connected directly to the Internet,
root logins for telnet or ftp are nonetheless forbidden.
This somewhat limits the utility of ftp for system configuration
files. We also have shared directories from our NIS master to all
of the other servers, but in either case, we still had to log onto
each server to deal with root-level files, whether they were available
by NFS mounts or by ftping them to a local directory.
Moving On Up with ssh2 and scp
Because we were installing ssh 2.0.13 anyway, we enabled
root logins in the /etc/ssh2/sshd2_config file. While we
would like to disable root ssh logins (in order to get scp
access for root) we are currently living somewhat uneasily with
this configuration. (scp is the secure equivalent of the
classic UNIX rcp remote file access utility.)
Essentially, the scp scripts were a refinement of the original
ftp scripts, and they serve three functions: 1) collection
of an identically-named file from all of the servers, which are
then retained with a suffix comprising the server's hostname;
2) redistribution of saved or individually tailored files back to
all or a subset of the servers; or 3) simply to distribute an identical
file to the servers, à la rdist.
We do not have any trusted hosts, but we do have a simplified
system by which certain servers that share certain applications,
or teams that utilize them, have the same root passwords. If each
were unique, not only would it be a major problem remembering them,
but there would be less utility in scripting file transfers. This
is especially true in our case because no passwords are allowed
to be committed to a file -- they must be manually entered each
time a script is run. After all, typing in a password for each server
is what we are trying to get away from.
The Centralized Economy
Once we had the basic infrastructure in place, a few logical extensions
came to mind. The first is to use the scp scripts to facilitate
maintenance of scripts and configuration files across the server
set. This is relatively easy to do, and we settled on a system whereby
our collection script (see Listing 6) would transfer files of the
same name to our "master" server, an Ultra-5 in this case,
place them in whatever directory was specified, and append the name
of the server from which the file originated to the end of the filename.
This allowed us to quickly view and individually modify the files,
and we could then return individualized files to each respective
server using scp1m-put (see Listing 7). If the substitution
of all files by a single file is desired, scp1m-rdist (see
Listing 8) provides that function.
We then had distribution abilities, but any updates that required
action on the server still required that we log in to each server
to perform the tasks.
Scripting with Intent to Distribute
Out of this was born the cron-based update scripts. We set up
two cron entries, one to run once a day for those jobs that shouldn't
be messing with prime time activities, and the other to run every
five minutes. The original design had the cron job execute if it
was present, but our SNMP guys complained about the errors cron
would register when no script was there to be executed. This led
to the development of a permanent script for each cron job, which
exits cleanly if there is no script to be executed. An additional
benefit was that if scp was not carrying forward the original
permissions on the file (which might occasionally happen when scping
a single file by hand and forgetting the -p option), the
permanent script now chmod's the script before running
it.
We now had to create a script to be executed on all or some subset
of our servers, scp it into the directory on the target system(s)
(set up for this purpose under the name sysupdt.qh or .qd
depending on how soon we want it executed), and then wait for it
to run. This is relatively unsophisticated because failing to wait
for the script to run before scping over another will wipe
out the one waiting. You can even do this multiple times!
After creating a few scripts to be run by our cron job on the
remote systems, we realized that using the sysupdt.qh file
every time on our central server was preventing us from saving and
easily recognizing the scripts we had created. Thus, we stared creating
descriptively named scripts we could use to reduce reinventing-the-wheel
overhead. When necessary to tailor scripts individually for each
destination server, we would append the server name to the script
(e.g., "dothis" script for "onthat" server would
be named "dothis.onthat"), which is how scp1m-put
(Listing 7) expects to see source filenames constructed. An example
of a single copy from the command line of the central server to
the file to be executed by our cron job on a destination server
would therefore look like:
>scp dothis.onthat root@destination-server:/.sysupdt/sysupdt.qh
On each server, the sysupdt.Q* scripts rename the scripts
they execute, and these are retained for 30 days. Cleanup is a part
of the daily sysupdt.QD execution. The /.sysupdt directory
is our default location on both our source and target servers.
Can Institutionalization Keep You SA[FN]E?
One example of how we put this to use is hardly a profound one,
but serves to simply illustrate a potential use of this infrastructure
(terminology that we are torn whether to describe as pedantic or
grandiose).
This was the development of a local log checking script. This
was exported to each of the servers by rdistall:
>./rdistall logchk /usr/local/bin/logchk
Next, the updates to root's crontab were distributed out,
to be executed by the cron jobs that run every five minutes:
>./rdistall cron.updt /.sysupdt/sysupdt.qh
However, the logchk cron entry executes at 23:59, so the
next morning we ran:
>./dailylogchk MMDDYY
where MMDDYY is yesterday's (or any previous day's date,
as we do on Monday for the previous three nights). This script calls
getall, passes the name and directory parameters that getall
needs, then permits examination of the files via "less",
and subsequently runs some additional checks via the scripts loginchk,
pscheck, cronchk, and dfkchk. cronchk
is a recent addition, and requires that Monday's checks be
run in chronological order.
Briefly, these extra scripts are:
loginchk -- Looks for native root logins, but su
will show them too, so they have to be tracked down and verified.
pscheck -- On Solaris, ptree is a variant of
ps. This is a cross-check, which would hopefully trip up
a ps replacement.
cronchk -- Differs for changes between the most-recent
and the next most recent crontabs for changes. We want to catch
unexpected ones.
dfkchk -- Parses df -k output for disk space
utilization greater than 90%.
shortlogchk -- Sometimes we forget and "q"
out of our less session. This then runs the above four scripts.
These illustrate the kinds of things we can do when we pull a
set of files from all, or a subset, of our servers. Examination
of such files on a single server or workstation is more efficient
than having them separated and trying to discover differences, similarities,
and discrepancies on a server-by-server basis.
Problems in Paradise
As in all endeavors, there are a few bugs that bite. Under Solaris
8 and ssh 2.0.13, the password that is passed into the scripts
without being displayed is nonetheless showing up when the script
executes the expect command. Oddly enough, we didn't
have this problem with a Solaris 2.6/OpenSSH 2.x combination. We
successfully tested a /dev/null redirection that we then
commented out in the scp1m-get script, but that suppressed
all output, including the output that might show any failures or
other anomalies. We hope, however, that a future revision of the
components we used under Solaris 8 will correct this problem. Let
us know if a solution is discovered.
Michael Watson liked computers better than his prospects with
his BS in Psychobiology, so he started out programming COBOL, assembler,
and Ada on a couple of coasts. He then suffered a brief digression
into Oracle programming before ending up in UNIX systems administration.
He now works as a consultant based in the middle of the country.
When working in the same place for long enough, he likes to write
labor-saving devices and contemplate whether he would really rather
be a commercial pilot.
| 
