PICA: Perl Installation and Configuration Agent

Miguel Armas del Río and Esteban Manchado Velázquez

At the ULPGC's (Universidad de Las Palmas de Gran Canaria, University of Las Palmas of Gran Canaria) Network Division, we administer several servers that run critical network services such as DNS, DHCP, network monitoring, etc. Because these services are critical, we run a number of scripts on every server to check sanity and try to fix basic error situations.

We needed a way to distribute all these scripts and the important services' configuration files from a centralized location with little differences to adapt them to each host. We also needed a way to register any change on the configuration files, to be able to detect when a particular error was introduced, and who did it, and we wanted to centralize all network incident notifications and alarm management.

To meet all these needs, we developed PICA (http://pica.ulpgc.es). With PICA, we have a central repository of configuration files and alarm scripts. This repository is managed using CVS, so we can recover old versions, see change logs, and let various admins work concurrently. Actually, every sys admin has a local copy of the working tree, and CVS does all the dirty work.

In this scenario, PICA is used to distribute the configuration files and alarm scripts to the various servers. PICA uses SSH to establish secure connections to the remote servers, which is very convenient, since we were already using SSH with RSA authentication to access all remote servers. The alarm scripts send incident notifications and service status reports to our central NetSaint (http://www.netsaint.org/) server using asynchronous checks (see NetSaint documentation). If a critical error is detected, an alarm is also sent via email and as a SMS message to the sys admin mobile phone.

Approach

From the beginning, flexibility and security were important to us. For that reason, we based PICA on SSH and, as the name already suggests, Perl. We also liked the idea of easing the common case and letting the user solve his own problems, if possible.

Now, you may be thinking "but there's PIKT" (http://pikt.uchicago.edu/pikt/). Sure. We used it for almost a year, but we never really loved it. It had too many features we didn't need. We didn't like the RPC approach it used (fewer open ports means fewer security problems). We already had SSH running on all servers, and we wanted it to be the only way to access some of them. We also had SSH's RSA authentication to log in securely to all servers without typing passwords. So why not use it in the configuration tool for secure encrypted connections to the remote servers?

We also didn't need PIKT's own scripting language (pikt). We didn't particularly like learning yet another script language. Sure, it had a lot of useful features like variable persistence, but we could also do that with various Perl modules. Perl is a much more complete programming language and allows us to do virtually anything we need. So, although we know PIKT is a great tool, and we like it's philosophy, it wasn't the solution for us.

PICA understands basically two concepts: objects and hosts. The purpose of PICA is to manage objects (files) in different hosts (computers). To achieve this in a simple manner, PICA lets us define object groups and host groups, so a collection of objects can be installed, deleted, listed, executed, or compared in a collection of hosts (just in a single process). Since PICA knows the hosts we are administering, it can also be used to execute standard UNIX commands in a given group of hosts.

Moreover, PICA includes the PICA Framework for Integrated Alarms (PIFIA), which is a collection of files and conventions to manage alarms. In this context, alarms are little programs installed in your machines that are executed periodically to check the sanity of your servers.

PICA Components

PICA is built from several components. There is a central executable, pica, a Perl preprocessor, and an alarm manager and Perl library. The central executable reads the supplied command-line arguments and does the job. The Perl preprocessor is used for every configuration and non-verbatim distribution file before processing (obviously). The alarm manager is used to set up and install alarms, and comes with a handy Perl library. The latter is the most primitive component, and will evolve a lot.

The preprocessor component is used on every PICA configuration file. That means the files are terribly dynamic, as the preprocessor is completely Perl-based. The recognized directives are:

• #perl/#lrep -- For executing completely arbitrary Perl code. Anything returned by this code will be substituted for the original code.
• #if/#elsif/#else/#fi -- For conditionals, from parenthesized arbitrary Perl conditions.
• <# ... #> -- Inline #perl/#lrep environment (for one-liners).

The PICA central executable begins by preprocessing and reading hosts.conf. Here, the program finds out about the machines, with their attributes, variables, and such. Then, and once for each host, PICA preprocess and reads objects.conf. Obviously, as the file is preprocessed once for each machine, the result can be different, because the preprocessor namespace is, in general, distinct. That way, we can define different distribution files, or in different ways, for each machine. We can:

1. Make the host definition depend on command-line variables.

2. Make the objects definition depend on command-line and host variables.

3. Make the contents of the objects depend on almost everything.

Understanding this is very important to understand PICA behavior. Once you get the hang of it, you will know how to make PICA do what you want.

The last PICA component, the alarm manager, is basically just design ideas, and is discussed further in the sidebar ("PICA Framework for Integrated Alarms (PIFIA)".

PICA Usage

To start using PICA, we create the config files where we specify which "objects" (files and alarms) we want to distribute, and which hosts will be administered using PICA.

The Hosts File

The file hosts.conf is used to configure the hosts that should be administered using PICA. We can organize the hosts in different, possibly overlapped, groups that can be later used in conditions, so we could install a given file only in hosts belonging to a given group. We can also define variables for a given host or group and use them later. This way we can specify different values for a variable depending on the group we are working on.

For each group or host, we can define attributes and local variables. Attributes are object properties or general preferences for PICA. Thus, they are limited to several that are already defined (like paths for different programs, uid and gid to set in distributed files, etc.). On the other hand, local variables are user-defined identifiers with scalar values associated. They are substituted in the Perl preprocessor expressions. Note that every host should be "declared" for PICA to know about it. This is to prevent big mistakes.

A simple example of the hosts.conf file is shown in Listing 1. For each host we can specify the attribute "fqdn", which is the name PICA will use to create the secure connection. If we don't specify it, PICA will use the name used to define the host object. This attribute is useful if you have hosts in different domains and don't want to use the FQDN for the host definition.

The Objects File

The objects.conf file defines all the objects that will be distributed using PICA, and depends on the current host. In general, you can include or generate dynamically parts of it, depending on evaluations of previously defined variables. These, at this point, will be:

1. Variables defined in the hosts.conf "defaults" environment.

2. Variables defined in the objects.conf "defaults" environment.

3. Variables defined in any of the groups to which the current host belongs.

4. Variables defined in the current host.

The latter, of course, have preference in case of multiple definition. We also can use the internal variable $picahost to get the name of the current host. Because of that, the possibilities explode here: we can write entire Perl scripts (or in other languages, as long as we call them from Perl) to dynamically generate configuration files, depending on defined variables, the day of the week, etc. And, here resides the power of PICA.

There are two types of objects in PICA: files and alarms. Depending on the object type, they will have some mandatory and some optional attributes:

Mandatory Attributes

• Files

• Alarms

So urce -- Where to read the alarm code from. If this is a relative path, $picasrc is prepended.

Additional Attributes

• uid -- uid to set in the installed file. Default: 0
• gid -- gid to set in the installed file. Default: 0
• verbatim -- If set to 1, the object will be installed without preprocessing it. It is useful to install binary files, and files that could include preprocessor directives that we don't want to parse. Default: 0
• perms -- File permissions to set in the installed file. Default: 644
• path -- Where to install the file in the remote host. If it isn't specified, use the same as "source" attribute. This attribute is only used in file objects.

A simple objects.conf file is shown in Listing 2. There is a defaults section that can be seen from any object, and we can define variables that can only be seen by the object they are defined in.

Any time PICA operates with an object, it will build a namespace containing all the variables seen by that object in the host it is working on. Of course, the most specific definition has precedence. That is, if we define a variable in an object, and that object belongs to a group where the same variable is also defined, the value used is the one given in the object definition. Besides these variables, we always have $picahost and $picaobject, which hold the current host and object we are processing.

Command-Line Syntax

Once we have built our hosts and objects definitions, we can start using PICA. Basically, PICA always does "something" on a list of objects on each of the given hosts. As the the help option (-h) states, PICA has the following command-line syntax:

PICA -- Perl Installation and Configuration Agent, Version: v0.0.1

Usage: pica -[ixtflh] [-d] [-v] +|-F objects +|-D defines +|-H hosts

-i -- Install objects
-x -- Execute object/command
-t -- Delete object
-f -- Diff object
-l -- List objects
-h -- Shows this help
-d -- Debug. Don't install/delete things, just testing

+|-F -- Build object list
+|-D -- Build defines list
+|-H -- Build hosts list

\The command line has three different mandatory parts:

1. The command -- This is the first group of options, and it determines what we want to do.

2. The object arithmetic -- Determines what objects we want to operate on. The object list is built using +F/-F to add/delete objects.

3. The hosts arithmetic -- Determines what hosts we want to operate on. The hosts list is built using +H/-H to add/delete hosts.

Object and Host Arithmetic

The object and host list on which we want to operate is built with +F/-F and +H/-H. With +H/-H we add/delete hosts or groups to the hosts list in the same order they are entered. For example, the expression:

+H dnsservers solaris -H deimos

The object list is built the same way using +F/-F, but the object arithmetic is evaluated for every host. This is because the objects available for every host can be different since we can use conditionals in the objects file. For example, in the objects file in Listing 2, all documentation objects will only be seen by members of the group "doc". In both lists, the implicit group "all" can be used to reference all defined objects and hosts.

We always need at least one host and object, so options +F and +H are mandatory. If after doing the host/object arithmetic PICA gets an empty list (either objects or groups), it gives an error message and aborts execution.

We can also use +D/-D to build an optional list of definitions that can be used to preprocess files and as variables in object definitions, as if they were defined in the "defaults" environment of the hosts.conf file. Actually, there is a small but very important difference; because they are defined before reading the hosts.conf file, these are the only variables that can be used to conditionally preprocess this file.

PICA Commands

PICA currently supports five different internal commands, but more could be added in the near future since we are still adding new features. With each of these commands, we can use the -v option for verbosity and -d for debug. Option -d gives a lot of output. It doesn't really do anything, but just prints what it would do.

All these commands are executed using SSH connections to the remote hosts. Right now PICA doesn't have any access control system, and will probably never have one. We like SSH, and it gives us everything we need to access remote servers securely.

Note that if you try PICA, be sure to properly configure the RSA authentication, or be ready to type a lot of passwords. One nice trick is to distribute the SSH's RSA authentication files using PICA as explained in one of the real-life examples described later.

Install (-i)

The install (-i) command will install the given objects in each of the given hosts. The install command first generates the objects it will install in a local dir ($picaroot/tmp/$picahost), and then install them in the remote host.

PICA supports three different method for remote file installation:

• SSH -- This is the default method. It installs each file using an SSH connection. This is the default method because it only needs SSH, but it's painfully slow because it makes a different connection for each file.
• tar -- This method uses tar over a SSH connection to transfer all files using the same SSH connection. Right now, this is the recommended method.
• rsync -- This method is still under development, because we have found some problems with the way rsync handles directory permissions and symlinks. It will be the recommended method once we fix this problem, because it's the fastest. Don't use it right now.

The following command:

pica -iv +F NTP -F step-tickers +H all -H deimos

Execute (-x)

The execute command will execute the given list of commands in each of the given hosts. If PICA finds an object with the given name, it will read its path attribute and use it for remote execution. This way we don't have to remember the location of the script. If it doesn't find an object with that name, PICA assumes it's a UNIX command and tries to run it.

For example, the command:

pica -xv +F DNSChkUrgent +H servers

pica -xv +F "ndc reload" +H dnsservers

List (-l)

This command lists the given objects. It basically executes ls -l for every object's path. It can be used to see whether an object is installed and whether the uid/gid and file permissions for that object are correct.

Delete (-t)

This command deletes the given objects in each of the given hosts.

Diff (-f)

The diff command finds differences between the object that should be installed in a host, and the one really installed. It basically generates the object for that host, and makes a diff -u between this object and the one installed in the remote host. It's very useful to see whether a host has the latest version of an object.

Future Commands

As mentioned previously, we may add new commands to PICA in the near future. One of the functionalities that still needs work is the alarm management, and we will add new commands to enable, disable, and list installed alarms in a remote host. For more information about the current alarm functionality, please see the sidebar.

Real-Life Examples

Two Files from the Same Source

Let's begin with an example for distributing two files from the same source. At the Network Division, we run the university's primary nameservers. But we also give support to people that run secondary servers. In one of our primary servers, we also wanted to publish the configuration file needed to run a secondary server. We wanted to use the same source to keep both files consistent. This apparently simple task has created a lot of problems with other tools we have used. With PICA, the solution is simple. In the case of the masters, we have to install one version for production (the master server file) and one for documentation (the slave DNS server file). Thus, we defined, in hosts.conf, the following:

hostgroup dnsservers {
    members { fobos, deimos, mercurio, ulpnet, ulpnet2 }
}
hostgroup dnsmaster {
    members { ulpnet, ulpnet2 }
}
hostgroup doc {
    members { ulpnet, ulpnet2 }
}

#if (ingroup('dnsservers'))
group DNS {
    file named.conf {
         path = '/etc/named.conf';
         source = "DNS/named_conf.cfg";
    }

    ## Documentation for this Service
#  if (ingroup('doc'))
    # ...
    file named.conf.sample {
       path = '<#$docdir#>/Servicios/DNS/named.conf.sample';
       source = 'DNS/named_conf.cfg';
    }
#  fi
}
#fi

# ...
zone "ulpgc.es" {
#if ((ingroup('dnsmaster')) && ($picaobject !~ /\.sample$/))
   type master;
   file "mydb.db";
   also-notify {
      # ...
   };
#else
   type slave;
   file "mydb.db.bak";
   masters {
      # ...
   };
#fi
};
# ...

Installing RSA Authentication Files within PICA

When you start administering a new server with PICA, you should first configure SSH's RSA authentication, to be able to access that server without typing any password. This task can be simplified by distributing the needed files using PICA. We use SSHv2, so we will assume this version of SSH. First of all, every sys admin needs a private/public key pair. Let's say our public keys are in SSHv2 format in the files sysadm1.pub and sysadm2.pub. We will add the following entries to the objects.conf file:

# SSH RSA authentication files
group RSAAuth {
    # SSHv2 authorization file
    file ssh_auth {
         path = '/root/.ssh2/authorization';
         source = "SSH/authorization.cfg";
    }
    file sysadm1.pub {
         path = '/root/.ssh2/sysadm1.pub';
         source = "SSH/sysadm1.pub";
    }
    file sysadm2.pub {
         path = '/root/.ssh2/sysadm2.pub';
         source = "SSH/sysadm2.pub";
    }
}

We could even generate the authorization file on the fly with the needed Key entries with the following code snippet:

#perl
my @return;
# Get key files reading group members and skipping 'ssh_auth'
my @keys=grep(/\.pub$/,members('SSHAuth'));
foreach my $key (@keys) {
  push @return,"Key $key\n";
}
# Return the array (will be printed)
@return;
#lrep

pica -iv +F SSHAuth +H new_server

Execution of Perl Code within Configuration Files

Here at Network Division, we have a very handy script to restart critical services via the Web, just in case SSH is down on a given server. Of course, it has a secret password to allow access only to authorized administrators. Before PICA, we had to manually create the password's MD5 hash using the "crypt" Perl function, and include the hash in the source file. Now, we can use something as elegant as this:

$passwd='<# crypt('secret-key-that-you-wont-ever-figure-out','salt') #>';

Conclusion

PICA is a great tool to ease the administration of large installations. It allows the sys admin to keep all the important configuration files and alarm scripts centralized. These files are created for each host using the same source file, and we can include arbitrary conditions depending on the target host, file name, and user-defined variables.

We are successfully using PICA in an environment where we have all the important configuration files centralized and distributed using PICA. We also distribute a number of alarms that check important conditions and notify them to our NetSaint (http://www.netsaint.org) monitoring server.

Note that if you are considering using PICA, it's still a work in progress. We have been working on it for only a month, and although it's very usable now, it still needs work. Specifically, the alarm framework is very basic and will probably be completely rewritten. In fact, when you read this article, a lot of things will probably have changed.

Miguel Armas is a 29-year-old Network and System Administrator at the University of Las Palmas' Network Division. He has also worked as a network and system consultant for five years. When not working, he enjoys sailing, reading, and messing with Linux. He can be reached at kuko@ulpgc.es.

Esteban Manchado is a 23-year-old Network and System Administrator at the University of Las Palmas' Network Division. In his spare time, he downloads MP3s, studies linguistics, listens to music, and picon-scratches Elvis Crespo songs. He can be reached at zoso@ulpgc.es.