Remote System Logs via SSH

David Beecher

You may work for a company with sites distributed over multiple geographic locations or for a company with multiple computer centers on a single campus. In either case, you probably have too many systems to review the individual logs on each one. Also, if you have custom parsing and monitoring scripts written, every time you make a change you must roll it out to perhaps dozens of servers -- cumbersome at best. If you have not set up central logging because you are afraid of exposing sensitive log information over the public Internet or your WAN, perhaps it is time to encrypt your logs in transit.

In this article, I will first show you how to configure remote logging. Then I will show how to transport your syslog and sysklog information over an encrypted link to a remote logging host where all your scripts and monitoring tools may reside.

Tools Needed for This Job

The SSH (SecureShell) tool (version 2) can make these things happen for you. SSH allows you to connect to a remote server with an encrypted tunnel over a single port and pass TCP traffic. Most often, SSH is used as a replacement for telnet. It can be used to pass other TCP traffic. I chose SSH2 because it solves many of the security issues currently related to SSH1 and as with most security tools, the latest versions are the best. SSH may be obtained from:

http://www.ssh.org

Netcat is one of those all-purpose tools that helps you do all kinds of things. Because the syslog uses UDP when logging to a central logging server and SSH is a TCP-oriented tool, we'll use Netcat to be the "glue" in our solution. An excellent reference can be found at:

http://www.l0pht.com/~weld/netcat/index.html

Use the man pages! Familiarize yourself with your syslogd and syslog.conf information.

Assumptions

I have made some assumptions here that match my own setup. Those are:

You need root access to two or more UNIX workstations or servers.
For the purpose of this article, both of the servers or workstations you use must be running UNIX.
You must have SSH2 client and server installed on both your office workstation and your home workstation. The SSHD server daemon must be running and listening for a connection (mine is on the default port 22) on the system you will be using as your central logging host.
You have Netcat downloaded, configured, built, and installed.
You know how to use SSH to do port forwarding (forwards data sent to a specific port over an encrypted tunnel); see the January 2001 issue of Sys Admin, p. 45.

Basic Remote Logging

If you already know how to set up remote logging, you may want to skip to the next section. This section will help you determine whether you can get remote logging working between your two test workstations. The next section will undo some of the changes you make in this section. Before you make any changes here, make backup copies of any files I ask you to change.

Basic remote logging can be done on two servers located in separate buildings, or on two workstations that are sitting side by side (my preference for testing). We'll call those workstations/servers "myworkstation" and "myloghost" for the purpose of this article. You will need to use the valid names in your DNS or /etc/host file. Please make backup copies of any files you modify as we will be undoing the next few changes when we move into encrypting your log data before it gets transmitted to your loghost.

First, let's establish the basis that you can do remote logging at all. You will need to check the man page on your syslogd. Some versions of syslogd listen to network connections by default. Others do not. I am using a Linux workstation for my central loghost. Normally my syslogd gets run at startup as:

syslogd -m 0     (that's a zero at the end)

It does not, by default, listen for network connections. To do that, I killed my syslogd and started it with (as root):

syslogd -r -m 0

If you keep this configuration, be sure to remember to update your startup files.

Now we need to go to your other workstation and tell it to send logging messages to your newly created central loghost. You will need to edit /etc/syslog.conf as root (make a backup copy first). Add the following line to the end of the file:

*.*     @myloghost

Save the file and exit. The reason we added this line and did not change anything else is to demonstrate that you can send logging information to multiple destinations (e.g., the local logfile, /dev/console, email, and a separate loghost if you want, all at the same time). In this case, your system will continue to log information normally, but it will also send a copy of every log event to "myloghost".

To make this work, restart your syslogd daemon on both workstations. You can do this by sending it the HANGUP signal:

kill -HUP <process-id-of-your-syslogd-program>

Begin tailing the default message log (in my case /var/adm/messages) on your "myloghost" and try doing things on your "myworkstation" that cause log messages to appear (e.g., sending email, logging in or out, ftp, etc.). You should see the messages appear in your "myloghost" logfiles:

tail -f /var/adm/messages

Don't go any further until you get the above working. Read your man pages. They are very good and having this working will give you the confidence to take the next step of encrypting the transmission of these log messages.

Protecting Your Logging Data in Transit

This section will undo some of the changes you made in the previous section.

1. We need to do things a little differently to make this work. For that reason, we need to undo the changes that we made above. Restore your original /etc/syslog.conf on your "myworkstation" and HUP your syslogd on that machine to restore things to normal.

2. On your "myloghost", you also need to tell syslogd NOT to listen to network connections. If you had to make a change above and add the -r option to syslogd, kill it and rerun it without that option. If your syslogd listens to the network by default, check the man page and use the option to turn off the behavior that is described there.

3. Establish a SSH tunnel between your two workstations on port 32000. Let's start with "myloghost":

ssh -R 32000:localhost:32000 -l jdoe myworkstation

The above command means: using SSH, map a port (32000) on the remote server (myworkstation) to my local machine (myloghost) port (32000), and login to the remote server (myworkstation...) as user "jdoe". If your username on your workstation at work is the same as your workstation at home, you can eliminate the "-l jdoe" portion of the command.

I picked port 32000 because it was not listed as a Well-Known-Service port (cat the file /etc/services). It is a "High Port" (above 1024), so you don't have to be root to attach to it and, most importantly, neither of my workstations were bound to it. You could use other port numbers (e.g., 9999, etc.).

4. As SSH connects to your "myworkstation", you will be asked for a password. This is the password on your "myworkstation" for the user "jdoe"; enter it.

5. If you have a problem and cannot get connected, simplify the command and just see if you can SSH to your "myworkstation" machine from your "myloghost" machine:

ssh -l jdoe myworkstation

6. If that doesn't work, see if you can traceroute or ping to your "myworkstation". If you can't, your "myworkstation" may have been disconnected. Test whether you can telnet to port 22 on your "myworkstation" (telnet 'myworkstation'). It should come up with an SSH notice after it connects. If that doesn't work, see if you can connect to any port you know is active on "myworkstation". If none of those work, something is amiss and you need to scrutinize your network or the setup of "myworkstation".

7. If you got this far, I'll assume you got connected and logged in. You are now talking to your "myworkstation" over an encrypted channel. Congratulations.

8. Review the messages that were sent while you were getting connected. If you don't see any messages saying "Remote port forwarding failed" then you're probably OK. If you do, it probably indicates that the port you tried to remote forward (in my example, 32000) is being used by somebody else. Exit from your workstation and try again with a port number of something like 32500. You can use any high port that is not in use by an existing daemon or other service.

Wrapping UDP

9. Now use Netcat to pipe UDP (used by syslogd) into a TCP stream (used by SSH). On the client ("myworkstation") side, make sure Netcat is in your PATH and run it as follows:

nc -l -u -p syslog | nc localhost 32000

10. On the "myloghost", run Netcat as follows:

nc -l -p 32000 | nc localhost -u syslog

Data logged to syslog on the "myworkstation" will be transmitted over an encrypted tunnel to "myloghost" and sent to the syslog facility. It is best to keep your "myloghost" behind a firewall to prevent DoS-type attacks.

Conclusion

This is a way of getting your logs passed in a secure tunnel if you really need to do so. It does not have any of the finess that generally indicates a good solution. If you have more than a handful of hosts that you need to deal with, you'll probably want to use a command-line tool like crypt. I chose ssh for this article because it is available to everyone and is quite secure. The crypt utility is not necessarily available outside North America. If you know of a better way of doing this, please let me know.

Again, I recommend reading the syslog man pages thoroughly; they are an invaluable tool. Also, the SSH Web site is an excellent source for information on SSH (http://www.ssh.org). The software provided there by SSH Communications Security, Inc. is free for evaluation purposes (please review their license agreement before using). You should license it if you are using it in a commercial setting. Also see: http://www.freessh.org.

Netcat is a very powerful tool that I have not thoroughly explored here. Read the Netcat man page and check out the information at:

http://www.l0pht.com/~weld/netcat/index.html

Use Netcat wisely. As with most powerful UNIX tools, it has a dark side and must be used with caution. However, you may find that it can solve many problems.

David Beecher is a Sr. UNIX System Administrator with more than 12 years of experience. He has a background in analog and digital electronics, robotic control systems, and many years of programming in assembly language on a variety of platforms. Dave currently works for a technical services company and is on assignment at a well-known Internet destination. He lives in Castle Rock, Colorado. He can be reached via email at: DAVEUSW1@QWEST.net.