Jeroen Hoppenbrouwers   project integrator

Many people who maintain server computers will know the challenge: how to keep track of all the small changes you make to the configuration of the machine, so that you know a) what you did, b) why you did it, and c) how to do it again? Especially when there is a group of people who share the responsibility of server maintenance, this becomes crucial.

Several options exist to document your changes. In this brief blog I will explain how I learned to do it, using the Subversion (SVN) revision control system and a small set of Makefiles called Caspar which originates at Tilburg University.

Maintaining Host Config Files

When working with Unix-like systems such as Linux, people modify text files to change the configuration of the system. Nearly all modifications to Unix-like machines involve opening a human-readable text file with a plain text editor, looking up the right line, adding a comment or two if you're doing something slightly odd, and changing some words on a text line. I'm not going into a quarrel about editing text files versus clicking on graphical items -- both variants have their pros and cons. In this blog, I assume text files, and will capitalise on a particular way to maintain them properly.

The best way to keep a repository of text files and their history is via a revision control system. I happen to use Subversion, but there are several options to choose from. In most cases, these systems allow you to work on a local working copy and when you're done, to check the changes back into a repository. All systems allow more users to concurrently work on multiple working copies, and take care that conflicts are discovered and solved.

It is not a good idea to convert the whole configuration tree of a system (the /etc branch) into a local working copy. This will bring all files and directories in that branch under revision control, but the majority of files never needs to be touched and is perfectly well maintained by the package system of Debian, Ubuntu, and other distributions. Adding these to a revision system would increase your workload and confusion!

We want to have just the config files that changed from the original distribution copy in the repository. I do this by maintaining a local copy of the relevant files in the /etc branch in my personal home directory, in ~hoppie/var/host.name/etc . Every time I want to change a config file, I go here, and not to the actual /etc directory. This takes a bit of time to get used to, but convenience is increased dramatically by this way of working, so you'll learn it fast.

If I find that the file I need to edit isn't yet in my working copy, I copy it straight from /etc and immediately add it to SVN with a log comment "Distro", i.e., unchanged from flat package install. All changes I subsequently make are checked into SVN and therefore logged and reversible. This system combined with the inherent versatility of comments in text files gives very good control over my servers and as a bonus a historical backup of all relevant config files.

Automating the Config Updates

Now of course you need to copy a changed local copy back into /etc if you want to make the changes available to the machine. You can do this manually, but that takes away much of the convenience. When I worked at Tilburg University, the people at the library and information systems department (a merger of the old library and the old computer centre) taught me how they did it. Joost van Baal wrote a small set of Makefiles which takes away the hard work, and allows easy and quick system updates from a working copy. Being one of the toughest Debian nuts I know, Joost of course packaged it all as a Debian package and called it Caspar. You can easily get it:

$ sudo apt-get install caspar caspar-doc

If you have this installed and have set up a few simple files in your working copy /etc directory tree, just typing

$ make

is all you need to do to copy the relevant files, and only those, from your working copy to the live /etc directory of the machine. Yes, you can do this in many other ways, but Caspar is the way you likely would do it anyway.

Caspar was originally developed with a much larger scale installation in mind: several dozens of servers, maintained by a handful of people that shared the workload. Each system administrator had a secure workstation with all working copies of all systems installed, and access to the actual systems was via ssh and private keys for security and convenience. Any config update would be done locally on the secure management machine, pushed over for testing, when approved checked into Subversion, and pushed over again to get the right revision number installed at the remote machine. Several Caspar features are designed to facilitate this work model, including rolling out similar changes over groups of similar machines. For any medium-size outfit, this looks like a good model.

For myself, it is a bit over the top. I have only very few servers to maintain, and I prefer to have direct access to all of them while working. So I don't use the concept of a management host (though I maintain a high level of security on my mobile display and keyboard). I have secure ssh accounts on all hosts, and use the local copy on the host itself, using "sudo cp" to make the actual changes to /etc . Caspar easily supports this model, plus quite a few more.

Setting Up Caspar

The Caspar man page has all the documentation you'll need, but I can of course tell you how I've set up my systems. Since Caspar is not rocket science, I am confident that you can figure out what you want and how to do it yourself, and else the author Joost van Baal is quite responsive to requests.

At the top of my local working copy, I created include/install.mk which contains the following lines (and Joost helped):

csp_UHOST = dummy
csp_PUSH = $(csp_sudocp_FUNC)
csp_sudocp_FUNC = sudo $(csp_CP) $(csp_CPFLAGS) $(1) $(3)
include caspar/mk/caspar.mk

The first line is required because the original Caspar working model assumes you want to push changes to a remote host. The last line includes the actual Caspar functionality from the Caspar package. The two lines in the middle set up a simple function/macro that performs "sudo cp" to copy changed files to their destination. That's it!

In each subdirectory you'll need a Makefile which contains at least two lines. For example, in etc/init.d/Makefile, I have:

csp_DIR = /etc/init.d/
include ../../include/install.mk

The first line tells Caspar where to copy the relevant files, and the second line reads in the general config file which we set up above. And again that's it. Do you see now why there is little to gain by building your own copy-to-/etc system?

Of course, Caspar can do much more, and this comes with more options on the man page. But the bare minimum as outlined here is sufficient for most directories. Excluding files is done by csp_TABOOFILES_ADD lines, which are great for helper or source files, and for local documentation files, that have no use or are unwanted in the actual /etc branch. My typical example are the HTTPS certificate signing request files which I want to keep, but not in the /etc/apache2 branch. And many directories also have my README which is not needed in /etc (but would not hurt either).

I am quite curious to learn how other people maintain their configurations, also on graphically oriented systems. You are invited to add a comment to this post, or else just drop me an e-mail.

Jeroen