Make syslog work harder, and spend less time looking through huge log files.
The default syslog installation on many distributions doesn’t do a very good job of filtering classes of information into separate files. If you see a jumble of messages from Sendmail, sudo, BIND, and other system services in /var/log/messages, you should probably review your /etc/syslog.conf file.
Syslog can filter on a number of facilities and priorities, including auth, auth-priv, cron, daemon, kern, lpr, mail, news, syslog, user, uucp, and local0 through local7. In addition, each facility can have one of eight priorities: debug, info, notice, warning, err, crit, alert, and emerg.
Note that most applications decide for themselves what facility and priority to log at (although the best apps let you choose), so they might not be logged as you expect. Here’s a sample /etc/syslog.conf that attempts to shuffle around what gets logged where:
auth.warning /var/log/auth mail.err /var/log/maillog kern.* /var/log/kernel cron.crit /var/log/cron *.err;mail.none /var/log/syslog *.info;auth.none;mail.none /var/log/messages #*.=debug /var/log/debug local0.info /var/log/cluster local1.err /var/log/spamerica
All of the lines in this example log the specified priority (or higher) to the respective file. The special priority none tells syslog not to bother logging the specified facility at all. The local0 through local7 facilities are supplied for use with your own programs, however you see fit. For example, the /var/log/spamerica file fills with local1.err (or higher) messages that are generated by our spam processing job. It’s nice to have those messages separate from the standard mail delivery log (which is in /var/log/maillog).
The commented *.=debug line is useful when debugging daemonized services. It tells syslog to specifically log only debug priority messages of any facility, and it generally shouldn’t be running (unless you don’t mind filling your disks with debug logs).
Another approach is to log debug information to a FIFO. This way, debug logs take up no space, but they will disappear unless a process is watching the FIFO. To log to a FIFO, first create it in the filesystem:
# mkfifo -m 0664 /var/log/debug
Then, uncomment the debug line in syslog.conf and amend it to include a |, like this:
*.=debug |/var/log/debug
Now, debug information will be constantly logged to the FIFO and can be viewed with a command like less -f /var/log/debug.
A FIFO is also handy if you want a process to constantly watch all system messages and perhaps notify you via email about any critical messages. Try making a FIFO called /var/log/monitor and adding a rule like this to your syslog.conf:
*.* |/var/log/monitor
Now, every message (at every priority) is passed to the /var/log/monitor FIFO, and any process watching it can react accordingly, all without taking up any disk space.
If you notice a bunch of lines like the following in /var/log/messages, you might be wondering why they’re there:
Dec 29 18:33:35 catlin -- MARK -- Dec 29 18:53:35 catlin -- MARK -- Dec 29 19:13:35 catlin -- MARK -- Dec 29 19:33:35 catlin -- MARK -- Dec 29 19:53:35 catlin -- MARK -- Dec 29 20:13:35 catlin -- MARK -- Dec 29 20:33:35 catlin -- MARK -- Dec 29 20:53:35 catlin -- MARK -- Dec 29 21:13:35 catlin -- MARK --
These are generated by the mark functionality of syslog, as a way of “touching base” with the system, so that you can (theoretically) tell if syslog has unexpectedly died. This generally only serves to fill up your log files, and unless you are having problems with syslog, you probably don’t need it. To turn this function off, pass the -m 0 switch to syslogd (after first killing any running syslogd processes), like this:
# killall syslogd; /usr/sbin/syslogd -m 0
If all of this fiddling about with facilities and priorities strikes you as arcane Unix-speak, you’re not alone. These examples are provided for systems that include the default (and venerable) syslogd. If you have the opportunity to install a new syslogd, you will likely want to look into syslog-ng . This new implementation of syslogd allows much more flexible filtering and offers a slew of new features. “Aggregate Logs from Remote Sites” [Hack #84] takes a look at some of what is possible with syslog-ng.
Rob Flickenger