Procmail

Being a celebrity on the Internet means that you get a lot of attention, just as celebrities do in the real world. The good news is that everyone can become celebrities: simply join a few public mailing lists, get yourself a home page, and you are all set. The bad news is that the attention is from spammers, who send you an enormous amount of suggestions about how you can become richer, extend certain body parts, and take most of their wealth if you want to help them get it out of Iraq.

The virtual bodyguards of your mail are a couple called Procmail and SpamAssassin. Procmail is a general-purpose mail filter, while SpamAssassin is a dedicated mail filter for fighting spam and the like (worms, viruses, etc.). This section discusses Procmail, and the next section is devoted to SpamAssassin.

Procmail Concepts

To understand Procmail, we need to start looking at how it is invoked. The usual sequence is that mail arrives at your account, and your MUA calls Procmail, giving it the mail as argument. The terms filter or rule, in many mail filtering programs, refer to both a set of conditions to check messages for and an action to perform on the messages that meet those conditions (such as putting them in a particular folder). Procmail refers to this set as a recipe, a term we will use throughout this section to describe each set of paired conditions and actions. Procmail goes through each of its recipes until one marks the mail as delivered. If no recipe blocks the mail, it is delivered in your inbox as if Procmail had never been in the picture.

Each recipe consists of two things: a set of conditions and a set of actions. The actions of a recipe are executed if all its conditions are met. In addition, a recipe may mark mail as delivered as described earlier.

The conditions may include the following:

The actions may include the following:

  • Reply to the sender that you are on holiday.

  • Forward the letter to another person.

  • Save the letter to a file.

  • Change some part of the letter (e.g., add a new header field, add some text to it etc.).

Before you dig too much into the details of this section, you should ask yourself if you really want to use Procmail at all. Many mail clients allow you to sort mail, and if we take KMail as an example, then it is much easier to use than Procmail. The following is a list of reasons why you may still want to use Procmail:

  • You are using a number of different mail clients, not always the same. For example, when you are on the road you use a web mail interface, but when you are home you use a normal mail client such as KMail or mutt.

  • You want to filter your mail the second it arrives, not at a later point when your mail client downloads it—an example of this may be out-of-the-office replies.

  • The amount of mail coming to your account is so big that you want filtering to be done before mail is loaded into your client (your client may be slow at filtering mail).

Preparing Procmail for Use

Procmail comes with most modern Linux systems nowadays, but should it not be available for your system, then you should have a look at http://www.procmail.org. At this site you will also find a large collection of sample recipes.

When you have ensured that Procmail is on your system, it is time to check if it is invoked by your MUA. The easiest way to do so is to place the following .procmailrc file in your home directory, and send yourself an email.

SHELL=/bin/sh
MAILDIR=${HOME}/Mail
LOGFILE=${MAILDIR}/procmail.log
LOG="--- Logging ${LOGFILE} for ${LOGNAME}, "

If the ~/Mail directory does not exist, then you need to create it for this script to work. If you store your email elsewhere, replace ${HOME}/Mail with the alternative location. Also please check that /bin/sh exists (it's quite likely that it does); otherwise, adapt the script.

If Procmail is invoked by default, then the file just shown should give you ~/Mail/procmail.log, with content similar to the following:

--- Logging /home/test/Mail/procmail.log for test, From blackie@blackie.dk  
Fri Mar 18 12:25:23 2005
 Subject: Fri Mar 18 12:25:22 CET 2005
  Folder: /var/spool/mail/test

If this file didn't come into existence by sending yourself an email, don't panic. All you need to do is to add the following line to the ~/.forward file:

|IFS=' ' && exec /usr/bin/procmail || exit 75 #myid

Replace /usr/bin/procmail with the path to your system's Procmail binary, and replace myid with your login name. (This part is necessary to avoid problems with MUAs trying to optimize mail delivery.)

Now send yourself a mail again, and check if it works this time. If you still do not see the file, then it might be a result of a system that is too closed. Check that the .procmailrc and .forward files are readable by others, and perhaps only writable by yourself. Possibly you also need to add the x flag to the attributes of your home directory, which you may do with this command:

chmod go+x ~/

If things still do not work, then it is time to panic—or at least consult the vendor of your Linux system.

Are you ready to lose your email while playing with Procmail ? If not, then it might be a good idea to create a sandbox for your tests. To do so, create a Test directory, and copy your .procmailrc file into that directory and name it proctest.rc. Now edit this file instead of your real .procmailrc file when testing things. In the Test directory, create this shell script:

#!/bin/sh
#The executable file named "proctest"
#
# You need a test directory.
TESTDIR=~/Test
if [ ! -d ${TESTDIR} ] ; then
  echo "Directory ${TESTDIR} does not exist; First create it"
  exit 0
fi

procmail ${TESTDIR}/proctest.rc < mail.msg

You may wish to adjust the LOGFILE line of the proctest.rc file so it doesn't write to your existing logfile, but instead simply writes to a logfile in the Test subdirectory. You may also want to add the following line to get improved debugging output from Procmail:

VERBOSE=yes
LOGABSTRACT=all

Finally you are ready to run the tests, which you do by placing a mail message in the file mail.msg, and running the script proctest. Most email programs allow you to save just one email to a file. Alternatively, send yourself an email, and copy it out of your /var/spool/mail/your-login file.

Recipe Syntax

With all the preparation done, we may now start looking at recipes. Recipes all follow this style:

:0 [flags] [ : [locallockfile] ]
  <0 or more conditions (one per line)>
  <exactly one action line>

Conditions start with a leading *. Everything after that character is passed on to the internal egrep literally, except for leading and trailing whitespace.

The action line may take several forms:

  • If it starts with a !, then the rest of the line is considered an email address to forward to.

  • If it starts with a |, then the rest of the line is considered a shell command to be executed.

  • If it starts with a {, then everything until the matching } is considered a nested block. Nested blocks consist of a number of recipes.

  • Anything else is considered a mailbox name.

The flags are a combination of a number of one-letter flags. The flags are described in Table 23-1 (taken from the procmailrc manpage). There is no need to read the table in detail now; instead, simply look back to it as we show examples in the following sections.

Table 23-1. Procmail flags

Flag

Function

H

Perform an extended regular expression search on the header (default).

B

Perform an extended regular expression search on the body.

D

Check against the regular expression in a case-sensitive manner (default is case-insensitive).

A

Execute the recipe only if there was a match on the most recent recipe without an A or a flag in the current block nesting level.

a

Same as A, but the preceding recipe must have completed successfully.

E

Execute the recipe only if the immediately preceding recipe was not executed. Execution of this recipe also disables any immediately following recipes with the E flag. This allows you to specify else if actions.

e

Execute the recipe only if the immediately preceding recipe was executed but did not complete successfully.

h

Send contents of header to the pipe, file, or mail destination (default).

b

Send contents of body to the pipe, file, or mail destination (default).

c

Create a copy of the mail message so it can be further processed by a later recipe, or delivered.

w

Wait for processing program and check its exit code.

W

Same as w, but suppresses any Program failure message.

i

Ignore any write errors (usually due to a closed pipe).

r

Raw mode. Do not ensure that message ends with an empty line.

Conditions are generally regular expressions found in the header or body of the email. Regular expressions are covered in Chapter 19. But some other special conditions can be used. To select them, the condition must start with one of the flags shown in Table 23-2.

Table 23-2. Procmail condition flags

Condition

Function

!

Act only if the specified condition is false.

$

Interpret text with double quotes in the rest of this condition as it would be interpreted in the bash shell

?

Use the exit code.

<

Run recipe on messages with a total length less than the following number.

>

Run recipe on messages with a total length greater than the following number.

variable

Match the following text against the value of this variable, which can be an environment variable or a combination of B for body and H for header

\

Escapes (leaves as a plain character) any of the entries in this table when it should start the line as a plain character without special meaning.

Examples

Procmail recipes are most easily understood through a number of examples, so the rest of this section will show examples of normal Procmail usage. See the manpage procmailex for more examples.

Each of the examples are simple recipes, not complete Procmail scripts, so you still need the initial content shown in "Preparing Procmail for Use."

Finally, when playing with recipes, remember that Procmail processes them in order. Thus, if a recipe marks mail as delivered, it doesn't show up with other recipes.

When you are playing around with Procmail, there is a risk that you might develop a recipe that throws away messages that should not have been thrown away. It is therefore a very good idea to use the following recipe which is supposed to be the very first recipe in your Procmail setup:

:0c:
backup

The first line of this recipe has the flag c, which says that even though this recipe matches (which it always will, as the recipe has no conditions), the mail should continue on to other recipes, rather than be stopped here.

After the flag, there is a colon, indicating that the recipe should use a local lock. Thus, before the actions of this recipe can execute, the lock must first be obtained; while the action is executing, the lock will be in place.

The final part of the recipe is the text backup, which indicates that the mail will end in the mailbox named backup. If $MAIL/backup is a directory, the mail will be put in a unique named file in that directory (this is known as maildir storage). Alternatively, if $MAIL/backup is a file, the mail will be appended to that file (this is known as mbox storage).

The next recipe might be what you most often do with Procmail — namely, to save mail from a mailing list into a dedicated mailbox. This is done with a recipe looking like this:

:0:
* Return-Path:.*kde-devel-bounces
kde-devel

Notice that this time we do not use the c flag, because we want mail from this mailing list to stay in the kde-devel mailbox, and not get to our inbox.

The line starting with an asterisk is the condition that must be met for this recipe to be triggered. This line is a regular expression that says that the header of the mail must contain the text Return-Path:, then any text (the regular expression .*), and then the text kde-devel-bounces. We got the idea for this regular expression by looking in an email from the mailing list. The trick is always to find a regular expression that will match any mail from the mailing list, but not match any other mail.

The following recipe forwards a message with a subject starting with the text SMS to a mobile phone in the form of a text message through an imaginary email-to-SMS gateway.

:0
* < 1000
* Subject: SMS
! 12345678@smsgateway.com

This recipe contains two conditions: the first is that the overall size of the letter be less than 1000 bytes, and the second is that the subject should start with SMS.

The action of this recipe starts with an exclamation mark, which indicates that the message is forwarded to the address following the exclamation mark.

The final example we show is how to send an out-of-office reply. Many systems provide a program named vacation that does this in a fairly robust way, but we provide something more customizable here so you can vary the message in any way your scripting skills allow. The basic recipe looks like this:

:0c
* !^FROM_DAEMON
* !^X-Loop: your@own.mail.address
{
   SUBJECT=`formail -zx subject:`

   :0
   | (formail -r -I"Precedence: junk" \
      -A"X-Loop: your@own.mail.address" ; \
      echo "I recived the mail with the subject \"$SUBJECT.\""; \
      echo "I'm out of the office and will answer it as soon as possible") | $SENDMAIL -t
}

Starting with the conditions again, this recipe sends an out-of-office reply only if (1) the mail is not from a mailer daemon, and (2) the mail does not contain the header line (this should, of course, be replaced with your actual email address). The first condition ensures we do not send out-of-office replies to mailing lists, and the second condition ensures we do not end up in a mail loop with someone else's out-of-office filter.

The action to take when these two conditions are met is a block of recipes. Whatever it says in between the braces is interpreted as if it were a normal Procmail script. If execution makes it to the end of the block (i.e., the mail has not yet been delivered), it will continue execution outside the block. This is, however, not the case in our setup.

The first line of the block is an assignment to the variable SUBJECT. The value comes from standard output from the formail command. This is a binary shipped with Procmail; its purpose is to either manipulate the emails or subtract part of them.

The second part of the block is the part that does the core work. It composes an answer and mails it back to the person who originally sent you an email. Let's take it bit by bit.

First we call formail -r to create an auto respond header from the incoming mail. That means that it will throw away headers that you do not want in the reply. We also hand the command-line options -I"Precedence: junk" and -A"X-Loop: " to formail. These two switches basically add new headers to the mail: the first telling the precedence of the mail, and the second adding the line that our condition checks against in order to avoid mail loops.

So far we have echoed the header of the reply mail to standard output. On standard output, we next print the out-of-office reply (i.e., the body part of the email). The whole mail is finally sent to sendmail. The -t option tells sendmail to look into the mail to figure out who it is meant for.