# Background

Here we cover the advantages of filtering mail during an incoming SMTP transaction, rather than following the more conventional approach of offloading this task to the mail routing and delivery stage. We also provide a brief introduction to the SMTP transaction.

## Why Filter Mail During the SMTP Transaction?

### Status Quo

If you receive spam, raise your hands. Keep them up.

If you receive computer virii or other malware, raise your hands too.

If you receive bogus s (DSNs), such as Message Undeliverable, Virus found, Please confirm delivery, etc, related to messages you never sent, raise your hands as well. This is known as .

This last form is particularly troublesome, because it is harder to weed out than standard spam or malware, and because such messages can be quite confusing to recipients who do not possess godly skills in parsing message headers. In the case of virus warnings, this often causes unnecessary concern on the recipient's end; more generally, a common tendency will be to ignore all such messages, thereby missing out on legitimate DSNs.

Finally, I want those of you who have lost legitimate mail into a big black hole - due to misclassification by spam or virus scanners - to lift your feet.

If you were standing before and are still standing, I suggest that you may not be fully aware of what is happening to your mail. If you have been doing any type of spam filtering, even by manually moving mails to the trash can in your mail reader, let alone by experimenting with primitive filtering techniques such as DNS blacklists (SpamHaus, SPEWS, SORBS...), chances are that you have lost some valid mail.

### The Cause

Spam, just like many other artifacts of greed, is a social disease. Call it affluenza, or whatever you like; lower life forms seek to destroy a larger ecosystem, and if successful, will actually end up ruining their own habitat in the end.

Larger social issues and philosophy aside: You - the mail system administrator - face the very concrete and real life dilemma of finding a way to deal with all this junk.

As it turns out, there are some limitations with the conventional way that mail is being processed and delegated by the various components of mail transport and delivery software. In a traditional setup, one or more (s) accept most or all incoming mail deliveries to addresses within a domain. Often, they then forward the mail to one or more internal machines for further processing, and/or delivery to the user's mailboxes. If any of these servers discovers that it is unable to perform the requested delivery or function, it generates and returns a DSN back to the sender address in the original mail.

As organizations started deploying spam and virus scanners, they often found that the path of least resistance was to work these into the message delivery path, as mail is transferred from the incoming (s) to internal delivery hosts and/or software. For instance, a common way filter out spam is by routing the mail through SpamAssassin or other software before it is delivered to a user's mailbox, and/or rely on spam filtering capabilities in the user's .

Options for dealing with mail that is classified as spam or virus at this point are limited:

- You can return a back to the sender. The problem is that nearly all spam and e-mail borne virii are delivered with faked sender addresses. If you return this mail, it will invariably go to innocent third parties -- perhaps warning a grandmother in Sweden, who uses Mac OS X and does not know much about computers, that she is infected by the Blaster worm. In other words, you will be generating .

- You can drop the message into the bit bucket, without sending any notification back to the sender. This is an even bigger problem in the case of s, because neither the sender nor the receiver will ever know what happened to the message (or in the receiver's case, that it ever existed).

- Depending on how your users access their mail (for instance, if they access it via the IMAP protocol or use a web-based mail reader, but not if they retreive it over POP-3), you may be able to file it into a separate junk folder for them -- perhaps as an option in their account settings.

  This may be the best of these three options. Even so, the messages may remain unseen for some time, or simply overlooked as the receiver more-or-less periodically scans through and deletes mail in their Junk folder.

### The Solution

As you would have guessed by now, the One True solution to this problem is to do spam and virus filtering during the SMTP dialogue from the remote host, as the mail is being received by the inbound mail exchanger for your domain. This way, if the mail turns out to be undesirable, you can issue a SMTP reject response rather than face the dilemma described above. As a result:

- You will be able to stop the delivery of most junk mail early in the SMTP transaction, before the actual message data has been received, thus saving you both network bandwidth and CPU processing.

- You will be able to deploy some spam filtering techniques that are not possible later, such as and .

- You will be able to notify the sender in case of a delivery failure (e.g. due to an invalid recipient address) without directly generating

  We will discuss how you can avoid causing collateral spam indirectly as a result of rejecting mail forwarded from trusted sources, such as mailing list servers or mail accounts on other sites Untrusted third party hosts may still generate collateral spam if you reject the mail. However, unless that host is an or , it presumably delivers mail only from legitimate senders, whose addresses are valid. If it is an Open Proxy or SMTP Relay - well, it is better that you reject the mail and let it freeze in their outgoing mail queue than letting it freeze in yours. Eventually, this ought to give the owners of that server a clue. .

- You will be able to protect yourself against collateral spam from others (such as bogus You have a virus messages from anti-virus software).

OK, you can lower your hands now. If you were standing, and your feet disappeared from under you, you can now also stand up again.

## The Good, The Bad, The Ugly

Some filtering techniques are more suitable for use during the SMTP transaction than others. Some are simply better than others. Nearly all have their proponents and opponents.

Needless to say, these controversies extend to the methods described here as well. For instance:

- Some argue that penalize individual mail senders purely based on their Internet Service Provider (ISP), not on the merits of their particular message.

- Some point out that ratware traps like and are easily overcome and will be less effective over time, while continuing to degrade the Quality of Service for legitimate mail.

- Some find that like the give ISPs a way to lock their customers in, and do not adequately address users who roam between different networks or who forward their e-mail from one host to another.

I will steer away from most of these controversies. Instead, I will try to provide a functional description of the various techniques available, including their possible side effects, and then talk a little about my own experiences using some of them.

That said, there are some filtering methods in use today that I deliberately omit from this document:

- Challenge/response systems (like TMDA). These are not suitable for SMTP time filtering, as they rely on first accepting the mail, then returning a confirmation request to the . This technique is therefore outside the scope of this document. Personally I do not think challenge/response systems are a good idea in any case. They generate , they require special attention for mail sent from automated sources such as monthly bank statements, and they degrade the usability of e-mail as people need to jump through hoops to get in touch with each other. Many times, senders of legitimate mail will not bother to or know that they need to follow up to the confirmation request, and the mail is lost.

- . These require training specific to a particular user, and/or a particular language. As such, these too are not normally suitable for use during the SMTP transaction (But see ).

- are not really suitable for weeding out junk mail until all the world's legitimate mail is sent with a virtual postage stamp. (Though in the mean time, they can be used for the opposite purpose - that is, to accept mail carrying the stamp that would otherwise be rejected).

Generally, I have attempted to offer techniques that are as precise as possible, and to go to great lengths to avoid s. People's e-mail is important to them, and they spend time and effort writing it. In my view, willfully using techniques or tools that reject large amounts of legitimate mail is a show of disrespect, both to the people that are directly affected and to the Internet as a whole. My view stands in sharp contrast to that of a large number of spam hacktivists, such as the maintainers of the SPEWS blacklist. One of the stated aims of this list is precisely to inflict as a means of putting pressure on ISPs to react on abuse complaints. Listing complaints are typically met with knee-jerk responses such as bother your ISP, not us, or get another ISP. Often, these are not viable options. Consider developing countries. For that matter, consider the fact that nearly everywhere, broadband providers are regulated monopolies. I believe that these attitudes illustrate the exact crux of the problem with trusting these groups. Put plainly, there are much better and more accurate ways available to filter junk mail.

This is especially true for SMTP-time system wide filtering, because end recipients usually have little or no control over the criteria being used to filter their mail.

## The SMTP Transaction

SMTP is the protocol that is used for mail delivery on the Internet. For a detailed description of the protocol, please refer to RFC 2821, as well as Dave Crocker's introduction to Internet Mail Architecture.

Mail deliveries involve an SMTP transaction between the connecting host (client) and the receiving host (server). For this discussion, the connecting host is the peer, and the receiving host is your server.

In a typical SMTP transaction, the client issues SMTP commands such as `EHLO`, `MAIL FROM:`, `RCPT TO:`, and `DATA`. Your server responds to each command with a 3-digit numeric code indicating whether the command was accepted (`2`xx``), was subject to a temporary failure or restriction (`4`xx``), or failed definitively/permanently (`5`xx``), followed by some human readable explanation. A full description of these codes is included in RFC 2821.

A best case scenario SMTP transaction typically consists of the following relevant steps:

### Simple SMTP dialogue

Client | Server
--- | ---

Initiates a TCP connection to server.

Presents an SMTP banner - that is, a greeting that starts with the code `220` to indicate that it is ready to speak SMTP (or usually ESMTP, a superset of SMTP):

    220 your.f.q.d.n ESTMP...

Introduces itself by way of an Hello command, either `HELO` (now obsolete) or `EHLO`, followed by its own :

    EHLO peers.f.q.d.n

Accepts this greeting with a `250` response. If the client used the extended version of the Hello command (`EHLO`), your server knows that it is capable of handling multi-line responses, and so will normally send back several lines indicating the capabilities offered by your server:

    250-your.f.q.d.n Hello ...
    250-SIZE 52428800
    250-8BITMIME
    250-PIPELINING
    250-STARTTLS
    250-AUTH
    250 HELP

If the `PIPELINING` capability is included in this response, the client can from this point forward issue several commands at once, without waiting for the response to each one.

Starts a new mail transaction by specifying the :

    MAIL FROM:<sender@address>

Issues a `250` response to indicate that the sender is accepted.

Lists the s of the message, one at a time, using the command:

    RCPT TO:<receiver@address>

Issues a response to each command (`2`xx``, `4`xx``, or `5`xx``, depending on whether delivery to this recipient was accepted, subject to a temporary failure, or rejected).

Issues a `DATA` command to indicate that it is ready to send the message. | Responds `354` to indicate that the command has been provisionally accepted.
Transmits the message, starting with RFC 2822 compliant header lines (such as: `From:`, `To:`, `Subject:`, `Date:`, `Message-ID:`). The header and the body are separated by an empty line. To indicate the end of the message, the client sends a single period (".") on a separate line. | Replies `250` to indicate that the message has been accepted.
If there are more messages to be delivered, issues the next `MAIL FROM:` command. Otherwise, it says `QUIT`, or in rare cases, simply disconnects. | Disconnects.