Mail Filtering Best Practices

Thu Feb 20 15:17:45 PST 2003

>Date: Thu, 20 Feb 2003 14:42:52 -0800
>From: Danny Howard <dannyman at toldme.com>

>It is not difficult to cache message-ids and deliver subsequent
>identical ones to /dev/null.

That may be, but dealing with the occasional source of messages that
does not supply a Message-Id header at all will tend to make this rather
less effective than it might otherwise be.  :-(

I get fairly aggressive about dealing with spam sometimes.  And I do the
filtering at the MTA.

The results of that filtering got to the point recently where it seemed
to me that a significant amount of the remaining spam -- gut feel was
around 30 - 50% -- had the distinguishing characteristic that the
messages in question arrived without a Message-Id header at all.

Now, in looking over RFC 2822, the Message-Id is an optional header; it
merely SHOULD (vs. MUST) be supplied.  However, I can think of no
legitimate justification for sending a message without one.

So I went ahead & implemented that check in the sendmail.cf I use for
the SMTP server here (catwhisker.org, not baylisa.org).

About a week later, I had occasion to interact witha mailing list server
over at LISTSERV.NODAK.EDU.  They have a straightforward-seeming Web
form for specifying stuff, and a CGI that reads the results of the form
and sends email to the specified address, requesting that you confirm
that you really want to be subscribed to the list.

Now, make no mistake:  I consider that much of the implementation a Good
Thing.

As it happens, because I had recently made the above-described change, I
happened to have a window open, doing a

	tail -F /var/log/maillog.0 | egrep '(reject=|did not issue)'

just so I could be aware of collateral damage from that change.

When I did not receive the confirmation within a couple of minutes, I
looked at the message log.  Sure enough:

Feb 16 18:17:30 janus sm-mta[60727]: h1H2HUi9060727: ruleset=check_eoh, arg1=5, arg2=406, relay=listserv.NoDak.edu [134.129.111.8], reject=553 5.0.0 Do not expect me to track your messages for you
Feb 16 18:17:31 janus sm-mta[60727]: h1H2HUi9060727: from=<LISTSERV at LISTSERV.NODAK.EDU>, size=1570, class=0, nrcpts=1, msgid=<200302170217.h1H2HUi9060727 at janus.catwhisker.org>, bodytype=8BITMIME, proto=ESMTP, daemon=MTA, relay=listserv.NoDak.edu [134.129.111.8]
Feb 16 18:17:31 janus sm-mta[60727]: h1H2HUi9060727: to=<david at CATWHISKER.ORG>, delay=00:00:01, pri=30406, stat=Do not expect me to track your messages for you

Nuts.  Since I really did want to subscribe to the list, I put an
exemption in for that machine.  :-(

A day later, I sent a query off to postmaster at listserv.nodak.edu, asking
if there actually was a reason they sent out messages without Message-Id
headers.  No response to date; I'm not holding my breath.  (OK; I confess
that I strongly suspect that the term "LISTSERV" explains a great deal
of the misconfiguration that I perceive.)

Oh -- I'll gladly receive suggestions for improving the message.  :-}

And if folks think such a check ought to be implemented for baylisa.org,
I'm willing to discuss it, and possibly even do it.  :-}

Cheers,
david       (links to my resume at http://www.catwhisker.org/~david)
-- 
David H. Wolfskill				david at catwhisker.org
WARNING: Use of Microsoft products may be hazardous to your system's integrity.