Linux Systems Design & Support

Procmail documentation (Large)

Table of contents

1.0 Document id
1.1 General
1.2 What is Procmail?
1.3 Abbreviations and thanks
1.4 Version information
1.5 Document layout and maintenance
1.6 About presented recipes
1.7 Variables used in recipes
1.8 About "useless use of cat award"

2.0 UBE in Internet
2.1 Terms used and foreword
2.2 UBE strategies
2.3 UBE and bouncing message back
2.4 UBE and "I don't mind" attitude
2.5 We need a law against UBE

3.0 Anti-UBE pointers
3.1 NoCEM, CAUCE and others
3.2 General Filtering pages (more than procmail)
3.3 Junk email and spam
3.4 Comprehensive list of spammers
3.5 Misc pointers
3.6 Questionable UBE stop services
3.7 UBE related newsgroups or mailing lists
3.8 Software: adcomplain -- Perl junk email rport
3.9 Software: Ricochet -- Perl junk email rport
3.10 Software: yell -- perl
3.11 Software: RBL lookup tool -- C
3.12 Software: mapSoN
3.13 Software: spamgard
3.14 Software: Spam Be Gone
3.15 Software: ifile - Perl
3.16 Software: ClearMail

4.0 Procmail pointers
4.1 Where to get procmail binary
4.2 Where is procmail developed
4.3 About procmail's Y2K compliance
4.4 Procmail mailing lists
4.5 Procmail recipe modules and faqs
4.6 Procmail mode for Emacs
4.7 Procmail module list
4.8 Where to get Procmail code and modules
4.9 Procmail code to filter UBE

5.0 Dry run testing
5.1 What is dry run testing
5.2 Why the From field is not okay after dry run
5.3 Getting default value of a procmail variable

6.0 Things to remember
6.1 Get the newest procmail
6.2 Csh's tilde is not supported
6.3 Be sure to write the recipe start right
6.4 Always set SHELL
6.5 Check and set PATH
6.6 Keep the log on all the time
6.7 Never add a trailing slash for directories
6.8 Remember what term DELIVERED means
6.9 Beware putting comment in wrong place
6.10 Brace placement
6.11 Local lockfile usage
6.12 Global lockfile
6.13 Gee, where do I put all those ! * $ ??
6.14 Sending automatic reply, use X-loop header
6.15 Avoid extra shell layer (check command for SHELLMETAS)
6.16 Think what shell commands you use
6.17 Using absolute path when calling a shell program
6.18 Disabling recipe temporarily
6.19 Keep message backup, no matter what
6.20 Order of the procmail recipes

7.0 Procmail flags
7.1 The order of the flags
7.2 Flag w and recipe with |
7.3 Flag w, lockfile and recipe with |
7.4 Flag f and w together
7.5 Flags h and b
7.6 Flag h and sinking to /dev/null
7.7 Flag i and pipe flag f
7.8 Flag r
7.9 Flag c's background
7.10 Flag c before nested block forks a child
7.11 Flag c and understanding possible forking penalty
7.12 Flags before nested block
7.13 Flags aAeE tutorial

8.0 Matching and regexps
8.1 Philosophy of abstraction in regexps
8.2 Matches are not case sensitive
8.3 Procmail uses multiline matches
8.4 Headers are folded before matching
8.5 Improving Space-Tab syndrome
8.6 Handling exclamation character
8.7 Rules for generating a character class
8.8 Matching space at the end of condition
8.9 Beware leading backslash
8.10 Correct use of TO Macro
8.11 Procmail's regexp engine
8.12 Procmail and egrep differences
8.13 Undesrtanding procmail's minimal matching (stingy vs. greedy)
8.14 Explaining \/ and ()\/
8.15 Explaning ^^ and ^
8.16 ANDing traditionally
8.17 ORing traditionally
8.18 ORing and score recipe
8.19 ORing by using De Morgan rules

9.0 Variables
9.1 Setting and unsetting variables
9.2 Variable initialisation and sh syntax
9.3 Testing variables
9.4 What does $\VAR mean?
9.5 Common pitfalls when using variables
9.6 Quoting: Using single or double quotes
9.7 Quoting: Passing values to an external program
9.8 Passing values from an external program
9.9 Incrementing a variable by a value N
9.10 Comparing values
9.11 Strings: How many characters are there in a given string?
9.12 Strings: How to strip trailing newline.
9.13 Strings: deriving the last N characters of a string.
9.14 Strings: Getting partial matches from a string.
9.15 Strings: Procmail string manipulation example
9.16 How to raise a flag if the message was filed
9.17 Dollar sign in condition lines.
9.18 Finding mysterious foo variable
9.19 Storing code to variable
9.20 Getting headers into a variable.
9.21 Converting value to lowercase

10.0 Suggestions and miscellaneous
10.1 Speeding up procmail
10.2 See the procmail installation's examples
10.3 Printing statistics of your incoming mail
10.4 Storing UBE mailboxes outside of quota
10.5 Using first 5-30 lines from the message
10.6 Using cat or echo in scripts?
10.7 How to run an extra shell command as a side effect?
10.8 Forcing "ok" return status from shell script
10.9 Make your own .procmailrc available to others
10.10 Using dates efficiently
10.11 Keep simple header log
10.12 Gzipping messages
10.13 Emergency stop for your .procmailrc

11.0 Scoring
11.1 Using scores by an example
11.2 Brief Score tutorial
11.3 Score's scope
11.4 Counting length of a string
11.5 Counting lines in a message (Adding Lines: header)
11.6 Determining if body is longer than header
11.7 Matching last Received header
11.8 How to add Content-Length header
11.9 Testing message size or number of lines
11.10 Counting commas with recursive includerc
12.0 Formail usage
12.1 Fetching fields with formail -x
12.2 Always use formail's -rt switch
12.3 Using -rt and rewriting the From address
12.4 Formail -rt and Resent-From header
12.5 Quoting the message
12.6 Without quoting the message
12.7 How to include headers and body to the reply message
12.8 Adding text to the beginning of message
12.9 Adding text to the end of message
12.10 How to truncate headers (save filing space)
12.11 Adding extra headers from file
12.12 Splitting digest
12.13 Mailbox: Splitting to individual files
12.14 Mailbox: Extracting all From addresses from mailbox
12.15 Mailbox: Applying procmail recipe on whole mailbox
12.16 Mailbox: run series of commands for each mail (split mailbox)
12.17 Option -D and cache
12.18 Option -D and message-id in the body
12.19 Reducing formail calls (conditionally adding fields)
12.20 Formail -A -a options
12.21 Formail -e -s options

13.0 Saving mailing list messages
13.1 Using subroutine pm-jalist.rc to detect mailing lists
13.2 Using plus addressing foo+bar@address.com
13.3 Using RFC comment trick for additional information
13.4 Simple mailing list handling
13.5 Archiving according to TO
13.6 Using Return-Path to detect mailing lists

14.0 Procmail, MIME and HTML
14.1 Mime Bibliography
14.2 Mime notes
14.3 Software to deal with mime or html
14.4 Mime content type application/ms-tnef
14.5 Trapping html mime messages
14.6 Complaining about html messages
14.7 Converting HTML body to plain text
14.8 Getting rid of unwanted mime attachments (html, vcard)
14.9 Sending contents of a html page in plain text to someone

15.0 Simple recipe examples
15.1 Saving: MH folders -- numbered messages
15.2 Saving: to monthly folders
15.3 Modifying: Filtering basics
15.4 Modifying: Squeezing empty lines around message body
15.5 Modifying: shuffling headers always to same order
15.6 Service: Auto answerer to empty messages
15.7 Service: File server -- send fileas as attachments upon request
15.8 Service: Ping responder
15.9 Service: simple vacation with procmail
15.10 Service: vacation code example
15.11 Service: Auto-forwarding
15.12 Service: forward only specific messages
15.13 Service: Making digests
15.14 Kill: killing advertisement headers and footers
15.15 Kill: simple killfile recipe with procmail
15.16 Kill: duplicate messages
15.17 Kill: spam filter with simple recipes
15.18 Kill: (un)subscribe messages
15.19 Time: Once a day cron-like job
15.20 Time: Running a recipe at a given time
15.21 Time: Triggering email and using cron
15.22 Decoding: Uudecode
15.23 Decoding: MIME
15.24 How to send commands in the message's body
15.25 Matching two words on a line, but not one
15.26 How to define personal XX macros?
15.27 How to change subject by body match
15.28 How to change Subject according to some other header
15.29 How to call program with parameters

16.0 Miscellaneous recipes
16.1 Matching valid Message-Id header
16.2 Sending two files in a message
16.3 Excessive quoting of message
16.4 Sending message to pager in chunks
16.5 Playing particular sound when message arrives
16.6 Combining multiple Original-Cc and Original-To headers
16.7 Forwarding sensitive messages in encrypted format

17.0 Procmail and PGP
17.1 Decrypt pgp messages automatically
17.2 Getkeys from keyserver
17.3 Auto grab incoming pgp keys

18.0 Includerc usage
18.1 Using: multiple rc files
18.2 Using: You can call rc file conditionally
18.3 Autoloading an rc file
18.4 Making: naming of the rc file
18.5 Making: Using namespace when saving procmail variables
18.6 Making: Public and private variables in rc file
18.7 The rules of thumb for constructing general purpose rc file
18.8 An includerc skeleton

19.0 Mailing list server
19.1 Mailing list server pointers
19.2 Simple Mailing list server

20.0 Common troubles
20.1 Procmail modes: normal, delivery, and mailfilter.
20.2 Procmail as sendmail Mlocal mail filtering device
20.3 Procmail doesn't pass 8bit characters
20.4 My ISP isn't very interested in installing procmail
20.5 My ISP has systemwide procmailrc; is this a good idea?
20.6 Procmail changes mailbox and directory permissions
20.7 Changing mbox permission during compilation to 660
20.8 The .forward file must be real file
20.9 Using .forward if procmail already is LDA
20.10 Mail should be put in the mailqueue if write fails
20.11 Qmail: how to make it work with procmail
20.12 Qmail: Procmail looks file from /var/spool/mail only
20.13 Qmail: patch to procmail 3.11pre7 to work with Maildirs
20.14 AFS: How to use Procmail when HOME is in AFS cell
20.15 Help, some idiot sent my address to 30 mailing lists
20.16 Help, Procmail beeps and prints to my console
20.17 Help, procmail dumps mail to console
20.18 Help, corrupted From_ line in mailbox
20.19 Directing user's mail to HOME instead of /var/spool/
20.20 NFS mounting /var/mail is a good way to get bad performance
20.21 I can't see the sendmail's response in LOGFILE
20.22 Compiling procmail and choosing locking scheme
20.23 Forwarding lot of mail causes heavy load
20.24 What happens to mail if MDA Procmail fails
20.25 Procmail reads entire 90Mb message into memory
20.26 Help, procmail uses occasionally huge chunk of memory
20.27 Procmail signalled out of memory in my verbose log
20.28 Variables DEFAULT and ORGMAIL
20.29 When DEFAULT cannot be mailed to
20.30 Variable DROPPRIVS
20.31 Variable HOME
20.32 Variable HOST
20.33 Variable LINEBUF
20.34 Variable LOG and LOGFILE
20.35 Variable TRAP
20.36 Variable UMASK
20.37 UMSAK and permissions
20.38 Performance difference between backtick and "|" recipe
20.39 Procmail's temporary file names while writing file out
20.40 Parameter $@
20.41 Procmail variables are null terminated (detecting null string)
20.42 FROM_DAEMON TO and TO_ and case-sensitiveness
20.43 TO_ macro deciphered
20.44 TO_ macro and RFC 822
20.45 FROM_DAEMON deciphered

21.0 Technical matters
21.1 List of exit codes
21.2 List of precedence codes
21.3 Sendmail and -t
21.4 RFC822 Reply-To and formail problem with multiple recipients
21.5 Procmail and IMAP server
21.6 Machine which processes mail
21.7 Compiling procmail and MAILSPOOLHOME

22.0 Smartlist
22.1 MLM RFC
22.2 Other mailing list software
22.3 SmartList code (mailing list implementation with procmail)
22.4 Installation trouble: getparams
22.5 Accepting mail only from users in whitelist(s)

23.0 Additional procmail or MUA software
23.1 Comstat to handle multiple mailboxes
23.2 Elm and pgp support (Mutt)
23.3 MH sites

24.0 Additional procmail software for Emacs
24.1 What is Emacs
24.2 Emacs and procmail mode and Lint
24.3 Emacs and lining up backslashes
24.4 Emacs and browsing mailbox files
24.5 Emacs and live-mode.el
24.6 Emacs and font-lock.el

25.0 Procmail, Emacs and Gnus
25.1 Gnus pointers
25.2 Why use procmail with Gnus
25.3 Setting up gnus for procmail - Basics
25.4 Gnus for procmail - More gnus
25.5 Emacs and Gnus -- Fiddling with spool files
25.6 Gnus and article snippets
25.7 Emacs GNUS - POP - Procmail

26.0 RFC, Request for comments
26.1 RFCs and their jurisdiction (munged Addresses)
26.2 Comments about addresses munging
26.3 RFC and valid email address characters
26.4 RFC and login-name@fdqn
26.5 RFCs and message's signature
26.6 RFC and using MIME in usenet newsgroups
26.7 Some RFC Pointers

27.0 Introduction to E-mail Headers
27.1 To find out more about email (Resources)
27.2 Lecture by Alan Stebbens
27.3 Applied to received messages
27.4 Bcc lecture by Alan Stebbens
27.5 Bcc lecture by Philip Guenther

28.0 Message's headers
28.1 What is correct From address syntax
28.2 What's that X-UIDL header?
28.3 What is that first From_ header?
28.4 Message-Id header
28.5 Received header
28.6 Return-Path
28.7 Errors-To
28.8 X-Subscription-Info
28.9 Reply-To header
28.10 Mail-Copies-To header
28.11 Mail-Followup-To and Reply-To-Personal headers
28.12 Content-Length header and From_ specification
28.13 Moral about CC copies in usenet

29.0 Other interesting code
29.1 Misc email related pointers
29.2 Expire mail pointers
29.3 Usenet News related pointers
29.4 Code: Perl Extract procmail man pages from 3.11pre7.tar.gz
29.5 Code: Sh remove matching lines from file

Table of contents

2.0 UBE in Internet
2.1 Terms used and foreword
2.2 UBE strategies
2.3 UBE and bouncing message back
2.4 UBE and "I don't mind" attitude
2.5 We need a law against UBE

5.0 Dry run testing
5.1 What is dry run testing
5.2 Why the From field is not okay after dry run
5.3 Getting default value of a procmail variable

12.0 Formail usage
12.1 Fetching fields with formail -x
12.2 Always use formail's -rt switch
12.3 Using -rt and rewriting the From address
12.4 Formail -rt and Resent-From header
12.5 Quoting the message
12.6 Without quoting the message
12.7 How to include headers and body to the reply message
12.8 Adding text to the beginning of message
12.9 Adding text to the end of message
12.10 How to truncate headers (save filing space)
12.11 Adding extra headers from file
12.12 Splitting digest
12.13 Mailbox: Splitting to individual files
12.14 Mailbox: Extracting all From addresses from mailbox
12.15 Mailbox: Applying procmail recipe on whole mailbox
12.16 Mailbox: run series of commands for each mail (split mailbox)
12.17 Option -D and cache
12.18 Option -D and message-id in the body
12.19 Reducing formail calls (conditionally adding fields)
12.20 Formail -A -a options
12.21 Formail -e -s options

17.0 Procmail and PGP
17.1 Decrypt pgp messages automatically
17.2 Getkeys from keyserver
17.3 Auto grab incoming pgp keys

19.0 Mailing list server
19.1 Mailing list server pointers
19.2 Simple Mailing list server

23.0 Additional procmail or MUA software
23.1 Comstat to handle multiple mailboxes
23.2 Elm and pgp support (Mutt)
23.3 MH sites

1.0 Document id

1.1 General

.@(#) $Id: pm-tips.txt,v 1.76 1999/11/10 13:55:12 Jari Aalto Exp $
.$Keywords: procmail sendmail formail mail UBE UCE spam filter $
.$URL: http://www.procmail.org/jari/ $
.$Contactid: $
.$FileServer: send mail to Contactid with subject "send help" $
.$UrlLinksLastChecked: 1999-04-30 $

.@(#) This is a procmail tips page: a collection of procmail recipes,
.@(#) instructions, howtos. The document also contains URL pointers to
.@(#) the procmail mailing list and sites that fight against Internet
.@(#) UBE. You will also find many other interesting subjects that
.@(#) discuss about internet email: headers, mime and RFCs. There is
.@(#) also lot of room dedicated to Emacs and Gnus, simply because
.@(#) those are the best tools you find from Unix to deal with your
.@(#) mail and news reading. And I happen to know Emacs quite well.

.@(#) The tips are compiled from the procmail discussion list,
.@(#) from comp.mail.misc and from the author's own experiences with
.@(#) procmail.

This document does not intend to teach you the basics of procmail;
instead you have to be familiar with the procmail man pages
already. You may want to read *Nancy's* and *Era's* procmail faq
pages before this page. Especially Era's link page contains an
excellent collection of useful procmail links and pointers to unix
programs that deal with email (eg. Perl *MHonArc* Email hyperarchiver
at http://www.oac.uci.edu/indiv/ehood/mhonarc.html). If you find
errors or things to improve in this document, please go ahead and
send mail to [jari].

Author's homepage is behind these redirections links. Please keep these
in your bookmark list, not the absolute addresses, because the sutes may
move. These link should point always to the correct location:

http://poboxes.com/jari.aalto/ eg. homepage.html
http://home.eu.org/~jari/

If you want to have automatic notification whenever this page changes,
please visit the link below. To get nicely formatted netmind messages,
see procmail module `pm-janetmind.rc'.

http://minder.netmind.com/

If a mentioned URL is not alive, you may still be able to
successfully find it using the ftp search located at
http://ftpsearch.ntnu.no/

1.2 What is Procmail?

[faq] Procmail is a mail processing utility, which can help you
filter your mail, sort incoming mail according to sender, Subject
line, length of message, keywords in the message, etc, implement an
ftp-by-mail server, and much more. Procmail is also a complete
drop-in replacement for your MDA. (If this doesn't mean anything to
you, you may not want to know.)

Procmail runs under Unix. See Infinite Ink's Mail Filtering and
Robots page for information about related utilities for various other
platforms, and competing Unix programs, too (there aren't that many
of either).

1.3 Abbreviations and thanks

People and documents, abbreviations referred to, tokens used,
are in no particular order.

[stephen] Stephen R. van den Berg, Author of Procmail Last heard
from stephen 1997-08 in procmail mailing list by using address
. Later 1998 due to his regular work activities and
lack of time he nominated Philip Guenther to the head of Procmail
development.

.[aaron] Aaron Schrab
.[alan] Alan K. Stebbens
.[dan] Daniel Smith
.[david] David W. Tamkin
.[ed] Edward J. Sabol
.[elijah] Eli the Bearded
.[hal] Hal Wine
.[jari] Jari Aalto
.[philip] Philip Guenther
.[richard] Richard Kabel
.[sean] Sean B. Straw
.[timothy] Timothy J Luoma
.[walter] Walter Dnes

.[faq] Procmail FAQ j1era+pr@iki.fi
.[manual] Quote from some procmail manual page

o PM-L, Procmail mailing list
o FAQ-L, Faq Maintainers mailing list
http://www.landfield.com/faq-maintainers/faq-server/
http://lists.consensus.com/scripts/lyris.pl?visit=faq-maintainers
http://www.qucis.queensu.ca/FAQs/FAQaid/
o DING-L, Emacs Gnus mail/newsreader mailing list (ding).
http://www.gnus.org/
o <> Text has been rephrased or something was added which
does not exist in original message.

I also thank following people

o Era Eriksson proof read the v1.12 and sent corrections.
o Karl E. Vogel
sent numerous new anti-spam links to be added to the document.
o John Gianni send some nice recipes: one is now
in the procmail module list and the other ideas I have added to
this tips file.
o Tim Potter had a spare moment with v1.27 and
sent lot of spelling corrections. Thank you.
o took 1.48 and sent a huge
55k patch to correct many English language typos. Thank you
very much Guido.
o 1998-10-28 Richard Kabel sent massive patch
to correct language and provided excellent improvement comments.
o 1999-01-08 Steven Alexander thought that
a small perl script would help me to fix spelling mistakes more
easily. The script has been much better correction program that
simple patches. Thank you.
o 1999-06-16 Mark Seiden Did a enermous work to
proofread the v1.74. He sent a massive 105k with many editorial
corrections. My wholeheart thanks to you Mark.

1.4 Version information

Here is version and file size log of the text file, which gives you
some estimate how often you should update your copy.

v1.01 1997-09-13 46 (k)
v1.05 1997-09-14 53
v1.5 1997-09-16 76
v1.6 1997-09-18 94
v1.8 1997-10-01 127
v1.9 1997-10-11 142
v1.10 1997-10-13 181 archive file 1995-10's tips included
v1.13 1997-11-08 218 Era's correction suggestions.
v1.14 1997-11-25 260
v1.17 1997-12-09 343 up till archive 1996-07 now included
v1.24 1997-12-30 415 up till 1996-12 is now included
v1.29 1998-01-30 429 "regexp" section rewrite.
v1.31 1998-03-10 469 Better ordering: ORing rules discussed
v1.32 1998-03-23 471 All recipes checked (by eye)
v1.34 1998-04-02 488 ORing and supreme scoring added
v1.36 1998-04-03 493 Includerc rewritten, plus addressing
v1.41 1998-06-17 510 How to disable recipe quickly with
v1.44 1998-06-19 516 Detecting mailing lists with pm-jalist.rc
v1.45 1998-06-23 521 All recipes checked by eye. Many fixes.
v1.46 1998-06-24 526 Added live urls to procmail archive
v1.49 1998-08-10 529 Guido.Van.Hoeck's 55k patch applied
v1.51 1998-08-18 541 Small changes. MIME notes
v1.52 1998-08-24 553 Flag c forking study, procmail wish list
v1.53 1998-08-24 554 Procmail doesn't pass 8bit characters
v1.55 1998-08-29 565 Fetching fields with formail -x
v1.57 1998-10-06 575 PLUS addr. Convert HTML body to text
v1.58 1998-10-12 583 SmartList and other MLM software discussed
v1.60 1998-10-21 591 UMASK, .forward if procmail already is LDA
v1.63 1998-10-30 595 Richard's english correction patch
v1.64 1998-11-26 602 More Richard's comments integrated
v1.66 1998-12-14 578 Philip took care of bugs/patches listing
v1.67 1998-01-07 579 Eli's procmail recipes in module section
v1.68 1998-01-29 587 Added "Lua" language pointer
v1.69 1999-02-23 590 RFC and using MIME in usenet postings
v1.70 1999-02-26 592 procmail's Y2K compliance
v1.71 1999-03-29 597 Ricochet -- Perl script to fight UBE
v1.72 1999-04-21 597 Links corrected
v1.74 1999-04-26 599 document moved to www.procmail.org
v2.0 1999-10-01 602k Mark Seiden's patch applied. Now under CVS.

1.5 Document layout and maintenance

This document is maintained in plain text format with Emacs and my
text formatting package *tinytf.el* (automatic TOC and indentation
control). Funny marks or indentation are in the text
version so that the Perl text-to-html
filter `t2html.pl' can be used. See more about this at:
http://poboxes.com/jari.aalto/t2html.html

Text version of this file was converted into HTML with command:

% perl5.004_04 t2html.pl \
--html-frame \
--title "Procmail tips page" \
--author "Jari Aalto" \
--email jari.aalto@poboxes.com \
--meta-keywords "procmail, sendmail, mail, filter, faq, ube" \
--meta-description "Procmail tips page" \
--base http://www.procmail.org/jari \
--document http://www.procmail.org/jari \
--url http://www.procmail.org/jari \
--html-body LANG=en \
--Out \
pm-tips.txt

Please also familiarise yourself with unix what(1) and GNU RCS
ident(1), if you have those commands in your system. It is
important that you mark interesting text to these tools so that
someone can get an overview of your supplied files

% what FILES - Print @( # ) tags
% ident FILES - Print $ $ keywords

Sending improvements

Because I'm not English speaking, I regret the bad language I may have
used in this document. If you have any time, 5-10 minutes to find some
spelling mistake or misuse of the English verbs, please go ahead and
send me a patch to correct the wording. The preferred way to send
corrections to this document is as diff(1) output. Here's how to make
corrections send them to me:

The diff option -u is only available in GNU diff, please try to
send the -u diff if possible. If you don't have -u option, use -c
switch.

% cp pm-tips.txt pm-tips.txt.orig

... load the pm-tips.txt to your text editor
... edit the file and save
... Print the version number first

% what pm-tips.txt > pm-tips.txt.diff # see man what(1)
% diff -u -bw pm-tips.txt.orig pm-tips.txt >> pm-tips.txt.diff

...Send content of pm-tips.txt.diff to document maintainer.

1.6 About presented recipes

The recipes presented here are collected from the net and procmail
archives. I have tried my best to keep the recipes as original as
possible, but I have generalised the examples when necessary. If
some recipe doesn't work as announced, please a) send note to
[jari] b) send email to procmail mailing list and ask how to
correct it. I will watch the procmail list and I'll replace any
faulty recipe with correct one.

Sometimes I have taken the liberty to use a simple dot(.) in
regular expressions, where the right, pedantic way would have
been to use an escaped dot. If you want to be very strict, you
should use the escaped dot where applicable.

[free hand version] [pedantic version]
:0 :0
* match.this.site * match\.this\.site

Procmail also accepts assignments without quotes, like this:

var = value
num = 1
dir = /var/mail

But I have adopted a style, where literal strings are assigned with
double quotes:

var = "value"

because the procmail code checker then won't warn you about missing
dollar-sign, which you might have very well forgotten. Emacs fon-lock,
a syntax highlighting package also displays double quoted string
in color.

# If you do this...

var = value

# then it is in fact not clear what was intended:

var = "value" # Did you mean: literal assignment?
var = $value # Did you mean: variable assignment?

Recipe flags are also _not_ stuck together, because for me the
visual distinction of `:0' and `flags' is a valuable one.

[Erm, all stuck] [I like this better]
:0ABDc: :0 A BDc:

1.7 Variables used in recipes

These are part of the procmail module *pm-javar.rc* and are used in
recipes.

# Pure newline; typical usage: LOG = "$NL message $NL"

NL = "
"

Refer to "improving Space-Tab syndrome" section for more details

WSPC = " " # whitespace: space + tab

SPC = "[$WSPC]" # Regexp: space + tab
SPCL = "($SPC|$)" # whitespace + linefeed: spc/tab/nl
NSPC = "[^$WSPC]" # negation

s = $SPC # shortname: like perl -- \s
d = "[0-9]" # A digit -- Perl \d
w = "[0-9a-z_A-Z]" # A word -- Perl \w
W = "[^0-9a-z_A-Z]" # A word -- Perl \W
a = "[a-zA-Z]" # A word, only alphabetic chars

Writing recipes is now a little easier and may look more clear.

*$ Header:$s+$d+$s+$d # Matches "Header: 11 12"

_SUPREME_ = 9876543210, is the highest score value that causes
procmail to bail out. [david] Actually the maximum is 2147483647,
but 9876543210 is easier to remember/type and will function just as
well.

_PMSRC_ = Procmail includerc code directory, where *rc files
reside. Anywhere you want it to be: usually $HOME/pm or
$HOME/.procmail. Here you keep the procmail files, logfiles and
includerc scripts. You can also use the synonym _PMDIR_.

_SPOOL_ = Directory where your procmail delivers the categorized
messages. Like mailing lists:

list.procmail, list.lyx-users, list.emacs, list.elm

and work mail:

work.announcements, work.lab, work.doc, work.customer

and your private message:

mail.usenet, mail.private, mail.default, mail.perl

and unimportant messages

junk.daemon, junk.cron, junk.ube

If you read the procmail-delivered files directly, this directory
is usually $HOME/Mail or $HOME/mail. If you use some other software
that reads these files as mail spool files (like Emacs Gnus), then
this directory is typically ~/Mail/spool/ or similar.

_MY_XLOOP_ = Used to prevent resending messages that have already
been handled. Typically `$LOGNAME@$HOST', but this can be any user
chosen string. Make it it unique to your address. In this document
the definition is:

MY_XLOOP = "X-Loop: $LOGNAME@$HOST"

_SENDMAIL_ = Program to deliver composed mail. Usually standard
Unix `sendmail', but it must have some switches with it. See man
page for more. We use following definition in scripts:

SENDMAIL = "sendmail -oi -t"

_NICE_ = In a Unix environment you can lower the scheduling priority
wth nice(1). If you are conscious of how many external processes you
launch for each piece of mail it would be nice to lower the
priority of such processes. You may see in this tips file that
external processes are called with `NICE' enabled:

:0 w # same as nice -10 script.pls
| $NICE script.pl

_IS_ functions; eg. IS_EXIST is defined as "test -e" and so on.
The definition of _IS_ functions are system-dependent.
E.g. On Irix the "-e" option is not recognized and
the nearest equivalent is "test -r". All _IS_ functions
are defined in the `pm-javar.rc' module.

1.8 About "useless use of cat award"

Randal Schwartz, a well-known Perl programmer and Perl book writer,
started giving emmy rewards for the "useless use of cat command"
whenever someone wrote examples without token "<". Like this:

% cat file.name.this | wc -l

Instead he insisted that the call should have been written like this,
which saves the pipe. (Never mind that `wc' can read the file
directly; this is an example.)

% wc -l < file.name.this

I stick my opinion in this soup and you're free to disagree. When
you see the shell commands used in this document, they are written
so that they can be read from left to right: The "<" is in my
opinion difficult to understand. As an example, I think that:

% cmd1 < file1 | cmd2 > file2

is less clear than my preferred way of writing such commands:

% cat file1 | cmd1 | cmd2 > file2

And now to the purist side: Is saving one pipe process so important?
Let me see, I use a 2Meg file in this test:

% time sh -c "cat some-file-name-is-here | time wc -l"
0.29u 0.11s 0:00.47 85.1%

% time sh -c "wc -l < some-file-name-is-here"
0.27u 0.05s 0:00.39 82.0%

There is not much difference, and this 2Meg file is not typical at
all. The files typically used are many times smaller. The nitpicking is
therefore pointless. Another reason why I use "left to right
pipe writing": when you recall the command in csh, you can edit the
last command's arguments easily. If you used the "<" token, tapping
keyboard is _much_ more tedious (try changing wc command's option
above). Oh yeah, you can write like this to get the command to the
right, but that's even more obscure.

% < some-file-name-is-here wc -l

Dallman Ross also mentioned that csh users can
replace any word in the previous command by use of caret(^) editing
commands, like this:

% cat some-file-name-is-here | time wc -l
% ^some-file-name-is-here^new-file^
--> cat new-file | time wc -l

Ahem, so there, I got it off my chest...

2.0 UBE in Internet

2.1 Terms used and foreword

[Part of this has been excerpted from the Email Abuse Faq]

._UBE_ = Unsolicited Bulk Email
._UCE_ = (subset of UBE) Unsolicited Commercial Email

_Spam_ = Spam describes a particular kind of Usenet posting (and
canned spiced ham), but is now often used to describe many kinds of
inappropriate activities, including some email-related events. It
is technically incorrect to use "spam" to describe email abuse,
although attempting to correct the practice would amount to tilting
at windmills.

_Spam_ = definition by Erik Beckjord. "Some people decide that Spam
is anything you decide you want to ban if you can't handle the
intellectual load on a list." Remember, not to be confused with
real spam, which is unwanted bulk mail.

People are nowadays seeking a cure which will stop
or handle UBE. That can be easily done with procmail (under your
control) and with sendmail (by your sysadm). In order to select the
right strategy against UBE messages, you should read this section
and then decide how you will be using your procmail to deal with it.

2.2 UBE strategies

[Excerpted from the Email Abuse Faq]

4g. I asked to be "removed" - guess what? I got another U*E

Not surprisingly, many UBE outfits treat a "remove" request as
evidence that the address is "live"; a "remove" request to some
bulk emailers will actually guarantee that they will send more to
you. For many others, the remove procedure does not work, either by
chance or design. At this point perhaps you're starting to get a
feel for the type of people with whom you are dealing.

Also, getting removed doesn't keep you from being added the next
time they mine for addresses, nor will it get you off other copies
of the list that have been sold or traded to others. In summary,
there is no evidence of "remove" requests being an effective way to
stop UBE.

4h. I asked to be "removed" - guess what? The message bounced

Probably the remove procedure was false. Any remove procedure that
tells you to send remove requests to AOL, CompuServe, Prodigy,
Hotmail, or Juno is certainly false. The bulk emailers are an
unpopular lot; they forge headers, inject messages into open SMTP
ports, use temporary accounts, and pull other stunts to avoid the
tirade of complaints that follow every mailing.

2.3 UBE and bouncing message back

Has anyone found that bouncing spam does any good at all?

_Note:_ There are several program packages out there that
can with a high degree of success (but not 100%) trace back a
spam even if some headers are faked. This will not help you against
spam houses (which don't care) but will speed you telling
the sysadmins of an open relay. Such tools need human interaction for
proper working. See pointers to them in this document later.

Examine the messages by hand first and feed them to automatic
complain script. See pointers in this document later.

[sean] I had a whole policy message written up that would be sent
out to spammers. Nothing but a waste of my resources. Most return
paths are either completely bogus, or end up bouncing pretty damn
soon after the spam, which just brings you more junk to deal with.

Instead, I choose to send messages occasionally to administrators
and upline providers of domains which spew. "Agreement by action"
is one of the legal standards I like to use (for "should you
continue to send mail to me, that constitutes acceptance of the
terms herein").

InterNIC recently 1997-07 removed the root files for .com, .org,
and .net (I think) from access at their ftp server. Too many
spammers were using them for the purpose of generating mailing
lists. Access to the files now requires an assigned FTP account
from InterNIC. When I get a domain-style spam, I immediately do a
whois to get DNS info on the domain, then grep the root files to
obtain a list of domains serviced by the same DNS. If they appear
spammy (as spam domains tend to), I add these to a list of domains
to filter (egrep) in my primary domain-based ruleset. Works for
me, though the list is getting big.

[Kimmo Jaskari ] Another good reason is
that all those bounces, which get ignored by the spammer/recipient
anyway, still take up needless bandwith on the net. The spam is bad
enough for that, bouncing it back with some more stuff added is just
plain silly. You become part of the problem rather than the solution.
If the bounce even gets to the spammer, the spammer drops it on the
floor unseen.

[1998-11-03 PM-L Mark Shaw ] Jari:
"Autoresponder is bad idea. You need more better heuristics than
what procmail can do. The UBE messages really need human
inspection before you send them out, otherwise you may have to
apologise from lot of people eg if the complaint was mistakenly
sent off to some mailing list or wrong address." Mark: Having
originally set up my anti-spam recipes to be autoresponders, I
absolutely agree with this. I recall one morning when my strongly-
worded no-spam message went out to *everyone* who sent me email for
several hours..... *** shudder ***

2.4 UBE and "I don't mind" attitude

...whenever you see a spam you don't want, hit the delete key and
move on. Grow up and get a life, folks. The spams just don't
bother me. Why the hell does everyone have to go up in arms
everytime someone sends a spam? Spams are harmless! Spams even
sometime are interesting and/or useful!

[Responses from thread in procmail mailing list 1995-10 to
"FREE 1 yr. Magazine" spam.]

[Soren Dayton ]

The simplest reason against UBE is that it is rude. It costs some
people money to get email on some commercial services. This is
fundamentally different than junk snail mail for this reason and
too much spam can prevent people from getting mail (mailboxes can
fill up). So it is both an intrusion into my life _and_ it can
conceivably end in me either loosing money or loosing mail (which is
far more important). It is a burden on the receiver _far_ beyond
just hitting the delete key.

[Mark Seiden ]

people who are able to monitor the incoming machines of one of the
larger online services (like me) can see a sizeable increase in
system load average and volume directly resulting from spams. this
competition for fixed resources inevitably translates to reduced
service for "first class" mail.

It is impossible to engineer a mail system that can cope with an
unlimited amount of abuse. this is in addition to the difficulties
of doing so on a fixed price economic model, and the difficulties
of keeping up with the successful rapid expansion of the population
to be served.

Even if you, an individual, aren't charged anything per piece of
mail, there are costs borne by your service provider per piece of
mail, and these are *somehow* passed on to you. (They've calculated
an average across their entire user population to come up with a
"monthly cost of Internet mail".)

Spamsters and bulk mailers are not at all concerned about
efficiency. as proof of that, many of them are not even courteous
enough to supply a proper return address, so they can prune their
lists of undeliverable mail. all they care about is getting their
message across without their paying anything whatsoever for that
service.

Watch how this will inevitably translate into increased costs for
you, the consumer, unless we change the mechanisms by which bulk
mail is delivered as well as putting an appropriate economic model
in place.

[Steve Simmons ]

If you tolerate spamming, it will only get worse. Spamming has been
stopped again and again. Almost without exception, the spammers
have been tracked down and, via one means or another, have been
convinced to stop spamming.

Spams are harmless? I've already seen the 'Magazine Sub' message
10 or 12 times. I have a low bandwidth line. If I continue to
tolerate spamming, I will pay a very real penalty in performance as
tens, then thousands of spammers do it. Not to mention the
personal time involved in taking care of the crap.

Don't think that the time involved is significant? Just wait. My
wife and I are fairly generous with our time and money. As a
result, we were getting an average of five telephone calls *per
night* asking for money for various causes. A year ago, I adopted
a new policy -- I will not under any circumstances give money to a
caller, and will only consider it upon written solicitation. I
ask them to put me on their `do not call list'. If they do
*anything else* to continue the conversation, I hang up on them.

My wife opposed this, and we agreed to disagree -- if they ask for
her, they get her. If they ask for me, they get my speech. After
a year, she is getting 2-3 calls per night and I'm getting one or
two a week.

My point here is that individual action *does* get re-action from
the mailers. For them, I copy their internet providers on my
complaints and call their Better Business Bureau. It works.

If one does this politely and consistently, 98% of the spammers
will stop. The remaining 2% will discover that they're in a
different world from direct mail or telephone solicitation. Their
mailboxes will be overloaded with complaints (when it takes a
single keystroke to invoke your complain macro, you're very likely
to complain). Then their suppliers mailboxes will be overloaded
with complaints. The free magazine folks, who've been hiding
behind false ids and forging mail, will find that they're on the
wrong side of the law. I'm considering contacting their local
legal officials and urging them to investigate, because it sure
looks like fraud to me (read `Consumer Reports' for a similar case
by surface mail). Should a few more like this come in, I *will*
contact their legal authorities. We have their fax number; it's
all we need to find them.

[Carl Payne ]

Um, I don't know about you or anyone else here, but this cutesy,
"it's-okay-by-me" spam has been circulated under half a dozen
different user names and "domains" on as many mailing lists. It's
obvious to me the sender is trying to make people pissed off--how
can he possibly think someone will buy that crap, and why does he
think it's okay to send 19 and 20K files over a billion groups?

AFAIC, it has to stop. Now. I'm tired of the spam, I'm tired of
the "Who cares" attitude about spam, I'm tired of ISPs letting
people spam, I'm tired of the jetwash of spam, and I'm tired of the
bleedinghearts that say, "Golly, just ignore it, and it'll go
away."

I've got news for you all: when this method of spamming becomes the
preferred method of "marketing" on the internet, and people like us
are the bad guys because we're not allowing such litter to fly
across the fiber, you will care. You will say something, most
probably, "Why didn't we do something about this sooner?"

The guy in the next cube from you, who's paying a per-message
charge through his ISP, is probably going, "Dammit, over three
dollars this month on mail I've itemized as being spam." While
that doesn't seem like a lot, I revert to my earlier statement: if
this becomes the preferred method, his bill (and yours) will go up,
and everyone will wonder why it's too out of control to do anything
about.

Spam has the letters *m-a-s* in it, which en Espanol, means "more."
I say no. Not only no, but hell no. And, I refuse to be told that
my thinking is out of line just because I don't want my mailbox
flooded. Do something now. Do anything now. But, don't be quiet
and listen to anything that sounds like an endorsement of litter

[Wolfgang Weisselberg ]

Worse is that it costs a spammer very little to spam, say, 2
million addresses with 5KB:

o 5 hours unattended time online
o phone costs
o a 'free x hours'-CD or a provider looking the othher way i.e.
something between $0 and $500 (an expensive provider)

It costs all recipients:

o on an average of 5 seconds per UCE to decide that, indeed, it
is one: 115.7 *DAYS* (2777.8 hours) of mailchecking (at $7.5/h
that is just $20833 --- excluding all taxes and so on!)
o 379.5 hours (15.8 days) download time (multiply with your local
phone costs and remember that in most places even in-city calls
cost by the minute)
o the same time as online time (multiply by your provider costs)
o indirect costs (more HDs for the provider (9.5 GB), faster
connections for all the spams, more transmission costs (9.5
GB), faster machines, ...

I can send you the complete calculation if you like :-)

Now, if UCE becomes more common ... how many businesses are
connected to the Internet? Say that every business spamms once
every 10 years, and that they are well distributed over the time.

Number_of_businesses / 3650 = UCE's iniciated per day
UCE's iniciated per day * 2_000_000 (or more)
/ number of email addresses
= UCEs in your mailbox

Guess we are going to need T1's to just get all our mail. And a few
100 secretaries as well. Wave good-bye to usable email.

2.5 We need a law against UBE

Ray Everett-Church , Attorney/Online Consultant
Co-Founder & Congressional Liaison Coalition Against Unsolicited
Commercial Email; article 1997-12 in remailer politics mailing
list

In developing what eventually became the Smith Bill, CAUCE
discussed this rather extensively among our drafting committee. The
bill gives a cause of action against the advertiser, not any of the
pathways taken between you and them. This is consistent with the
interpretation of the fax law (and many other laws for that matter)
wherein the advertiser -- not the advertiser's agent -- is
responsible for the act committed.

As for the single UCE versus bulk issue, the general consensus has
been that while a single piece of spam does not do much damage, it
is fundamentally no less a cost shift than 10 identical messages,
or 100, or 1000, or a million. The only difference is that the
costs being shifted are greater and greater. We discussed many cut
off points... would 50 spams be acceptable? 25? 10? One really well
crafted, hand written, heartfelt and personalized spam be
permissible? And in the end we felt like we were discussion angels
on the heads of pins.

While virtually nobody's system will crash because of one piece of
spam (although George Nemeyer had trouble with three or four pieces
as I recall), what is the ultimate difference if you only get one
piece from each of 15 different advertisers a day? If one spam is
ok, but two are bad, what is the interval... a day, a week?
Enforcement depends on knowing when the threshold is crossed.

So here's a scenario: you receive three spams from what is,
unbeknownst to you, the same person (one advertising weightloss
pills from WeightLoss Associates at PO Box 1, one for an MLM from
MLM Company at PO Box 2, and Bee Pollen from Pollen Partnership at
PO Box 3). Each were individually crafted and appeared to be mailed
only to you.

Under the scenario above, if the law permits one spam, will you
sue?

Would you risk suing one or all of them, gambling that they sent
the spam to anyone other than you (or whatever the threshold is...
10, 25, 50)? Would you risk suing one or all of them on the chance
that they were somehow related? What if there was a chance that
you'd find out that the three companies were really different? What
if you did sue and found that they were owned by the same person,
but were legally organized separate entities and were therefore
each entitled to one spam a piece?

In short... if one spam is permitted, it could make enforcement
incredibly cumbersome, difficult and unlikely, and would present
spammers with many reasons to violate the law knowing the odds of a
suit and successful enforcement are greatly reduced. While bulk
spam is really bad on many levels, whether it's parsed out in very
small volumes makes little or no difference to the ultimate
recipients as far as the diminished utility, cost, and annoyance.

We need a clear, bright line. And the Smith Bill is that.

3.0 Anti-UBE pointers

3.1 NoCEM, CAUCE and others

"NoCEM"
http://www.cm.org/

"Dougal's NoCeM-E"
http://advicom.net/~dougal/antispam/
... Dougal is sysadm for an ISP. His page has wealth of information
about Anti-SPAM Tools. You also find his mailing list for NoCeM-E.

"The Coalition Against Unsolicited Commercial Email (CAUCE)"
http://www.cauce.org/faq.html
...The Problem: Unsolicited commercial email, more commonly known as
"spam", is a growing problem on the Internet. If you've used the
Internet for any length of time, you've probably received
solicitations via email to purchase products or services.

A Solution: A group of Internet users who are fed up with spam have
formed a coalition whose purpose is to amend 47 USC 227, the
section of U.S. law that bans "junk faxing", so that it will cover
electronic mail as well.

"Teergrubing against Spam"
http://www.iks-jena.de/mitarb/lutz/usenet/teergrube.en.html
...`Teergrubing' It's German and means Tar-Pit. Once you have been
stuck you can't get out. ...slow down internet connections in order
to stop UBE abuse. Several hundred teergrubes are able to block
spamming worldwide without blocking any e-mail. How do I start: If
you are the admin of a MX host, install a teergrube.

"Obtuse smtpd for UNIX"
http://www.obtuse.com/smtpd.html
Main (configurable) features:

o deny unauthorized relay (no more relay rape!)
o permit selective relay exceptions (eg. UUCP downstream)
o regex() filtering [block those spamming dialins!]
o deny access for no MX, no PTR, etc.
o defeat % hack
o support MAPS, ORBS, DUL, IMRSS, etc RBLs plus your local RBL
o support exception list for domains for which you will accept mail
o support selective tarpit'ing on refused connections
o individually configurable rejection messages
o precedence and override ordering
o informative log summary scripts

"Lot of good articles about spam"
http://www.sun.com/sunworldonline/swol-12-1997/swol-12-spam.html

"(anti-spam Law) US Representative Chris Smith's statement on junk
e-mail"
http://www.sun.com/sunworldonline/swol-08-1997/swol-08-junkemail.html
...considerable variation in the approaches at the federal level,
and state legislation varies widely as well. Professor David Sorkin
of John Marshall Law School, who summarized and provided links to
the major spam-related lawsuits noted above, also provides status
summaries and links to state and federal legislation

"Select email court cases -- Lots of them"
http://www.jmls.edu/cyber/cases/spam.html
America Online, Inc. v. Cyber Promotions, Inc.,
Compuserve Inc. v. Cyber Promotions, Inc., etc.

"Anti-Spam Directory of Information and Resources"
http://www.ao.net/waytosuccess/nospam.html
"Forum for Responsible and Ethical E-mail (FREE)"
http://www.ybecker.net/

"Ethical Marketing Using FREE Resources"
http://www.ao.net/waytosuccess/index.html

3.2 General Filtering pages (more than procmail)

"Nancy McGough - Mail Filtering FAQ"
http://ssil.uoregon.edu/~trenton/autopage/page7547.html
http://www.ii.com/internet/faqs/launchers/mail/filtering-faq/

"Information Filtering Resources"
http://www.ee.umd.edu/medlab/filter/ Doug Oard
...This page lists all known internet-accessible information
filtering resources.

3.3 Junk email and spam

"Spam FAQ"
ftp://rtfm.mit.edu/pub/usenet/alt.spam/
http://www.cs.ruu.nl/wais/html/na-dir/net-abuse-faq/spam-faq.html

"The email abuse FAQ"
http://members.aol.com/emailfaq/emailfaq.html
What is UBE, UCE, EMP, MMF, MLM, Spam, it is all explained here.

"Get that spammer -- A VERY GOOD LINK"
http://kryten.eng.monash.edu.au/gspam.html
...All about Spam; traceroute, netabuse etc. Full of links and docs

"Whois"
http://www.networksolutions.com/cgi-bin/whois/whois/

"Advertising on Usenet: How To Do It, How Not To Do It"
ftp://rtfm.mit.edu/pub/usenet/advertising/

"Dealing with Junk Email"
http://www.mcs.com/~jcr/junkemaildeal.html
...What you should do (and not do) when you have been victimized by
a junk emailer. This document teaches you how to read headers in
order to trace the origin of junk email, and includes detailed
examples to show you how it is done. Headers are designed for
computers to read, not people, so they can be a little hard to
follow. Therefore, I hereby grant permission to print or
electronically save a copy of this page on your local machine for
your personal use while tracing junk email. Please check back for
updates and corrections, though.

o What Not To Do: Stuff that doesn't work
o What to do: effective techniques, including how to trace junk
email back to its source
o Stay Calm (take a deep breath...)
o Stay Mad (don't get discouraged)
o How to identify the sender and who gives them Internet access
o Who to complain to, abuse addresses, online services
o What to say and how to say it, effective complaining

"How to fight back."
http://www.oeonline.com/~edog/spamstop.html

. Look at the header of the advertising message. Find the
"Message-ID" line. (You might have to tell your e-mail program to
display this.)
. The words after the @ sign are the sender's real--not
faked--Internet Service Provider, or ISP. (Spammers often try to
disguise their address, but the Message-ID is a good clue.)
. Write a complaint to the postmaster of that ISP, similar to the
one below. (If the ISP is junkmail.com, then let
postmaster@junkmail.com hear from you.)

"Practical Tools to Boycott Spam"
http://spam.abuse.net/spam/
...We have been actively engaged in fighting spam for years. Recent
events, including pending court battles, prompt us to present this
page to the public. Fight spam to keep the Internet useful for
everyone.

o Filtering mail to your personal account
o Blocking spam email for an entire site
o Blocking Usenet spam for an entire site
o Blocking IP connectivity from spam sites
o Other tools and techniques for limiting spam
o Sample Acceptable Use Policy statements for ISPs

"Spam -- stop that!"
http://com.primenet.com/spamking/buyerbeware.html

"The Campaign to stop junk email web site"
http://www.mcs.com/~jcr/junkmail.html
...we will attempt to teach victims and potential victims (that's
everyone with an email address) the most effective methods of
prevention and retribution.

"news.admin.net-abuse.* Homepage"
Timothy M. Skirvin
http://www.ews.uiuc.edu/~tskirvin/home/nana/

"The automated spamhandler beta information heap."
http://www.halcyon.com/natew/

"Preventing relaying in Sendmail"
...This package adds two independent features to sendmail,
access control and relay control. They will be described here
simultaneously, but you can elect to include support for only one
of them (either one) on your mail server. Access control lets you
deny access to the server based on the senders envelope address or
his IP address. Relay control lets you decide who gets to relay
email through your server.
ftp://ftp.xyzzy.no/sendmail/access.tar.Z

"Anti-Spam Provisions in Sendmail 8.8"
http://www.sendmail.org/antispam.html
http://maps.vix.com/tsi/
http://www.informatik.uni-kiel.de/%7Eca/email/check.html#check_rcpt

o Preventing relaying through your SMTP port
o Refuse mail from selected hosts
o Restrict mail acceptance from certain users to avoid mailbombing

[1998-06-15 PM-L walter] Somebody's starting to exploit a hole in
sendmail 8.8, where giving a HELO longer than 1024 bytes causes
buffer overflow, and all following "Received:" headers are lost. If
it's done off a relay, we have no clue who sent it. There may be a
more elegant solution, but here's a quick-n-dirty procmail filter
for this stunt...

"Blocking Email"
http://www.nepean.uws.edu.au/users/david/pe/blockmail.html

o Do you or your users, receive "junk email" (aka., "spam")
o Do you have Sendmail R8.8.5 running at your site?
o Would you like to block known "junk email" senders' addresses?

Now you can - and there's no need to patch any source code, either.
Take advantage of Sendmail's check_mail rule, to see if the
sender's address is a member of a nominated "class" - drawn from
the contents of the named file. Additional information and links:

o Prospective Addresses/Domains to Block
o Limiting Unsolicited Commercial Email
o EFF "Net Abuse and Spamming" Archive
o [U.S.] Court Lets AOL Block Email
o Anti-Spam HOWTO
o Net Abuse FAQ
o Figuring out Fake Email & Posts
o Fight Unwanted Email
o Unsolicited Junk Email - Bad for Business
o Fight Unsolicited Email and Mailing
o Yahoo's Junk Email Resources
o jmfilter
o Complaints Addresses at U.S. ISPs
o news.admin.net-abuse.* Homepage
o Processing Mail With ProcMail
o Panix's rc.shared ProcMail Configuration
o ProcMail Workshop
o Email Self Defence
o The SPAM-L mailing list

"US Federal Trade Commission"
http://www.ftc.gov/
...staff publicized the Commission's UCE mailbox, "uce@ftc.gov,"
and invited consumers to forward their UCE to it. spam complaints

"Spam Spade Web based tracking tool"
http://www.blighty.com/
...Figuring out forged headers and verifying IP addresses and
whois information.

"Misc"
http://www.junkbusters.com/
http://www.well.com/~jbremson/spam
http://www.wolfenet.com/~jhardin/procmail-security.html

3.4 Comprehensive list of spammers

"Against Spam -- The garbage collecting."
http://www.spam-archive.org/
To support this archive please forward email spam to
. Everybody is invited to bounce Mail-Spam
he/she has got to this list. This is a mailing list to distribute
actual spam-eMail. All incoming mail will be checked by subject and
from/sender-address wether it has already been distributed or not.
No discussions in this list. To discuss about this list please
subscribe to .

To subscribe to _blacklist-update_ mailing list
TO:
BODY: subscribe blacklist-update you@somewhere.com
Mail to discuss about blacklist if
your name is on it. (maintained by Axel Zinser )
Get the updated blacklist from
ftp://ftp.spam-archive.org/spam/blacklist/

3.5 Misc pointers

Is there a way to block local users from spamming other sites?
Maybe somehow force sentmail to read a rc file that would maybe
then grab the from field and see if the user exists on the system
or not. Or run it through some sort of filters.

[philip] You can and should do this purely in sendmail. I ended up
crafting a check_from ruleset that verifies that the envelope
sender address is either a) not local; b) a local user; or c) a
local alias. At the time I did this mainly to force people to
configure their Eudora clients so they didn't say "Return Address:
yourname@gac.edu" but it also covers the outgoing bogus source
address spam case. For those interested in this kinda thing I've
(just) put it up for FTP:

ftp://ftp.gac.edu/pub/guenther/

"IBM's Secure Mailer -- open source"
http://www.postfix.org/

[1998-12-15 PM-L Matthew McGehrin ] The
official project is known as 'IBM's Secure Mailer'. The
unofficial codename was Vmailer, but they had to rename that, to
Postfix to agree with the lawyers. I should know, I have been
alpha testing this mailer for the past year, and it so blazing
fast, its amazing. It's faster and simplier to use than sendmail,
and also faster and more secure than qmail. It works fine with
procmail. (look in my headers). set
"mailbox_command=/usr/bin/procmail" in /etc/postfix/main.cf

[1998-12-15 PM-L Liviu Daia ] it has
explicit hooks for both procmail and RBL. In fact it's incredibly
easy to setup, I got it compiled and configured (with an actually
usable configuration) in about 15 minutes after downloading it.
Adding masquerading and a virtual domain took another 2 minutes.
:-) You should really give it a try, it's faster than QMail and
_much_ faster than sendmail. So far, I'm quite impressed.

"Qmail"
http://pobox.com/~djb/qmail.html
http://www.qmail.org/

"Sendmail"
http://www.sendmail.org/

"Fetchmail -- old pop3 replacement"
ftp://ftp.ccil.org/pub/esr/
http://www.ccil.org/~esr/
http://www.tuxedo.org/~esr/fetchmail/

"Maildrop filter utility"
http://www.geocities.com/SiliconValley/Peaks/5799/maildrop.README.html
...Alternative to procmail

"Lua"
http://www.tecgraf.puc-rio.br/lua/
[possible replacement for procmail language] ... *Lua* is a
programming language originally designed for extending
applications, but also frequently used as a general-purpose,
stand-alone language. Lua combines simple procedural syntax
(similar to Pascal) with powerful data description constructs based
on associative arrays and extensible semantics. Lua is dynamically
typed, interpreted from bytecodes, and has automatic memory
management with garbage collection, making it ideal for
configuration, scripting, and rapid prototyping.

3.6 Questionable UBE stop services

"IEMMC: Internet E-Mail Marketing Council Formed 1997-03"

The IEMMC was formed to provide an industry wide trade association
for the purpose of promoting responsible e-mail marketing, and to
establish an industry standard code of procedures and ethics which
will internally regulate and govern the commercial e-mail marketing
industry....Under this system, all e-mail of a commercial,
unsolicited nature must pass through a universal filtration system
which will block the sending of any and all commercial e-mail to the
address on the list. Bulk e-mailers will be required to join the
organization

Others have commented that:

...IEMMC is a joke. you are probably not doing yourself any favors

...Don't take that IEMMC seriously! Many people registered with
them and got as many or even more spam as before. After all,
Cyberpromo (the operator of IEMMC) knows that the registered
addresses will be valid for some time, so they can use and sell
this valuable list to other junk mailers.

"Spammer blacklist"
http://www.netchem.com
... Dear Sir/Madam, Your email address may be on
many spammers' lists. We are compiling a *remove* list. Forward the
original junk to

"No Junk E-Mail database"
http://pages.ripco.com:8080/~glr/nojunk.html
...We will help stop unwanted email to you..the list is submitted to
us, and those addresses that appear in the "do not mail" list are
removed and the "cleaned" list is returned

3.7 UBE related newsgroups or mailing lists

alt.kill.spammers
alt.hackers.malicous
alt.2600

[1997-08-13 alt.privacy.anon-server by anonymous poster] Proper
etiquette demands you contact their ISP. However, if the ISP are
not interested in helping you, you should consider a posting in
alt.kill.spammers (or even alt.hackers.malicous or alt.2600) - give
as many details as you can about the spammer.

A certain spam-provider targeted the alt.hackers.malicious
newsgroup. Not the most sensible thing to do. The ISPs IPs were
found, their MX host was hacked. All their DNS entries was
published on alt.2600 (so that everyone could add filters to ignore
all mail from this company). Oh yeah, their password file also made
it to the group! The ISP then posted a complaint to alt.2600, much
to the enjoyment of everyone who took part. That host basically
died a horrible death. I'm pretty sure that not many people are
going to lose any sleep over this! I might as well mention that the
ISP's complaint mentioned that their "freedom" was being
abused. hehehe. Most of these postings can be seen in dejanews
or altavista archives of usenet.

"SPAM-L mailing list and Doug Muth's Page"
http://www.claws-and-paws.com/spam-l/
... "The SPAM-L FAQ" - A FAQ for SPAM-L, an anti-spam mailing list.
This FAQ discusses how to join the list and what to post there, AND
it also delves into the technical aspects of spam. For instance,
the various kinds of forgeries seen in spams are discussed here,
along with information on how to recognise them. If you hate spam,
this is something worth checking out... "TheGoodsites List" - I
maintain this list, which is part of the Spam Boycott, to show
which Internet providers out there act responsibly when dealing
with spam. If you're looking for an ISP and want to know where they
stand on spam, this is the list for you.

Send an email message to
with the words "subscribe SPAM-L " in the
body of the message (no quotes). f you would like to contact the
owner, the convention is the same as with all listserv lists. Just
send e-mail to

3.8 Software: the net abuse page

Scott Hazen Mueller
http://spam.abuse.net/spam/tools/

3.9 Software: adcomplain -- Perl junk email rport

http://www.rdrop.com/users/billmc/adcomplain.html

Adcomplain runs under Unix, Windows-NT, and Windows-95. Adcomplain
is a tool for reporting inappropriate commercial e-mail and usenet
postings, as well as chain letters and "make money fast" postings.

It automatically analyzes the message, composes an abuse report,
and mails the report to the offender's internet service provider.
The report is displayed for your approval prior to mailing.
Adcomplain can be invoked from the command line or automatically
from many news and mail readers.

#todo: url missing

[a user happy user reports] ...About 95% of all cases can be
traced correctly --- unless they come from a known spamhouse;
where complaining to them would not do much good anyway. Mailing
lists with strange Received-Headers also can present problems in
tracing

3.10 Software: Ricochet -- Perl junk email rport

http://www.vipul.net/ricochet/
Vipul Ved Prakash

MailingLi´st: with subject
"subscribe"

A lot of unsolicited email goes unreported because tracing the
origins of a possibly forged mail and finding the right people to
report to is complicated and time-consuming. Ricochet, a smart net
agent, automates this process. It traces the names and add resses
of the systems where the spam originated from along with the
servers that provide domain name resolution services to these
systems (in most cases their ISPs). Then it collects/generates a
list of email addresses of tech/billing/admin/abuse contacts of
these system and mails them a complaint and a copy of the spam.
Detailed description of its workings can be found in the README
file that comes with the package.

3.11 Software: yell -- perl

ftp://ftp.netcom.com/pub/bo/bobmacd/yell (57k)
Bob MacDowell

yell - auto-responds to "spam" e-mail. Scans for site names, e-mail
addresses and Web site names and sends appropriate messages to
users, postmasters and Webmasters.

3.12 Software: ifile - Perl

http://www.cs.cmu.edu/~jr6b/ifile/
Jason Daniel Rennie

...ifile is different from other mail filtering programs in
three major ways: 1) ifile does not require you to generate a set
of rules in order to successfully filter mail 2) ifile uses the
entire content of messages for filtering purposes 3) ifile learns
as you move incorrectly filtered messages to new mailboxes ifile is
not dependent upon any specific mail system and should be adaptable
to any mail system which allows an outside program to perform mail
filtering. Currently, ifile has been adapted to the MH and EXMH
mail systems.

3.13 Software: RBL lookup tool -- C

[1997-12-04 PM-L Edward S. Marshall ]

...rblcheck is a lightweight C program for doing checks against
Paul Vixie's Blackhole List. It works well in conjunction with
Procmail for filtering unwanted bulk email (under QMail, for
example, you can invoke it with the value of the environment
variable TCPREMOTEIP). rblcheck is extremely simple:

% rblcheck 1.2.3.4

where 1.2.3.4 is the IP address you want to check.

This is a quick note to announce the availability of a new tool for
using Paul Vixie's RBL blacklist (see http://maps.vix.com/rbl/ for
more information about the blacklist itself, if you don't already
know). Most tools which use the blacklist block email on a
site-wide basis. For many networks, this treads on both the ideals
of the administration, and on the perceived freedoms of the end
user.

Personally, I don't care either way. :-)

This tool was to fill the need I personally had to reject mail,
since one of the systems I receive mail through cannot, for various
political reasons, implement the available RBL filters on a
site-wide basis.

rblcheck is a simple tool meant to be used from procmail and
other personal filtering systems under UNIX in the absence of a
site-wide filter, as an alternative to imposing site-wide
restrictions, or as a means of imposing restrictions on systems
that cannot support the existing RBL filter patches.

Simply put: you hand it an IP address, and it determines if the IP
is in the RBL filter, providing the caller with a positive or
negative response. With the package, a sample procmail recipe is
provided, and examples of using it under QMail and Sendmail are
given.

.http://maps.vix.com/rbl/
.http://www.isc.org/bind.html The official home page
.http://www.xnet.com/~emarshal/rblcheck/

It has only been tested under Linux 2.x and Solaris 2.5.1. Success
stories, patches, questions, suggestions, and flames can be
directed to me at .

[PM-L Aaron Schrab ] Here is my rbl
setup, but, this depends both upon the format of the Received:
lines, and the way that mail passes through your mail system.

I currently grab the IP address from the first Received: header
inserted by my ISP (I'm a sysadmin at the ISP, so I have a good
knowledge of how mail gets passed around internally). Here's the
recipe that I use.

# if there's a Received: header from one of these servers, it's
# (probably) the right one

BACKUPSERVER = "([yz]\.mx\.execpc\.com)"
VIRTSERVER = "(vm[0-9]+\.mx\.execpc\.com)"
LOCALSERVER = "([abc]\.mx\.execpc\.com)"

# Match a header containing:
# Received: []) by

:0
* $ $SUPREME^0 ^Received:.*\[\/[0-9.]+\]\)$s+by$s+${BACKUPSERVER}
* $ $SUPREME^0 ^Received:.*\[\/[0-9.]+\]\)$s+by$s+${VIRTSERVER}
* $ $SUPREME^0 ^Received:.*\[\/[0-9.]+\]\)$s+by$s+${LOCALSERVER}
{
IP = $MATCH

# trim it down to just the IP address

:0
* IP ?? ^^\/[0-9.]+
{
IP = $MATCH

:0 W
* ! ? /home/aarons/bin/rblcheck -q $IP
{
SPAM = "$SPAM $IP is rbl'd$NL"
}
}
}

It seems to be a procmail issue with letting the IP info
from sendmail pass through to the rblcheck program. I have not
been able to find anyone using rblcheck successfully with
procmail as a delivery agent...

[1998-03-26 PM-L Edward S. Marshall ] This is a
standard problem; you should be able to change the invocation of
procmail the same way as the example (run env, which in turn runs
procmail). Make sure that there is a '-p' argument passed to
procmail; this preserves the environment you're constructing with
env (newer sendmail revisions sanitize the environment for you, so
that's not really an issue).

If you're still having troubles, make sure you're using the latest
incarnation of rblcheck, with the latest supplied procmail recipe;
earlier revisions had rather insidious bugs.

[1998-03-26 PM-L Xavier Beaudouin (kiwi) ] Also it
seems that sendmail 8.9.0Beta3 has builtin rules for
rbl.maps.vix.com. This is somewhat really efficient. I use it with
sendmail 8.8.8 and tcpwrapper every day and there is about 80%
spam rejected. Sounds very good. In your /etc/hosts.allow just add
the following lines :

sendmail: ALL: spawn /usr/local/bin/rblcheck -q %a && \
exec /usr/sbin/sendmail -bs || /bin/echo \\
"469 Connection refused. You are in my Black List !!!\r\b\r\n"
&& \
(safe_finger -l @%h 2>&1 | /bin/mail -s "%d-%h %u" root)

In your /etc/inetd.conf just add this line :

smtp stream tcp nowait root /usr/sbin/tcpd \
/usr/sbin/sendmail -bs

And check that your sendmail is _not_ working as a daemon. That's
all. Also if you have huge queue you can add a /usr/sbin/sendmail -q
in the root crontab... This should help to send some waiting
messages. I think we can use this to wait for official 8.9.0
sendmail since there is some cf/feature/rbl.m4 there.

[timothy] ...I think there's a much more efficient way to do
this: you can compile sendmail -DTCPWRAPPERS and let it run as a
daemon

3.14 Software: mapSoN

Note: You can do exactly the same as below with procmail with one
of the listed procmail modules: pm-jacookie.rc. See the code.

"mapSoN (NoSpam backwards) -- The no spam utility"
http://mapson.gmd.de/
ftp://ftp.gmd.de/gmd/mapson/

Most spam filtering tools I've seen so far are based on procmail, or
a similar tool, and use a list of keywords or addresses to drop
unwanted junk mail. While this might be nice to filter mail from
known spam domains like "cyberpromo.com", it won't catch faked
headers.

mapSoN must be installed as filter program for your incoming mail,
usually by adding an appropriate entry to your $HOME/.forward file.
This means that mapSoN will get all your incoming mail and it will
decide whether or not to actually deliver it to your mailbox.

. First of all, an user defined ruleset is checked against the
mail. If any keywords or patterns match, the mail will be dealt
with according to your wishes. This is useful to drop some
sender's mail completely, or to sort mail into different mail
folders.
o If no rule matches the mail, mapSoN will check whether the mail
is a reply to an e-mail you sent, or whether it is a reply to a
USENET posting of yours. If it is, the mail will always be
delivered.
o If no signs of a reply-mail can be found, mapSoN will check
whether the sender stated in the From: header has sent you mail
before. If he has, the mail will pass. If this is the first time
you receive an e-mail from this address, though, mapSoN will
delay the delivery of the mail and spool it in your home
directory. Then it will send a short notice to the address the
mail comes from, which may look like this:

From: Peter Simons
To: never_mailed@me.before
Subject: [mapSoN] Request for Confirmation

mapSoN-Confirm-Cookie:

The person who tried to contact you will then reply to this
"request for confirmation", citing the cookie stated in the mail.
When your mapSoN receives this confirmation mail, it will deliver
the spooled mail into your folder. Furthermore, the address will be
added to the database, so that mail from this person will pass
directly in future.

If no confirmation mail arrives within a certain time, mapSoN can
either delete the spooled mails, or send them to a special folder,
or whatever you prefer.

3.15 Software: spamgard

[similar to MapSon]
ftp://ftp.netcom.com/pub/wj/wje/release/sg-howto

...sppamgard(tm) screens from your e-mail unsolicited bulk mail. It
does this in a way that you only have to change things if you have
a new person from whom you _do_ want to receive mail; you don't
have to change things every time a spamster thinks of a new trick
to pull, or a new spamster comes along. And spamgard(tm) is
designed so that those who aren't in your "Good Guys" list can get
mail to you anyway until you put them there. The instructions for
them to get mail to you are simple and newbie-tested, but will
still keep out bulk mail. If you're on a mailing list you _want_ to
be on, there are provisions for accepting all mail from a set of
mailing lists that you specify.

3.16 Software: Spam Be Gone

"Spam Be Gone"
http://www.internz.com/SpamBeGone/
...uses machine learning and artificial intelligence technologies
to examine incoming mail messages and determine their
priority... is more than just a Spam filter, it's a general purpose
mail message prioritiser. You train the system, telling it which
are good, and which are bad messages. As Spam Be Gone! learns it
becomes customised for each individual user.

PM-L W. Wesley Groleau comments:

.> They only distribute binaries, and I'm paranoid. Anyone able to
.> convince me it's not really a Trojan Horse to collect addresses of
.> spam-haters or something even worse?

I did some sleuthing. I am 95% convinced that SpamBeGone is not
a front or cover for any spammer(s). To protect the author's
privacy, I won't say why I'm convinced or how I got the info.
Sorry. If you're paranoid like me, you'll have to do your own
sleuthing before you use it.

I'm also convinced SpamBeGone's theory is sound. I won't judge
the implementation until I've used it for a while.

PM-L R Lindberg & E Winnie comments:

I have to agree with the recent comments about Spam Be Gone, I
found it tends to be inaccurate. I first set it up about a week
ago, followed the directions and trained it on several (15 to 20)
messages. One from each list we get, and the remainder from my
logs of SPAM messages.

The first day it missed about half the SPAM, and nailed about 1/3
of the real messages. So I tuned the key-words a bit, trained it
on about 100 more SPAMs and trained it on all the good messages
it nailed. Since then it has nailed every SPAM received, however
the second day it nailed about 20% of the good messages, which I
then trained it to like. Since then it has been nailing about
10% of the good messages, despite continual training. I also
added every list to the address book, and it still nails posts
from this list, and my wife's lace list.

I even went through my entire log of SPAM and trained it on every
one that didn't come out a 5 (bad). Being the kind of person I
am, I also checked after I trained it, and found four SPAMs, the
despite my training it that they were bad (5) came out as not so
bad (4). I don't dare kill 4's as far too much of my mail (like
this list) ends up as 4's.

For me, this program is not ready for prime time. If the comments
are correct that it only learns on Subject and From headers, it's
not even worth trying. Since lists use the TO and CC headers to
be identified, and there are several excellent other headers
(X-Advertisement comes to mind) that would be assests for killing
SPAM.

3.17 Software: ClearMail

http://www.clearmail.com/ 1998-08-27
Scott R Carter

ClearMail offers individuals some very strong control over spam
through a quite unique concept. The software includes Procmail,
Perl and C code. System Requirements include:

ClearMail helps to control spam by allowing a user to classify
e-mail as high or low priority based on an Address Book or "White
List" of known senders. Unknown senders can also send high priority
mail by including a special Mail Key (token) in their message
(initial message from unknown sender without valid Key results in a
bounceback message with instructions).

What makes ClearMail different from similar concepts is that
spammers are not able to easily obtain the Mail Key to bypass the `
system because it is conveyed as an image.

o Unix operating system
o Shell accounts for users
o Individual .forward, .procmailrc files
o Sendmail
o Procmail
o Perl
o Public Web server

3.18 Software: TinyGnus - Emacs Gnus plug-in

http://poboxes.com/jari.aalto/ema-tiny.html

Platform: win32 and Unix Emacs versions.

*TinyGnus* Is Emacs lisp extension package that integrated directly
to Gnus mail/newsreaders. It includes simple but efective UBE
fighting hotkeys that make it possible to complain bunch of UBE
messages a once. Features:

o USER MUST DECIDE WHICH IS *ube* MAIL.
o User selects messages that are ube with Gnus select commands.
o Hotkey C-c ' u examines messages' headers and runs `nslookup'
for each Received header to determine *abuse* *spam* and
*postmaster* addresses where to send the complaint.

4.0 Procmail pointers

4.1 Where to get procmail binary

ftp://ftp.informatik.rwth-aachen.de/pub/packages/procmail/
On-Line manual: http://www.voicenet.com/~dfma/intro.html

4.2 Where is procmail developed

Philip Guenther is currently taking care of and
coordinating procmail bug fixes. Please send any procmail bugs to
the mailing list or to . The development mailing
list is running SmarList at . Further
patch and bug info can be found at:

http://www.gac.edu/~guenther/procmail/todo.html
http://www.gac.edu/~guenther/procmail/warts.html

Newest Procmail code:

http://www.procmail.org/
ftp://ftp.procmail.org/

4.3 About procmail's Y2K compliance

Please consult Philip Guenther for more up to date
details. Philip is the Procmail maintainer currently.

[1998-09-23 Bennett Todd in Message-Id:
<19980923164230.C30594@fcmc.com>] Well, from a simple ogle of the
grep over the sources, it looks like there may be a Y2038 problem
in the autoconf test code: unsigned otimet = time(). And another,
possibly less likely to express itself, in formail.c: unsigned long
h1 = time(). Those could express themselves when 32-bit signed
time_t wraps; long before then the time_t define should have been
changed to something that is bigger, even if it's "long long". The
above type-mixes may fail to profit from a suitably redefined
time_t, and so may overflow on 2038.

I don't see any Y2K problems, though. And email headers use
four-digit years pretty consistently, so that should all be cool.
This estimation doesn't constitute an in-depth Y2k audit of
procmail, but the source code to procmail is ... kinda dense for
in-depth auditing.

[1998-09-25 Bennett Todd Message-Id:
<19980925093902.B12428@fcmc.com>] As I see it there are at least
three measures that a whole email system, taken in aggregate, could
use for Y2K checking. First, capture a vast cross-section of
traffic and make sure no email software is using 2-digit years. I
don't recall having seen any, but it's still worth checking.
Second, generate a load of traffic with 2000 and 2001 dates and
shove it through all the channels. And third, run all the systems
end-to-end with their system clocks rolling over the millenium.

4.4 Procmail mailing lists

Traffic in this list is about 5-20 messages per day. Do not join
if you can't handle that much traffic. The list is run by SmartList,
which is a procmail-based list management and distribution package.

._MailingList_: questions/answers
.subscription requests
.digest request

To get off the procmail mailing list

To get off the list: send a message to *procmail-request* with:

unsubscribe user@domain in the subject line
unsubscribe first line in the body

If that fails, try email to
(purportedly that should
go to a person). See also the original subscriptions message that
you will received http://www.iki.fi/~era/procmail/welcome.txt

4.5 Procmail recipe modules and faqs

Procmail is discussed in usenet newsgroup *comp.mail.misc*.

"Procmail archive"
ftp://ftp.informatik.rwth-aachen.de:/pub/packages/procmail/
Articles from procmail mailing list: covers from 1994-08 to 1995-05
(A .gz file: ~2Meg when uncompressed)

And latest articles can be found here, hosted by Achim Bohnet
Covers from 1995-10 to the present day.
. The www page has nice search capabilities.
http://www.rosat.mpe-garching.mpg.de/mailing-lists/procmail/
http://www.rosat.mpe-garching.mpg.de/~ach/exmh/archive/procmail/

"Era's Procmail faq"
http://www.iki.fi/~era/procmail/mini-faq.html
http://www.dcs.ed.ac.uk/~procmail/faq/ [mirror]
Also available by email, the ITEM can be: links.html, mini-faq.html,
procmail-faq

To:
Subject: send ITEM

"Era's Procmail Link collections"
http://www.iki.fi/~era/procmail/links.html
...A page full of good links to the world of procmail

"Catherine's Getting Started With Procmail"
http://shell3.ba.best.com/~ariel/nospam/proctut.shtml
This is a quick tutorial intended to get a procmail neophyte
started using procmail with as little trouble and fuss as possible.

"Joe Gross's short Procmail tutorial"
http://www.procmail.net/
...Using procmail and a
feature of ph you can set up your own mailing list without
needing root on your own machine.

"Unix manpages"
http://www.xs4all.nl/~pater/manpages/
...If you don't have procmail manpages at hand, check this site.
It contains a wealth of Unix related manpages online.
Jeroen Paternostre

4.6 Procmail mode for Emacs

If you use Emacs, please download the Procmail
programming mode, `tinypm.el'. Lint is included in there and it can
auto-correct mistakes on the fly. You can get it from the mentioned
_uta_ ftp site. Here is an example of its output:

*** 1997-11-24 22:13 (pm.lint) 3.11pre7 tinypm.el 1.80
cd /users/jaalto/junk/
pm.lint:010: Warning, no right hand variable found. ([$`']
pm.lint:055: Pedantic, flag orer style is not standard `hW:'
pm.lint:060: Warning, message dropped to folder, you need lock.
pm.lint:062: Warning, recipe with "|" may need `w' flag.
pm.lint:073: Warning, Formail used but no `f' flag found.

4.7 Procmail module list

Where to get the modules

The UBE stop procmail modules are not listed here. See pointers in
"procmail code" section later.

o All pm-ja*.rc modules are in Jari's procmail kit.
The Procmail code library page is at
http://www.procmail.org/jari/pm-code.html

o Other modules are by Alan Stebbens http://reality.sgi.com/aks/

o 1998-12-08 Eli the Bearded <*@qz.to> announced in
comp.mail.misc that he had made his procmail modules available
at http://www.qz.to/eli/src/procmail/. You may find
interesting procmail code there but the modules themselves are not
general purpose *plug-in* modules that you could use right
away. Some functionality included:

Inline decoding of MIME text attachments (rc.mime-decode)
Cleansing of obscure "Re:" formats in subject (rc.pre-list)
Nifty autoresponder (rc.qz-2)
Sophisticated dupicate email catching (rc.dupes)
Example of using my mail bouncer (rc.lists-out)
Detection of some classes of autoreplies (rc.daemon)
Various junk mail filtering (rc.filter)
Daily log files (rc.vars)

Terminology

*subroutine* = A piece of code that gets something in `INPUT' and
responds with `OUTPUT'. Subroutine is not message specific.

*recipe* = A piece of code that is somewhat self contained:
It reads something from the message or does something
according to matches in message. Recipe may be message-specific.

Foreword to using modules

In the module listing, some of the modules are recipes and some can
be considered subroutines. Let's take the address exploder module
that was discussed a while ago. First, visualise following familiar
programming language pseudo code:

(ret-val1, ret-val2 ...) = Function( arg1, arg2, arg3 ...)

*Function* may return multiple arguments and multiple arguments can
be passed to it. Clear so far. Let's show how this applies to
procmail modules:

RC_FUNCTION = $PMSRC/pm-xxx.rc # name the subroutine/module
RC_FUNCTION2 = ...

INPUT = "value" # Set the arg1 for module
INCLUDERC = $RC_FUNCTION # Call Function( $arg1 )

:0 # Examine function ret val
* ERROR ?? yes
...

This should be pretty clear too. You just have to look into the
subroutine/module which you intend to use, to find out what
arguments it wants which you _need_ _to_ set (INPUT) before calling
it. The documentation also tells you what values are returned, e.g.
one of them was ERROR.

If it were recipe/module, the call would be almost the same, but
instead of returning values, the recipe/module most likely does
something to your message or writes something to the data files
etc. A *Recipe/module* is much higher level, because it may
call multiple subroutine/modules. The distinction between
subroutine and recipe module type is not crystal clear, but I hope
the above will clarify a bit the Procmail module/subroutine/recipe
concept.

Header file modules

These are like #include .h files in C, they define common
variables, but do not contain actual code.

o pm-javar.rc -- Defines standard variables: SPC WSPC NSPC SPCL and
perl styled \s \d \D \w \W and \a \A (alphabetic characters only)
o headers.rc -- From Alan's procmail-lib. Define standard regexp
and macros: address, from, to, cc, list_precedence

General modules

o *pm-jafrom.rc* -- Derive FROM field without calling `formail'
unnecessarily. If all else fails, use formail.
o *get-from.rc* -- From Alan's procmail-lib. get the "best" From
address. Sets FROM and FRIENDLY, the latter being the "friendly"
user name sans address.
o *pm-jaaddr.rc* -- Subroutine to extract various email components
from INPUT. Like address=foo@some.com, net=com, account=foo...
o *pm-jastore.rc* -- Subroutine for general mailbox delivery.
Define MBOX as the folder where to drop
message and this subroutine will store it appropriately.
Supports single mboxes, ".gz" mbox files, directory files and
MH folders with rcvstore.

Date and time handling

For these, you get the date string from somewhere, then feed
it to some of these subroutines:

o *pm-jatime.rc* -- a low-level subroutine. Parse time "hh:mm:ss"
from variable INPUT
o *pm-jadate1.rc* -- a low-level subroutine. Parse date
"Tue, 31 Dec 1997 19:32:57" from variable INPUT
o *pm-jadate2.rc* -- a low-level subroutine. Parse ISO standard date
"1997-11-01 19:32:57" from variable INPUT
o *pm-jadate3.rc* -- a low-level subroutine. Parse date
Tue Nov 25 19:32:57 from variable INPUT
o *pm-jadate4.rc* -- Call shell command "date" once to construct RFC
"Tue, 31 Dec 1997 19:32:57" and parse the YY MM HH and other
values. You usually use this subroutine if you can't get the date
anywhere else.

Date and time handling

You use these recipes to get the date directly from the message:

o *pm-jadate.rc* -- higher-level recipe. Read date from message's
headers: From_ Received, or call shell `date' if none succeeds.
o *date.rc* -- higher-level recipe.
From Alan's procmail-lib: parse date or from headers
Resent-Date:, Date, and From

Forwarding and account modules

o *pm-japop3.rc* -- Pop3 movemail implemented with procmail. You can
send a "pop3" request to move your messages from account X to
account Y. Each message is send separately. This recipe listens
to "pop3" requests.
o *pm-jafwd.rc* -- control forwarding remotely. You can change the
forward address with a "control message" or turn
forwarding on/off with a "control message"
o *pm-japing.rc* -- Send short reply when subject contains the word
"ping" to show that the account is up and email address is
valid.
o *correct-addr.rc* -- From alan's procmail lib. To help forward mail
from an OLD address to a NEW address, and do some mailing list
mail management. This recipe file is intended to make it easy
for users to forward their mail from their old address to a new
address, and, at the same time, educate their correspondents
about it by CC'ing them with the mail.

Vacation modules

o *pm-javac.rc* -- A framework for your vacation replies. This
recipe will handle the vacation cache and compose an initial
reply; which you only need to fill in. (Like putting vacation
message to the body)
o *ackmail.rc* -- From Alan's procmail lib. procmail rc to
acknowledge mail (with either a vacation message, or an
acknowledgement)

Message-id based modules

o *pm-jadup.rc* -- Handle duplicate messages by Message-Id.
Store duplicate message in separate folder.
o *dupcheck.rc* -- From Alan's procmail-lib. If the current mail has
a "Message-Id:" header, run the mail through "formail -D",
causing duplicate messages to be dropped. Can use MD5 hash in
cache.

Cron modules

o *pm-jacron.rc* -- A framework for your daily cron tasks. This
recipe contains all the needed checks to ensure that your
includerc is called whenever a day changes. (Day change is
subject to messages you receive). Your own cron includerc is
run once a day.

Backup modules

o *pm-jabup.rc* -- Save messages to backup directory and keep only N
messages per day. Idea by John Gianni, packaged by Jari. Note:
The implementation will always call shell for each message you
receive; so using this module is not recommended if you get
many messages per day. Instead, use the cron module to clean
the messages' backup directory only once a day, and not everytime
a message arrives.

Confirmation modules

o *pm-jacookie.rc* -- Handle cookie (unique id) confirmations.
Also known as Procmail authentication service (PAS).
This simple procmail module will accept messages only from
users who have returned a "cookie" key. You can use this to
to protect your mailing list from false "subscribe" messages
or from getting mail from unknown people, typically spammers
who won't send the cookie back to you to "validate" themselves.
Uses subroutine pm-jacookie1.rc, which generates the unique
cookie; CRC 32 by default.
o See also Michelle's confirmation module for SmartList

File Servers

o *pm-jasrv.rc* -- A Mime Procmail file server (MPFS) It contains
all the instructions and supports several MIME encoding types:
text/plain and gzip. The keyword SEND is configurable. You
can set up as many file servers as you need to different
directories by changing the SEND keyword. MPFS supports
password for file access.
o *commands.rc* -- From Alan's procmail-lib, check for commands
in the subject line. Handles commands (send|get)
[help|info|procmail info|procmail lib|procmailrc] and a few
others.
o *send-file.rc* is a very simplistic piece of procmail code
to send file (non-MIME support) requested in subject line.
http://www.universe.digex.net/~mbr/unix/send-file.html

Mime modules

o *pm-jamime.rc* -- Subroutine to read MIME headers and put the
mime version, boundary string, content-type information to
variables.
o *pm-jamime-decode.rc* -- recipe to decode quoted-printable
or base64 encoding in the body.
o *pm-jamime-kill.rc* -- Recipe for attachment killing: wipes out the
extra mime cruft leaving only the plain text. Applications for
killing: ms-tnef attachment (MS Explorer 7k),
html attachments (netscape, MS Express) vcard (Netscape),
PCX attachment (Lotus Notes).
o *pm-jamime-save.rc* -- Recipe for saving simple file attachment.
When you receive _ONE_ file attachment in a message, this
recipe can save it in a separate directory. The content is
also decoded (base64,qp) while saving.

Filtering message body or headers

o *pm-jadaemon.rc* -- Handle DAEMON messages by changing subject to
reflect a) the error reason b) to whom the message was originally
sent c) original subject sent and what was the subject. Store the
DAEMON messages to separate folder.
o *pm-jasubject.rc* -- Standardize Subject "Re[32]: FW: Sv: message"
or any other derivate to de facto "Re: message"
o *pm-janetmind.rc* -- Reformat http://minder.netmind.com/ messages,
The default 4k message is shortened to a few important lines.

Miscellaneus modules

o *pm-jaempty.rc* -- check if message body is empty (nothing
relevant). Define variable BODY_EMPTY to "yes" or "no" if
message is empty.
o *pm-janslookup.rc* -- Run nslookup on given address. If you
compose return address with "formail -rt -x To:" you can
verify if domain is registered before sending reply. Uses cache
for already looked up domains.
o *guess-mua.rc* -- Guess the Mail User Agent and set MUA:
MH,PINE,MAIL

Mailing list modules

o *Microlist* a small mailing list module by david hunt
...This version contains vars set for my environment and needs,
and requires resetting of those vars before use. Its exact
function and use will remain a mystery until I get a readme
file written for it. If anyone wants to use it, I suggest you
write to me first. If anyone has any suggestions or criticisms
(no matter how harsh) please write
http://www.west.net/~dh/homedir/microlist/microlist4.3
o *pm-jalist.rc* -- Subroutine to extract mailing list name from
message. Do you need to add a new recipe to your .procmailrc
every time you subscribe to new mailing list? If you do,
take a look at this module, which examines the message and
defines variable `LIST' to hold the mailing list name. You
can use it directly to save the messages adaptively to
correct folders. No more hand work and manual storing
of mailing list messages.

4.8 Where to get Procmail code and modules

"Alan's procmail modules"
Send subject "send procmail library" to Alan Stebbens
http://reality.sgi.com/aks/

"pm-code, Jari's Procmail modules"
http://www.procmail.org/jari/ --> See pm-code.zip or *shar* file.

"Elijah's"
http://www.qz.to/~eli/src/procmail/rc.master.html

"Concordia scripts"
http://alcor.concordia.ca/topics/email/auto/procmail/
...We provide sample sets of recipes to get you started. The great
thing about the concordia scripts is the fact that they are
designed to run from a central location and be called from a
.procmailrc installed in the user's ~/home directory.

"Meng on procmail"
http://icg.resnet.upenn.edu/procmail/
http://res2.resnet.upenn.edu/procmail/
...goes into exhaustive detail about how I manage my mailing lists

"David's" David Hunt
...My .procmailrc and .forward files can be viewed at
http://www.west.net/~dh/homedir/pmdir/

4.9 Procmail code to filter UBE

_Sysadms_ _remember_ : Spam filtering is much more efficiently done
in the MTA, especially if you are just looking at From and To lines.
For example, you can setup in Exim a rule that blocks \d.*@aol\.com
(that is any aol.com local part that begins with a digit). AOL
guarantees that _none_ of their addresses begin with a digit. Exim
rejects such bogus addresses at the SMTP level before the message
is received.

"Daniel's smap filter"
1997-09-13 Daniel Smith sent excellent spam filter
called `spamc.rc'. It used some nice heuristics and filters from
various people, including [david] and [philip].
Later Dan made substantial changes to it and the new version is
available from ftp://ftp.bristol.nl/pub/users/DanS/spamcheck

"pm-jaube.rc Jari's ube filter (compiled from others)"
After Daniel Smith posted his spam recipes to procmail mailing
list, Jari investigated them and compiled other recipes to a
general purpose UBE module that needs no special setup and can be
installed via simple INCLUDERC. No additional ube-list files are
used, all UBE detection happens using procmail rules. The module
is included in kit `pm-code.zip'.

"Catherine A. Hampton's Spambouncer"
http://www.best.com/~ariel/nospam/
...The attached set of procmail recipes/filters, which I call
The Spam Bouncer, are for users who are sick of spam (unsolicited
junk email) and want to filter it out of their mail as easily
as possible. These recipes can be used as shared recipes for a
whole system, or by an individual for their own mailbox only.

"Protect yourself from spam: A practical guide to procmail"
http://www.sun.com/sunworldonline/swol-12-1997/swol-12-spam.html
...take you, step by step, through everything you need to know in
order to enlist the aid of a Unix host in filtering unwanted e-mail
traffic. This page is excellent to get you started with procmail
and filtering with simple recipes and how to store messages to
folders. Recommended for newcomers to Procmail.

"Junkfilter" by Gregory Sutter
http://www.pobox.com/~gsutter/junkfilter/
...Junkfilter is a user-configurable procmail-based filter system
for electronic mail. Recipes include checks for forged headers,
key words, common spam domains, relay servers and many others.

"Download procmail spam filters"
http://www.telebyte.com/stopspamr
This is excellent site and contains many other spam stop pointers.

"SpamDunk"
http://www.interlog.com/~waltdnes
http://www.interlog.com/~waltdnes/beta/techie.htm
...This webpage shows a commented example of a working .procmailrc
file that works for me. I have tried to make things as generic as
possible, but there are no guarantees that it will work for anyone
else.

5.0 Dry run testing

5.1 What is dry run testing

It means that you call your procmail test script directly with sample
test mail

% procmail $HOME/pm/pm-test.rc < $HOME/tmp/test-mail.txt

The script pm-test.rc has the procmail recipe you're testing or
improving. The test-mail.txt is any valid email message containing
the headers and body. You can make one with any text editor, e.g.
`vi', `pico' or `emacs' in your Unix system. Here's a
simple test mail skeleton:

From: me@here.com
To: me@here.com (self test)
X-info: I'm just testing

BODY OF MESSAGE SEPARATED BY EMPTY LINE
txt txt txt txt txt txt txt txt txt txt

Remember that you can define environment variables as well in
the dry run call. Here's an example where procmail just executes
the script and does nothing fancy.

% procmail VERBOSE=on DEFAULT=/dev/null \
~/pm/pm-test.rc < ~/txt/test-mail.txt

Suppose the script prints something to logfiles, but you'd instead
like to get it all dumped to screen. No problem, first find out
your tty value by calling `tty' at shell prompt and pass
that on the command line. Here the default LOGFILE is directed
to take care of redirecting "LOG=" commands and statement
"MYTEST_LOG=${MYTEST_LOG:-$HOME/pm/pm-test.log}"

# `tty' tells what to fill in /dev/..

% procmail VERBOSE=on DEFAULT=/dev/null \
LOGFILE=/dev/pts/0 MYTEST_LOG=/dev/pts/0 \
~/pm/pm-test.rc < ~/txt/test-mail.txt

5.2 Why the From field is not okay after dry run

It now says "From foo@bar Mon Sep 8 14:38:06 1997"

[philip] Don't worry about this. It's a side-effect of running the
message through formail after having generated any auto-reply --
the auto-reply generated by "formail -rt" doesn't have a "From "
header (it's pointless for outgoing messages), so the second
formail adds one, not knowing that it'll just be ignored by
sendmail later (well, sendmail will extract the date from it, but
that's ignorable). You only see it because you're saving to a
folder instead of the mailing it.

5.3 Getting default value of a procmail variable

[david] There's always this way to learn a variable's
initial value (note the strong quotes), which Stephen uses to get
procmail's value for $SENDMAIL in the scripts that build SmartList:

procmail LOG='$PATH' DEFAULT=/dev/null /dev/null < /dev/null

Since LOGFILE hasn't been defined, $PATH will be printed to the
screen. One caution: if there are any variables in the definition
of $PATH (such as $HOME), they'll be expanded in the output.

6.0 Things to remember

6.1 Get the newest procmail

Lot of troubles surface only because you have an old procmail version.
Be sure to have the latest which is 3.13.1 since 1999-04-05. Here is
a command to check your procmail version number:

% procmail -v

Knock your sysadm or ISP until he installs this version; don't give
up, if you're serious about using procmail.

6.2 Csh's tilde is not supported

Real csh or Emacs freaks have grown accustomed to using tilde (~)
everywhere, but must drop that habit now. Procmail doesn't support it;
just use `$HOME'. When you write procmail recipes, think *sh* not
*csh*. This mindset will automatically get your brain tuned to the
right programming habits.

6.3 Be sure to write the recipe starting right

The recipe starts with `:0' or just with `:' but the latter one is
somewhat dangerous and easy to miss. Beware writing it `0:' as it
happens easily. The Procmail code checker, Lint, also requires that
you use the `:0' recipe start convention.

[philip] Always put a zero after the colon that begins the recipe. In
the first versions of procmail, you would put the number of
conditions, with a default of 1. That was annoying, and the computer
can do the counting easier, so Stephen made it so that a count of 0
indicates that the conditions are all the lines beginning with a `*'.
The default is one, unless the `a', `A' , `e', or `E' flags is given,
in which case the default is zero. *ALWAYS* *START* a *RECIPE* *WITH*
`:0'.

6.4 Always set SHELL

[faq] If your login shell is a C shell (csh or tcsh), avoid
havoc: as a precaution, always put following at the top of
your .procmailrc.

SHELL = /bin/sh

If system has no /bin/sh and you're forced to use csh/tcsh

[] Csh and tcsh execute the .cshrc
first, THEN if, and only if it is the login shell (not a sub shell)
it executes the .login, which should contain basic important system
setting like `stty' commands. Likewise, bash and ksh users are
taught to define and export PATH in .profile, so our per-shell
startup files would not have clobbered the PATH set in .procmailrc
the way your .cshrc did.

[philip] ...I have been told by other sysadmins that there are
systems on which csh was hacked to source the .login before the
.cshrc. For various reasons I suspect these to be systems based on
older versions of BSD (say, 2.3 BSD).

As for tcsh, the order in which the .login and .cshrc is sourced is
a compile-time option which defaults to the .cshrc (or .tcshrc)
before the .login. There may be some wackos out there who change
the default in memory of the system(s) that they were raised on. I
suggest electroshock as the proper treatment.

...done sys admin on Crays, Convexes, Suns, SGIs, Decs, PC
running BSDI, Linux and Free BSD, and I have never run into a
system where the .cshrc is sourced AFTER the .login. If someone
goes to the trouble to change the order, I would love to know a
valid reason for it.

Procmail won't work well with SHELL set to csh derivate

[1998-08-17 PM-L Volker Kuhlmann]
...The blame lies with procmail and its documentation. Obviously,
procmail is programmed with the assumption that the login shell is
a sh derivative. This assumption is a) not very nice, and b) not
stated in the otherwise very good documentation. Of course a user
can set SHELL to tcsh. If then procmail is too stupid to hack it,
it ought to say so clearly, and the above-mentioned questions of
people using tcsh will disappear from this list. One could also be
nice and point out pitfall (3) mentioned above in the procmail
docs. It is customary to have terminal configuration in .login. If
it is shifted to .cshrc it should be properly surrounded by if ..
endif. Perhaps it is not customary to configure the terminal in
.bashrc (where else then? - only a rhetorical question), but that
is no reason to blame it on tcsh.

My .cshrc only setenvs the environment when it is a login shell
(shell level 1). Obviously procmail runs a login shell. As I said
earlier, there are good reasons for setting a full PATH
independently whether the shell is interactive or not. So, when
procmail executes programs with SHELL=tcsh, PATH is set to the tcsh
defaults. That may or may not be desirable, depending on the
individual case. No problem with that and avoidable (run tcsh with
-f). Nice if it was in the procmail docs.

But then, the PATH getting clobbered is not the point here (just a
side-effect I didn't realise until 2 people pointed it out).

6.5 Check and set PATH

[jari] It is very likely that the default PATH environment variable
that your .procmailrc sees it not enough. To play safe, so that all
the needed binaries can be found when escaping to shell in
.procmailrc, set the `PATH' variable as a very first statement.
Here is one example that I use for HP-9 HP-10 and in SUN-OS.
You can add paths that don't exist, that way you can use the
same .procmail on multiple servers (On HP and SUN as I do)

PATH = $HOME/bin:\
/usr/contrib/bin:\
/bin:/usr/bin:/usr/lib:/usr/ucb:/usr/sbin:\
/usr/local/bin:/opt/local/bin:\
/vol/bin:/vol/lib:/vol/local/bin:${PATH}

[Richard] It is dangerous to have many directories in the PATH,
especially if you do not control the content of any of them. A
sysadmin could put a newer, incompatible version of a program you
rely upon in one of them and you cause difficult-to-diagnose
problems. It may make more sense to link the binaries you need into
your own ~/bin directory and include just that in your PATH.

[jari] In principle I agree with Richard's advice, but in practice
the newer version of the program seldom breaks the procmail code
you have written. It depends on your "threat level": be more
cautious and use Rik's advice; alternatively trust the system and
adapt to (rare) changes. Your call.

6.6 Keep the log on all the time

It's best that you put these variables at the very start of
your .procmailrc. When you start using procmail, you also want to know
all the time what's happening there and why your recipes
didn't work as expected. The answer to almost all your questions can
be found in the log file. As the log file will grow to be quite big,
remember to set up a cron job to keep it moderate size.

LOGFILE = $PMSRC/pm.log
LOGABSTRACT = "all"
VERBOSE = "on"

6.7 Never add a trailing slash for directories

[philip] Drop the trailing slash: it'll choke if you ever end up on
Apollo's DomainOS where double slashes are network references. If
the directory has a trailing slash, it will choke
on most OSes (they treat it like "/.").

DIR = /full/path/to/www/directory/ # Wait...
FILE = $ARCHIVEDIR/file # Ouch !

6.8 Remember what term DELIVERED means

[alan] When procmail delivers a piece of mail, whether to a
file or a pipe-command, if the write succeeds, then the mail is
considered to have been delivered, and processing stops with that
recipe file. Here is the relevant text from man page:

...There are two kinds of recipes: delivering and non-delivering
recipes. If a delivering recipe is found to match, procmail
considers the mail (you guessed it) delivered and will cease
processing the rcfile after having successfully executed the
action line of the recipe. If a non-delivering recipe is found to
match, processing of the rcfile will continue after the action
line of this recipe has been executed.

6.9 Beware putting comment in wrong place

You like commenting a lot, sticking them everywhere possible?
Yes, I do that too, and got into trouble because one is not that
free to comment code in procmail. Pay attention to the following
example

:0 # comment, nice tune...
* condition # OUCH, Ouch, ouch. This comment must not be here!!
# Hm, Old procmail versions don't understand this
# Are you sure you want to put comments inside
# Condition line?
* condition
{ # comment ok
# comment ok
:0 # comment ok
/dev/null # comment ok
} # comment ok

So, the place to watch is the *condition* line. Some later procmail
versions promised to correct this misfeature, but it never came
true. No procmail exists yet that allows putting comments
on the same line with a condition clause.

6.10 Brace placement

Be careful with your braces and remember that old procmail
versions aren't as forgiving as newest version. Below you see
classical "Test OK condition first, and if that fails then do
something else". See the side comments.

:0
* condition
# No space allowed here!
{} # Wrong, at least _one_ empty space
:0 E
{do_something } # Again mistake, must have surrounding spaces

6.11 Local lockfile usage

Lockfiles are only needed when procmail is doing something that
should be serialized, i.e., when only one process at a time should
be doing it.

This generally means that any time you write to a file, you should
have a locallock, preferably based on the name of the file being
written to. Forwarding actions ('!'), and 99% of all filters don't
need lockfiles. However, if a filter action writes to a file while
filtering, then you may need a lock. Procmail always does kernel
locking when it writes mail to files via simple file actions. So
even if you forgot the lock colon, procmail tries to play safe if
kernel locking has been compiled in.

Beware misplacing the lock colon(:)

:0: a # Ouch! Wrong unless you want a lockfile named a
:0 a: # Okay.

Note that in delivering recipes where you manually write the
content, you must use local lockfile with `>' token, because
procmail can't determine lock by itself. It can only determine the
lockfile from the `>>' token. [stephen] However, putting a
lockfile on a recipe like this is, of course, utterly useless. So
you might as well omit the locking entirely.

# Save last body of message to file mail.body

:0 b: mail.body$LOCKEXT
| cat > mail.body

o If the command line in the procmail rcfile contains ">>",
a name for the local lockfile will be implicit, and the second
colon alone is enough.
o If the command doesn't write to a file, or doesn't write to the
same file as anything else (including a matching letter that makes
procmail run the same command) that might run at the same time,
the local lockfile is unnecessary.

[philip] Watch this too. A nesting block that does not launch
a clone cannot take a local lockfile on the recipe that starts the
braces. A nesting block that does launch a clone can. (see
the error)

:0: file$LOCKEXT
{
# error: "procmail: Extraneous locallockfile ignored"
# - This lock file will be ignored
# - If the recipes inside the braces try to use file.lck
# as a lockfile, then you'll have a deadlock situation.

:0 :
/tmp/tmp.mbx
}

Let me also explain why the `w' is so important. Notice, that the
two here are equivalent. The `W' here is implicit. _NOTE_: this is
only true on the recipe that opens a nested block. On a recipe with
a program, forward, or delivery action, `W'' is different from `w'
is different from missing both.

:0 c: file$LOCKEXT :0 Wc: file$LOCKEXT
{ ... } { ... }

To quote the comment in source code, "try and protect the user from
his blissful ignorance". The parent will always wait for the cloned
child to exit when a lockfile is involved. The only question is
whether or not it should be logged. If you want failure of the
cloned child to be logged, then you should use the `w' flag, ala:

:0 wc: file$LOCKEXT
{ ... }

A local lockfile can be used to lock a clone; the parent procmail
will remove it when the clone exits (thus it serves as a global
lockfile for the clone). If the braced block does not launch a
clone, asking for a local lockfile generates an error.

6.12 Global lockfile

[david] If you want to block everything while the recipe runs, even
during the _conditions_, use global lock. For example in this
construct the `formail' which updates the message-id cache file
must be protected with a global lockfile.

MID_CACHE_LEN = 8192
MID_CACHE_FILE = $PMSRC/msgid.cache
MID_CACHE_LOCK = $PMSRC/msgid.cache$LOCKEXT

LOCKFILE = $MID_CACHE_LOCK

:0
* ^Message-ID:
* ? $FORMAIL -D $MID_CACHE_LEN $MID_CACHE_FILE
{
LOG = "dupecheck: discarded $MESSAGEID from $FROM $NL"

:0 # no lockfile !
$DUPLICATE_MBOX
}

LOCKFILE # kill variable

You cannot use local lockfile as below:

:0 : $MID_CACHE_FILE$LOCKEXT
* ^Message-ID:
* ? $FORMAIL -D $MID_CACHE_LEN $MID_CACHE_FILE

because the local lockfile named on the flag line will be created
only if the conditions have matched and the action is attempted.

One more note: watch carefully, that there is _no_ `:' lock when
delivering to `DUPLICATE_MBOX' because the outer global lockfile
already prevents all other procmail instances from executing this
part of the recipe.

6.13 Gee, where do I put all those ! * $ ??

Ahem. I can't tell you exactly what to do or how to write your own
procmail recipes, but I can tell how I'm writing them. Here is my
condition line token order:

* $ ! ? BH VAR ?? test

That won't say much unless I give you something to compare with.
Here is one perfectly valid rule, but not my style

:0
*$ ^Subject:.*$VAR
*! ^From:.*some
*B ! ?? match-the-string-in-body
*$? $IS_EXIST $FILE
*VARIABLE ?? set

I prefer lining up things in the condition lines. The first column is
reserved for dollar sign, the second for *not* operator and so on.
The important thing is that I can see at a glance if I have set the
variable expansion dollar in the line (leftmost).

:0
*$ ^Subject:.*$VAR
* ! ^From:.*some
* ! B ?? match-the-string-in-body
*$ ? $IS_EXIST $FILE
* VARIABLE ?? set

6.14 Sending automatic reply, use X-loop header

Do not send automatic reply without checking "! ^FROM_DAEMON"
condition and always include `X-Loop' header and check its existence
to prevent mail loops

:0
* conditions-for-auto-reply
*$ ! ^$MY_XLOOP
* ! ^FROM_DAEMON
| $FORMAIL -A "$MY-XLOOP" ...other-headers...

6.15 Avoid extra shell layer (check command for SHELLMETAS)

[dan] It is very important to study your shell command calls and try to
save the overload of the extra layer of shell. It may be extra work
once when you write your rcfile but it saves effort on each piece of
arriving email. When procmail sees a character from `SHELLMETAS', it
runs

# Default SHELLMETAS: &|<>~;?*[
# Default $SHELLFLAGS: -c

% $SHELL $SHELLFLAGS "command -opts args"

instead of

% command -opts args

That is because procmail's ability to invoke other programs does not
include filename globbing ([, *, ?), backgrounding (&), piping
(|), succession (;), nor conditional succession (&&, ||). If it
sees any of those characters (before expanding variables), it hands the
job over to a shell.

Sometimes those characters appear in arguments to a command without
having their shell meta meaning and procmail really could invoke the
command directly without the shell. You can see the distinction in a
verbose logfile: if procmail runs the command itself, it logs

Executing "command,-opts,args"

with a comma between each positional parameter, but if it calls a
shell, the original spacing from the rcfile appears unchanged in
the logfile:

Executing "command -opts args"

So, if you know you won't be needing shell expansion, wrap your
shell calls with this:

savedMetas = $SHELLMETAS
SHELLMETAS # Kill variable

..command that does not need shell expansion features..

SHELLMETAS = $savedMetas

6.16 Think what shell commands you use

For every message, procmail launches the processes you have put into
your .procmailrc. If you haven't paid attention to optimization
before, now it's serious time to take a magnifying glass and check
every recipe and the processes in them. When you write you private
shell scripts, the performance hit is not so important, but for
mail delivery, the matter is totally different. First, let's see
some programs and sizes: The following is from HP-UX 10, where the
binaries seem to include debug and symbol table code.

131072 Aug 21 1996 /usr/bin/awk
196608 Oct 1 1996 /usr/bin/sort
245760 Jun 10 1996 /usr/bin/grep
262144 Jun 10 1996 /usr/bin/sed
303552 Dec 7 1995 /usr/local/bin/gawk
544768 Jun 10 1996 /usr/contrib/bin/perl [perl 4.36]
822232 Aug 25 13:58 /opt/local/bin/perl5.00401

text data bss
awk: 72727 + 51316 + 15317 = 139360
sort: 173225 + 18496 + 183076 = 374797
sed: 237248 + 16992 + 56252 = 310492
grep: 221591 + 16176 + 53816 = 291583
perl4: 502220 + 36044 + 65632 = 603896
perl5: 633812 + 69612 + 2385 = 705809
gawk: 160018 + 5264 + 7168 = 172450

The binary siszes above are not the typical cases: these are from
another system

4 Sep 28 14:25 /usr/local/bin/awk -> gawk
32768 Nov 16 1996 /usr/bin/grep
49152 Nov 16 1996 /usr/bin/sed
114688 Oct 20 1996 /usr/local/contrib/gnu/bin/grep
155648 Nov 16 1996 /usr/bin/awk
155648 Nov 16 1996 /usr/bin/nawk
221184 Nov 16 1996 /usr/bin/gawk
311296 Jan 27 1997 /usr/local/bin/gawk
958464 Nov 2 16:34 /usr/local/contrib/bin/perl
1196032 Sep 14 1996 /usr/local/bin/perl

Stan Ryckman wants you to know that:

Comparing byte sizes on disk means nothing here... these
things may or may not have been stripped. Any symbol tables included
in the byte counts you see above won't affect process start-up time.

The `size' command will give a better handle on what will be needed
in starting a process. The three segments may each have their own
overhead, though, and the relative contributions of those segments
to startup time may well be system-dependent.

Hm. Can we draw some conclusion? Not anything definitive, but at
least something:

o While sed and grep may be bigger than awk in some systems, this
is an exception. They are usually much smaller and fast to use.
o But complex commands that would require many processes to be
chained together, like `grep -v | grep | sed' combination could
be usually accomplished with one awk call. Ask somewhere how to
do it with awk if you don't know the language, it's quite alike
perl
o Don't use anything else but standard awk, _gawk_ and _nawk_
are bigger and may not be found on all systems.
o Avoid perl at all costs; it's many times (6) bigger than awk.
Perl is slow-to start up, due to intermediate compilation
process at startup.
o Remember that if procmail is running in a dedicated mail host, it
probably doesn't even have any goodies installed, just the boring
standard versions; which may not be even the same as what you see
on current host. e.g. My mail host is running HPUX-9, while my
login is on HPUX-10. They have very different installations.

Here are some more programs. Don't even think of extracting fields with
`grep' or `awk', like "grep Subject", because `formail' is
much smaller and more optimized for tasks like that.

37007 Sep 5 15:53 /usr/local/bin/formail # 3.11pre7
28672 Jun 10 1996 /usr/bin/tr
20480 Jun 10 1996 /usr/bin/tail
20480 Jun 10 1996 /usr/bin/cat
20480 Sep 26 1996 /usr/bin/expr
16384 Jun 10 1996 /usr/bin/head
16384 Jun 10 1996 /usr/bin/cut
16384 Jun 10 1996 /usr/bin/date
16384 Jun 10 1996 /usr/bin/uniq
16384 Jun 10 1996 /usr/bin/wc
12288 Jun 10 1996 /usr/bin/echo

6.17 Using absolute paths when calling a shell program

Shell programmers know that if you use absolute path when you call
the executable, shell doesn't have to search through long list of
directories in $PATH. This may speed up shell scripts remarkably.
The correct way to use such an optimization is to define variables to
those programs.

Hm, should you use such optimization in your procmail code? That's
two folded question and I....would say yes and no. How many
shell calls do you have? Do you use grep or formail a lot? Then you
could optimize these calls. To be portable, define variables for
executables:

# perhaps defined in separate INCLUDERC
#
# INCLUDERC = $PMSRC/pm-mydefaults.rc

FORMAIL = /usr/local/bin/formail
GREP = /bin/grep
DATE = /bin/date

:0 fhw
| $FORMAIL -rt

And when you port your .procmailrc to different environment that
has different paths, you could use this recipe in addition to one
just mentioned above:

FORMAIL = ...as above

:0
* HOST ?? second-host
{
# In this host the paths are different. Reset.

$FORMAIL = "formail"
$GREP = "grep"
$DATE = "date"
}

6.18 Disabling a recipe temporarily

If you have a recipe that you would like to disable for a while,
there is an easy way. Just add the "false" condition line before
any other conditions. The "!" also nicely visually flags that
"this recipe is NOT used".

# This recipe stops at "!" and doesn't get past it.

:0
* !
* condition
* condition
{
...
}

6.19 Keep message backup, no matter what

It's good to have a safety measure in your .procmailrc.
Although you are an expert and have checked your recipes 10 times,
there is still a chance that something breaks. One morning, when you
browse your *BIFF* reminder log; you notice "Hm, there is that
interesting message but it was not filed, where is it?". And when
you go to study the procmail logs (you do keep the log going all
the time) and it hits you: "Gosh; a mistake in my script! Message was
fed to malicious pipe and I had that `i' flag there... *sniff*".
And you greatly regret you didn't back up the message in the first
place.

So, before your procmail does anything to your message, put the
message into some folder which is regularly expired. For example I
use Emacs Gnus to handle the expiring. One could also use a cron job
instead. Then you can relax knowing your email is safe.

SPOOL = $HOME/Mail/spool

# Backup storage
# - This could be directory too. In that case you could use
# cron job to expire old messages at regular intervals
# - For once a day expiration, see procmail module list
# and pm-jacron.rc

BUP_SPOOL = $SPOOL/junk.bup.spool

:0 c:
$BUP_SPOOL

Naturally you can filter out mailing list messages from the backup,
because losing one or two (hundred) of them may not be that serious.
Maybe you could use two backup spools, one for mailing lists and the
other for your non-list messages.

:0 c:
* ! mailing-list1|mailing-list2
$BUP_SPOOL

If you have the date variables set up as described below, you
could also create a backup folder per day:

$BUP_SPOOL = $SPOOL/$YYYY$MM$DD

This makes it very easy to delete backups that are older than
a given number of days, either manually or through a cron job.

6.20 Order of the procmail recipes

When you start writing a lot of procmail recipes, you soon find out
that it matters a great deal in which order your put your recipes. When
each group of recipes starts growing too big, it's good practice to
move each group to a separate rc file. Here is one recommended order:

- backup important messages
- cron-subroutine
- handle duplicate messages
- handle DAEMON MESSAGES
- handle plus addressed message (RFC plus or sendmail plus addresses)
- handle server requests (file server, ping responder...)

- drop MAILING LIST messages

- send possible vacation replies only after all above
- apply kill file
- detect mime and format or modify the message body
- save private messages

- and last: FILTER UBE.

The backup, cron and duplicate handling go naturally to the beginning
of your .procmailrc. Next comes a grey area where Daemon, plus handling
(#REF #using_rfc_comment_trick_for; Note plus;); and server messages can
be put.

Mailing lists should be handled as early as possible, but after the
server messages, because you want your services handled first.

Do not send vacation replies before you have handled mailing lists
to prevent annoying vacation replies to lists.

After that you are left with "known" private messages and those of
unknown origin. A kill file (to block based on sender) for rapid
spammers, who send you message or several per day need to be
checked before checking other messages.

Last but not least: Put your UBE checkers to the end to avoid mishits
of valid mail. DO NOT SEND AUTOMATIC COMPLAINT BACK. Drop the
UBE to a folder, manually select the messages that need actions
and send message to postmasters in the Received chain explaining that
their mail relay has been hijacked.

7.0 Procmail flags

7.1 The order of the flags

Order does not matter of course, but here is one stylistic suggestion.
The idea here is that the most important flags are put to the left,
like giving priority 1 for `aAeE', which affect the recipe
immediately. Priority 2 has been given to flag `f', which tells if
a recipe filters something. Also (h)eader and (b)ody should
immediately follow `f', this is considered priority 3. In the
middle there are other flags, and last flag is `c', which ends the
recipe, or allows it to continue. In addition according to [david]:
"...I'm quite sure that putting anything other than the opening
colon and the number to the left of `AaEe' will cause an
error."

:0 aAeE HBD fhb wWir c: LOCKFILE
| | | | |
| | | | (c)ontinue or (c)lone flag last.
| | | (w)ait and other flags
| | (f)ilter flag and to filter what: (h)ead or (b)ody
| (H)eader and (B)ody match, possibly case sensitive (D)
The `process' flags first. (A)nd or (E)lse recipe

You can write the flags side by side

:0Afhw:$MYLOCK$LOCKEXT

Or, as I prefer, leave flags in their own slot for more
distinctive separation. Note that $LOCKEXT must be next to $MYLOCK,
because it contains string ".lock".

:0 A HB fhw: $MYLOCK$LOCKEXT

7.2 Flag w and recipe with |

[alan] If the filter program exits with a 0 status (0 == okay), then
procmail will replace the original input body with the output of the
filter program. If the filter program exits with anything but zero,
procmail will report an "error" to the log, and "recover" the input
(not filter it)

[david] I am very sure that that's the case _only_ if you have the
`w' or `W' flag on the filtering recipe. Without `w' or `W',
procmail won't care about a bad exit status from the filter and will
replace the filtered portion with whatever standard output the
filter produced. It may still report an error to the log but it
won't recover the previous text. This, for example, will destroy the
body of a message, even without `i':

:0 fb
| false

With this, however, procmail will recover the original body:

:0 fbW # same results even if we add `i'
| false

[stephen] No, not on all occasions. Procmail will not care about the
exitcode here. However, if procmail detects a write error, it *will*
recover (because of the missing `i' flag). Procmail will only detect
a write error in such a case if the mail is long enough and does not
fit in the pipe buffer that's in the kernel (typically 10KB).

7.3 Flag w, lockfile and recipe with |

[manual] In order to make sure the lockfile is not removed until the
pipe has finished, you have to specify option `w' otherwise the
lockfile would be removed as soon as the pipe has accepted the
mail. So if you see anything that looks like ">" or ">>" in your
recipe, then that should immediately ring your bells. immediately
check that you have included the `w' flag _and_ the lockfile `:'.

:0 hwc: headc$LOCKEXT
* !^FROM_MAILER
| uncompress headc.Z; cat >> headc; compress headc

7.4 Flag f and w together

The w tells Procmail to hang around and wait for the script to
finish. [Wouldn't you think this ought to be implied by the f
already?]

[david] Of course the `f' flag is enough to make procmail wait for
the filter to finish, but the `w' means something more: to wait to
learn the exit code of the filtering command. If sed fails with a
syntax error and gives no output, without `W' or `w' procmail would
happily accept the null output as the results of the filter and
go on reading recipes for the now body-less message. On the other
hand, with `W' or `w' sed will respond to a non-zero exit code by
recovering the unfiltered text.

7.5 Flags h and b

[david] `hb' is the default; you need to use `h' only when you
don't want `b' or vice versa. You can think of it this way: `h'
means "lose the body" and `b' means "lose the header," but the two
together cancel each other out.

[philip] `hb' (feeding whole message) is the default for actions.
You need to specify `h' without `b' if you want the action applied
only to the head. `H' is the default for conditions. You need to
specify `HB' or `BH' if you want to test a condition against the
entire message.

7.6 Flag h and sinking to /dev/null

When you drop something to /dev/null, use the h flag so that
procmail does not unnecessarily try to feed whole message there.

:0 h
* condition
/dev/null

[philip] Procmail knows that it shouldn't create a locallock on
/dev/null and that it shouldn't kernel lock /dev/null, and it knows
to write it "raw" (no "From " escaping or appended newline). This
means that procmail simply opens /dev/null, does its write with
one system call, and closes it.

I'm not sure if adding the `h' flag makes a real difference on
modern UNIX kernels. I suppose it depends on how optimized the
write() data is and in particular, whether a user-space to
kernel-space copy is _required_, or whether it's delayed. If it's
delayed then the code for handling /dev/null would presumably not
do it, and the size of the write wouldn't actually matter.

7.7 Flag i and pipe flag f

Flag `i' is useless in mailbox deliveries.

[faq] The following will work some of the time, when the message is
short enough, but that's a coincidence. With a longer message,
though, Unix starts paying attention to what is happening, because
it will have to buffer some of the data, and then when the buffered
data is never read, an error occurs. The error is passed back to
Procmail, and Procmail tries to be nice and give you back your
original message as it was before this malicious program truncated
it. Never mind that in this case you wanted to truncate the
data. Anyway, the fix is easy: Just add an `:i' flag to the recipe
( `:0fbwi' instead of `:0fbw') to make Procmail ignore the error.

:0 fbw
* condition
| malicious-pipe

[dan] here's why the `i' flag is needed (courtesy of Stephan): You
told procmail to filter the entire mail (header and body), so it
does and it attempts to write out header and body to the filter.
Then procmail notices that not the entire body is being consumed.
Procmail, being rather paranoid when it comes to delivery of mail
assumes something went wrong and considers this a failure of the
filter.

:0 fbwi
| head -2

7.8 Flag r

[philip] Procmail automatically turns on the `r' (raw mode) flag for
deliveries to /dev/null, so there's no need to do it yourself.

:0 r # you can leave out the `r'
* condition
/dev/null

[david] You can use the `r' flag (for raw mode) on every recipe
where you do not want a From_ line added. I'm assuming that there
isn't one already there; the `r' flag keeps procmail from making
sure that there are a From_ line at the top and a blank line at the
bottom, but it will not make procmail remove them if they are
already present. Also, be careful to use the `-f' option on all
calls to formail so that formail won't add a From_ line.

Someone who didn't need From_ lines -- I forget who -- found it
annoying to put `r' onto every recipe and altered the source to
prevent procmail from adding From_ lines at all, ever. I think a
better idea would be a procmailrc Boolean to enable or disable them
for all recipes without affecting other users. (Then perhaps we'd
need a reverse `r' flag to undo raw mode for one recipe at a time?)

7.9 Flag c's background

...Interesting. My vision of `c' is to think of CONTINUE
with message processing afterwards even if conditions matched.

[david] Precisely: when you have braces, thinking "continue"
instead of "copy" or "clone" can get you into trouble.

Early versions of procmail, before braces and before cloning,
called the `c' flag "continue" in their documentation; I think it
is still called that in the source.

When Stephen introduced braces (but not cloning at this point), it
was of course implicit that an action line of "{" was
non-delivering, and a `c' was extraneous. People put c's there
because they wanted procmail to continue to the recipes inside the
braces on a match, and procmail brushed it off with an "extraneous
c-flag" warning. No harm done.

When Stephen introduced cloning, though, I was rather upset that he
was giving double duty to `c' instead of introducing something new
like `C' for it, especially because people who absolutely wanted no
clone but intended the recipes inside the braces to run in the same
invocation of procmail as everything else were mistakenly putting
c's on their braces to make sure procmail would "continue". People
would (and did) get double deliveries.

Roman Czyborra, though, said that if you consider `c' to stand for
"copy", that covers both uses of `c': provide a copy to a simple
recipe or, if there are braces, to a clone procmail that will
handle the recipes inside the braces. Stephen agreed and changed
the documentation accordingly.

Longtime users of procmail and people who read old docs may still
think of it as "continue", but since the introduction of clones,
that is not a good way to look at it. "Copy" is much safer.

7.10 Flag c before nested block forks a child

[alan] The combination of a nested block and the `c' flag causes
procmail to fork a child process for the nested block, while the
parent skips over it and continues on. The child process doesn't
necessarily stop unless a *delivering* recipe (without the `c' flag)
action succeeds.

7.11 Flag c and understanding possible forking penalty

... I run shell commands that need not to be serialized, so
instead of doing the standard way:

:0 hic # nbr.1 / standard way
| command

I assume I can avoid the extra fork caused by (c)lone flag
altogether by using these. Any difference between these two?

:0 # nbr.2 / alternative
* ? command
{ } # ...No-op, Procmail syntax requires this

dummy = `command` # nbr.3 / alternative

[philip] There is a misunderstanding here. Let me clarify:

Procmail only forks a full-blown clone on a recipe with the 'c'
flag whose action is a nested block.

If it's a simple mailbox deliver, pipe, or forward action then
procmail does not fork a 'clone' (for pipe and forward actions
procmail does have to fork, but only so it can execute the
action). `nbr.1' and `nbr.2' take the same number of forks to
execute. They also take the same effective number of writes
(in case you're concerned about that). The latter also
requires that procmail wait for the command to finish.

`nbr.3' is worse than the above two, as procmail has to not
only wait for the command to complete but also save the output
into the named variable.

7.12 Flags before nested block

Given the following recipe, let's examine the flag part

:0 $FLAGS
{
do-something
}

[david] `HB' `AaEe' and `D' affect the conditions and
thus are meaningful when the action is to open a brace. `HB' and
`D' would be meaningless, of course, on any unconditional recipe, but
they should not cause error messages.

Generally, flags that affect actions are invalid there, and `bhfi'
and `r' always are, but the others are partial exceptions: if you
are using `c' to launch a clone, then `w' `W' and a local lockfile can
be meaningful. If there is no `c', then `w' `W' and a local lockfile
are invalid at the opening of a braced block.

7.13 Flags aAeE tutorial

[david] `AaEe' are mutually exclusive and no more than one should
ever appear on a single recipe. [philip] Actually, this is not
true. e does not work with `E' or `a' (and procmail gives a warning
if you try), and `A' is redundant if a is given, but at least some
of the other combination make sense and work.

o *A* = try this recipe if the conditions succeeded on the most
recent recipe at that nesting level that did not itself have an
A nor an a
o *a* = same as `A', but moreover the action must have succeeded
on the most recently tried recipe at that nesting level
o *e* = Almost like `A', try this recipe if the conditions matched
but the action failed on the most recently tried (not skipped)
recipe at this nesting level. universe, `e' is the opposite of `a'.
`e' only looks backwards past `E' recipes that were skipped
because of their `E'. It doesn't care whether a previous recipe
had an `A' or `a' flag.
o *E* = try this recipe if the conditions have failed on the most
recent recipe at that nesting level that did not have an `E' and
on since then every recipe at that level that did have an `E';
essentially opposite of `A'

These mnemonics might help:

o *A:* if you did the recipe at the start of the chain, try this one
(A)lso
o *a:* if the last action at that nesting level was (a)ccomplished)
o *e:* if the last action at that nesting level (e)rred
o *E:* (E)lse because the conditions down the chain so far have not
matched. Or "try this recipe unless the last tried recipe matched".

# [philip] demonstrates `e'

:0 : # match, but action fails
/etc/hosts/foo

:0 A # no match
* -1^0
/dev/null

:0 e # this is skipped because the last tried recipe didn't match
{
...whatever
}

How they interact with one another when used consecutively has not
been fully tested to my knowledge. Consider this:

:0
* conditions
non-delivering-action1

:0 a
action2

:0 e
action3

Is action3 done if action2 failed or if action1 failed (or perhaps
in both situations)? [philip] Action 3 is only done if action2 failed.

If the answer is action2, does this work to get action3 done if
action1 failed? I think it does, but does it also run action3 if
the conditions didn't match on the first recipe? [philip] Yes, and
yes.

:0 # [david]
* conditions
non-delivering action1

:0a
action2

:0E
action3

[philip] If that's not what you want, combine some flags:

:0
* conditions
non-delivering action1

:0 Ae
action3

:0 a
action2

If the conditions match, action1 will be executed. action3 will
then execute if action1 failed, otherwise action2 will be executed
[if action1 succeeded].

[david] I know what this structure does because I use it:

:0
* conditions
non-delivering action1
:0A
action2

:0E
non-delivering action3
:0A
action 4

If the conditions match, action1 and action2 are performed and
action4 is not (of course action3 is not either), even if action2
is non-delivering; if they fail, action3 and action4 are performed.
The `A' on the fourth recipe refers back to the third and no farther.
But I don't know about this:

:0
* conditions
non-delivering action1
:0A
* more conditions
action2

:0E
non-delivering action3
:0A
action 4

Now, suppose the conditions on the first recipe match but those on
the second recipe do not match. Would the third recipe (and thus
the fourth one) be attempted? I would expect so. [philip] Yes. The
last tried recipe didn't match, therefore the `E' flag will be
triggered.

If that isn't what you want, you can prevent it this way:

:0
* conditions
{
:0
non-delivering-action1

:0
* more-conditions
action2
}

:0 E # ignores mismatch inside braces, looks only at same level
non-delivering action3

:0 A
action4

If that is what you want, you can be positive this way:

# if action2 is non-delivering or vulnerable to error that
# would cause fall-through

DID2 # Kill variable

:0
* conditions
non-delivering-action1

:0 A
action3

:0
* ! DID2 ?? (.)
non-delivering-action3

:0 A
action4

# if action2 is delivering and sure to succeed
:0
* conditions
non-delivering-action1

:0 A
* more-conditions
action2

:0
non-delivering-action3

:0 A
action4

[philip] or those who are interested, I'll note that there are only
3 combinations of the `a', `A', `e', and `E' flags that aren't
either illegal or redundant. They are `Ae', `aE', and `AE'. I've
shown a use for `Ae' up above. Here's an example of `AE':

:0
* condition1
non-delivering action1

:0 A
* condition2
non-delivering action2

:0 AE
action3

action3 will only be executed if condition1 matched but condition2
didn't match. Without the A flag, action3 would be executed if
either of them failed. This can also be done with a instead of A
with analogous results.

Procmail's "flow-control" flags may not be particularly easy to
describe in straight terms (and this can all be made more
complicated by throwing in a more varied mix of delivering vs
non-delivering recipes), but I've found that it usually does what I
expect it to do, and when it doesn't or I'm in doubt or I want to
be particularly clear, I can always fallback to doing it explicitly
via nesting blocks. Pick your poison...

8.0 Matching and regexps (regular expressions)

8.1 Philosophy of abstraction in regexps

Here are two ways to view or write regexps. Make up your own mind.

People who are in favor of writing pure native regexps in the
recipes:

[ ]<[ ]*("([^"\]|\\.)*"|[-!#-'*+/-9=?A-Z^-~]+)... # "

o I'm not planning on "maintaining" that code, as the syntax for
XXX will not ever change <>>
o I some how doubt that anyone else will change that regexp more than
trivially
o If none of your other regexps use the categorical variables, and
you're not changing the regexp, then what's the point?
The variablized version will be slower, and will clutter the
environment with subprocesses.

Where someone that immediately wants to abstract things says
(This is from philip's great Message-Id matching recipe)

dq = '"' # (literal) double-quote
bw = "\\" # (literal) backwhack
atom = "[-!#-'*+/-9=?A-Z^-~]+"
word = "($atom|$dq([^$dq\]|$bw.)*$dq)'
local_part = "$word($s\.$s$word)*"

$s<$s$local_part... # ignore comment here

....abstraction: It makes code clearer when you break it
to manageable parts, which possibly surfaces reusable parts. It
also makes thing look simpler, and enables even novices
to understand what's going on there. After we're not
connected to the net anymore, others could possibly understand
it too.

So, naturally we can't agree with any of the previously mentioned
arguments presented for keeping regexp "in pure native format".

o Although you won't maintain it, it's an example for others. What
you post first, people will save it to their mailboxes and
circulate elsewhere in the net: "Hey, I've saved this, try it"
o You can write cryptic regexps or break them into parts where
the whole looks much simpler. Consider novice's welfare :-)
This has nothing to do with the "It never changes in my lifetime".
o The speed penalty imposed by additional variables is not
something we can measure in practice. CPU won't even hiccup.
An extra `formail' call in your recipes is 10x as expensive as
100 variables. (I don't know how to measure that, but launching
a shell and creating a process is a much more expensive task).
o Cluttering the env process? C'm on. That won't matter either.
No outside process uses lowercase environment names, or then it
must be real special program. So called "cluttering" of
environment space is also no-issue. CPU won't even get a hiccup
for that.

8.2 Matches are not case-sensitive

Okay, okay; if you read the manual you knew that already. But
sometimes someone with years of experience with Unix may take it for
granted that procmail would be case-sensitive as the rest of the
unix tools are. Use the `D' flag to turn on case-sensitivity.

8.3 Procmail uses multiline matches

Procmail uses multiline matches by default. This means that ^ and $
match a newline, even in the middle of a regexp. Now you know this,
you can easily interpret e.g. `$[^>]' as: `a newline followed by a
line not starting with a `>'.

If you put a '$' after the '\/' match token then procmail will
include the matched newline if there's one there. Solution? Don't
put a dollar sign there unless you really want a newline, use period
that matches all but newline:

:0 B
* ^Search-string: \/.+

8.4 Headers are folded before matching

If you have a header that continues on separate lines, you don't have
to worry about the linefeeds. Procmail silently folds the header onto
one line, before matching it

Received: from unknown (HELO Desktop01) (208.11.179.72) by
palm.bythehand.net with SMTP; 4 Dec 1997 23:29:09 -0000

:0 # note, match on continuation line
* ^Received:.*bythehand\.

8.5 Improving Space-Tab syndrome

Procmail doesn't know about standard escape codes like `\t' and `\n'
or [\0x00-\0x133]:

# Not what you think # You have to write: space + tab
[ \t] [ ]

But using the space+tab is not very readable and it's a very error
prone construct. I suggest using the following to improve the
readability:

WSPC = " " # whitespace = space + tab
SPC = "[$WSPC]" # regexp whitespace, the short name
# SPC was chosen because you use this
# a lot in condition lines.
NSPC = "[^$WSPC]" # negation of whitespace

# match anything except space and tab

*$ var ?? $NSPC

# match anything ecxept space and tab and newline

*$ ! var ?? ($SPC|$)

But you cannot use newline inside brackets.

WSPCL = " "'
'

# Won't work although WSPCL definition is correct.

*$ var ?? [$WSPCL]

Instead use variable syntax:

SPCNL = "($SPC|$)" # space + tab + newline

If you absolutely need a range of characters, see if you have `echo'
command in your system to define variables like this:

NUL_CHAR = `echo \\00`
DEL_CHAR = `echo \\0177`
REGEXP_NON_7BIT = "[^$NUL_CHAR-$DEL_CHAR]"

8.6 Handling exclamation character

[philip] you do need the first backslash, to keep procmail from
considering the backslash as a request to invert the sense of the
match. For example, these two conditions are equivalent:

* ! 200^1 foo
* 200^1 ! foo

Therefore, a leading '!' must either be backslashed, enclosed in
either parens or brackets (I suspect that parens would be more
efficient), or prefaced with an empty pair of parens. I would
recommend writing the condition with one of these:

* 200^1 \!!!!
* 200^1 ()!!!!
* 200^1 (!!!!)

8.7 Rules for generating a character class

In a "character class" (things between "[" and "]"), metacharacters
don't need to be escaped. Well, a backslash is an exception.
e.g. [$[^\\] would match any one of the literal characters dollar,
opening bracket, caret, and backslash.

o To match "])" use [])]
o To match "[(" use [[)]
o To include a literal ^ must not be first
o To include a literal - must be first, last or \-
o To include a literal \ you must use \\

o To include a literal ] must be first
o To include a literal [ ( ) or $ just use it anywhere

[elijah] If you are inverting a character class "first" means just
after the(^). So the character class that contains everything but ]
^ and - must look like this:

[^]^-]

[david] What if I want literal $ inside bracket? A $ inside
brackets, unless it begins a variable name and the "$" modifier is
on, always means a literal dollar sign. It cannot mean a newline if
it appears inside brackets. A good way to keep it exempt from "$"
interpretation is to put it last inside the brackets (unless one
also need to include a literal hyphen and one can't put the hyphen
first; then you'll need to escape the dollar sign with a backslash
and put the hyphen last -- well, you could alternatively escape the
hyphen, I guess), because procmail knows that "$]" cannot possibly
be a reference to a variable.

General guideline:

o ($) always matches a newline, with or without "$" interpretation;
o [$] always matches a dollar sign, with or w/o "$" interpretation;

8.8 Matching space at the end of condition

[david] If you need to have tab or space at the end of condition line
you can use these:

* rest of string .*
* rest of string[ ]
* (rest of string )
* rest of string ()
* rest of string( ) # I prefer this one

[philip] From my looking at the source, the last two should be
equal in efficiency, and except for a trace difference in regcomp
time, should match at the same speed as a solitary trailing blank.
The character class version [ ] will be slower.

Of course, I suspect that neither you nor your sysadmin will ever
notice the difference in speed, and given that 99% of all systems are
I/O bound and not CPU bound, the system is incredibly unlikely to
notice either. I can't complain though, as I also go to various
extremes to seek out every last bit of possible performance. Ah well.
The first one would be slower yet, though perhaps no slower than the
bracket form.

8.9 Beware leading backslash

I am trying to come up with a procmail recipe that among other
things should have the condition 'body does not contain a
particular word'. Here is what I tried:

* ! B ?? \

[david] You have fallen into the leading backslash problem, If the
first character of a regexp is a backslash, procmail takes it as "end
of leading whitespace" and strips it. What you coded means "a less-than
sign, then the word, then any non-word character." (It also prevents
the less-than sign from being taken as a size operator.) Unless the
non-word character immediately to the left of the word was a less-than
sign, that regexp would fail (and thus the condition would pass). Try
this:

* ! B ?? ()\

This would work too:

* ! B ?? \\

but in a casual reading it would look like "literal backslash,
less-than sign, the word, word boundary character," so we on the list
generally recommend the empty parentheses.

Do note that the difference in meaning of \< and \> in procmail (where
they must match a non-word character) from their meaning in perl and
egrep (where they match the zero-width transition into and out of a
word respectively) does not come into play here. Because procmail's \<
and \> can match newlines (both real and putative), it rarely is a
factor. It's a problem only when a single character has to serve both
as the ending boundary of one word an also the opening boundary of
another. Well, it's also a problem when you have one as the last
character to the right of \/, but that's easily solved.

8.10 Correct use of TO Macro

o `TO' is not a normal regular expression; it is a special
procmail expression that is designed to catch any destination
specification. For details, see the miscellaneous section of
the `procmailrc(5)' man pages.
o Prefer `TO_' instead of `TO' if you have new procmail. `TO_' is
better because TO used to be too loose
o Please remember to write `^TO', with the anchor in it.
o Do not put a space between the caret (^) and the word `TO' in
`^TO'.
o Do not put a space between the `^TO' and the text that you are
matching on; it must be `^TOtext' If this bothers you, you can
use `TO()text' instead to get better separation of text.
o Both letters in `TO' must be capitalized.

8.11 Procmail's regexp engine

[philip] procmail's regexp engine has no special optimization
for anchoring against the beginning of the line. Most program that
have such an optimization have it because they need the line
distinction for other reasons (for example, grep by default prints
the entire line containing a match). Procmail has no such other
reason, so it treats newline like any other plain character in the
regexp. There should be no speed difference as long as procmail
can say: "the first character I see must be a 'foo'". Note that
case insensitivity is handled by making everything lowercase, so a
letter being first doesn't bring in the spectre of character-classes
or anything like that.

.> recipe may have just changed the size of the head, procmail
.> cannot keep a byte-count pointer nor a line-count pointer to
.> where the body begins but must scan through the head to find the
.> blank line at the neck before it begins a body search.

Procmail does this when it reads in the head, not when it goes to
search the body, so that cost can't be avoided. Let me repeat; that
searching the body is no slower than searching the header, if we
forget the minimum impact of the size of these two.

8.12 Procmail and egrep differences

[By david]

o ^ and $ are non-zero-width and anchor to real or putative
newlines (rather than to the zero-width start and end of a line);
o An initial ^^ or a final ^^ anchors to the opening or closing
putative newline respectively;
o ^ and $ in the middle of a procmailrc regexp match to an embedded
newline (and must be escaped to match to a caret or a dollar sign);
o \< and \> are non-zero-width and match to a character that
wouldn't be in a word (or to a real or putative newline) [rather
than to the zero-width transition into or out of a word]; it
always matches one non-word character. It will fail when there is
no whitespace after the colon. This is rather pathological but
still perfectly compliant with RFC822. For this reason,
you should use (.*\<)? instead of just .*\< after the colon that
terminates a header field name:

^Subject:.*\ # Wrong
^Subject:(.*\<)?humor\> # Right, notice ?

o *, ?, and + in the absence of \/ are stingy rather than greedy,
and that generally won't matter, but in the presence of \/ they
are stingy to the left of \/ and greedy to the right of \/,
while in most applications the leftmost wildcard on a line is
the greediest and greed decreases from left to right.

8.13 Understanding procmail's minimal matching (stingy vs. greedy)

...I want to have a procmail recipe that will save certain mail to
folders where the folder name (always a number) is specified in
the subject.

:0 :
* ^Subject: *\/[0-9]*
$HOME/Mail/$MATCH

[philip]...and this won't quite work. For a subject with a space
after the tab, the '*' on the left hand side will be matched
minimally (zero times), and then the stuff on the right hand side
will be matched maximally, but starting at the space still, which
will match nothing. This is a case were procmail's minimal matching
can cause massive confusion and frustration. The solution is
usually the following:

FORCE THE RIGHT HAND SIDE TO MATCH AT LEAST ONE CHARACTER

By Changing the recipe to:

:0 :
* ^Subject: *\/[0-9]+
$HOME/folders/$MATCH

it'll work, because then the left hand side will have to match all
the way up to the first digit (but not the digit itself). If you
follow the rule in caps then you'll almost always be able to ignore
procmail's weirdness in this area.

[david] And examine how procmail matches "Subject: Keywords 9999"

* ^Subject:.*Keywords.*\/[0-9]*

procmail: Match on "^Subject:.*Keywords.*\/[0-9]*"
procmail: Matched ""

The right side was as greedy as it could be; the problem is that we
seem to expect greed on the left as well. MATCH is set to null, in
contrary to our expectation. It is not a bug but rather a frequently
misunderstood effect of the way extraction is advertised to operate.

Remember that only the right side is greedy; the left side is
stingy, and left-side stinginess takes precedence over right-side
greed.

Extraction is implemented this way: the entire expression, left and
right, is pinned to the shortest possible match; then the division
mark is placed and the right side is repinned to the longest
possible match starting at the division. The tricky part is to
remember that the division is marked during the stingy stage.

If the expression is

^Subject:.*Keywords.*\/[0-9]*

and the text is

Subject:Keywords9999

then the shortest possible match to the entirety is

Subject:Keywords

because ".*" and "[0-9]*" both match to null. Then the division
mark is placed on the space after "Keywords" and procmail looks for
the longest possible match to [0-9]* starting with that space.
That, again, is null, so MATCH is set to null.

We see that it works as expected if regexp is changed to this:

^Subject:.*Keywords.*\/[0-9]+

That is a whole other ball of wax. Now the shortest match to the
entirety is

Subject:Keywords9

and the division mark is placed at the 9. Then procmail refigures
the longest match to the right side starting at the division mark
and sets MATCH=9999. However here

^Subject:.*Keywords\/.*[0-9]*

the second ".*" would have reached not just up to the digits but
through them to the end of the line. MATCH would contain the rest of
all of it matched to ".*" plus null match "[0-9]*".

[for curious reader]

Given line

Subject: Keywords 9999

the second, which differs only by inserting the extraction marker,
would not match and would not set $MATCH:

^Subject: Keywords *9999 # matches ok
^Subject: Keywords *\/9999 # won't !

because the left side would be matched to "Subject:
Keywords" and the immediately following text, " 9999", did not match
the right side. It would actually make the condition fail and keep
the recipe from executing. It took a lot of circuitous coding to
allow for not knowing in advance exactly how many spaces there would
be before the digits.

Call it counterintuitive, but it's not a bug. General advice:
always make sure that the right side cannot match null or that the
last element of the left side cannot match null. Or in other words:
force the right-hand side of the \/ to match at least one character.

8.14 Explaining \/ and ()\/

`MATCH' strips all leading blank lines in 3.11pre7

[david] \/ with nothing to the left of it means "one foreslash". To
start a condition with the extraction operator, use ()\/ or \\/;
the latter looks counter intuitively like "literal backslash and
literal foreslash" (as it would mean if it appeared farther along
in the regexp), so most of us prefer the former.

*$ var ?? $s+\/$d+ # ok, \/ in the middle
*$ var ?? \/$d+ # Wrong, when \/ is at the beginning
*$ var ?? ()\/$d+ # No ok, () at the beginning

8.15 Explaining ^^ and ^

[philip] Procmail doesn't think *lines* when it matches; but it
concatenates all lines together and then runs the regexp
engine. This may be a bit surprising, but consider the following where
we want to discard any message that is likely a html advertisement

# Body consists entirely of html code
# something which'll match any message which has ""
# in the body

:0 B:
*$ $s*
html.mbox

The condition test is applied to the entire body. If you want to
limit it to match only against the beginning of the body, you have
to say so using the ^^ token, as you discovered. A simple line
anchor (^ or $) just says that there must be a newline (or the
beginning or end of the area being searched) at that particular
point in the text being matched. notice the leading anchors below.

# trap spam where the *very* first line of the body started with
#

:0 B:
*$ ^^$s*
html.mbox

What, exactly, does "Anchor the expression at the very start of
the search area..." i.e. the ^^ ?

[dan] Technically, an opening ^^ anchors to the putative
newline that procmail sees before the first character of the search
area (and a closing ^^ anchors to the putative newline that
procmail sees after the end of the search area). When the search
area is B, that is a point equivalent to the second of the two
adjacent newlines that enclose the empty line that marks the end of
the head.

The reason I'm bringing that up is this: if there are multiple
empty or blank lines between the head and the body, ^^ will mark
the start of the second of those lines, not the start of the first
line of the body that contains some text.

So if you want to test whether is the first printing text
in the body, even if it is not necessarily flush left on the very
first line, you might need a condition like the following, where
there is space/pipe/tab/pipe/dollar.

*$ B ?? ^^$SPCNL*

8.16 ANDing traditionally

Erm, you knew this already if you read the man pages. Stacking
condition lines one after another does the AND operation, where
all of the conditions must be present:

* condition1
* condition2

8.17 ORing traditionally

Here is simple OR case. There are some cases where it's impossible
to OR conditions with this style. [philip] knows more about those
cases.

* condition1|condition2

Likewise, two exit code tests can often be ORed like this

* ? command1 || command2

But there are many situations where two tests cannot be ORed by
combining them into one condition:

o a regexp search of one area ORed with a regexp search of a
different area
o a positive regexp search [i.e., for a match to its pattern] ORed
with a negative regexp search [i.e., for the absence of any
match to its pattern]
o an exit code condition ORed with a regexp search condition
o an exit code condition seeking success ORed with an exit code
condition seeking failure
o a size test ORed with anything else (even another size test)

How can I make OR conditions that all use the SAME action? I want
to be able to test for a number of variants on certain requests,
all in one block.

[hal] Yes, this can be easily done

CASE = ""

:0
* case 1 tests
{
CASE = 1
}
:0 E
* case 2 tests
{
CASE = 2
}

:0
* ! CASE ?? ^^^^
{
# real work, perhaps with explicit tests on CASE
}

Case study: Finding text from header and body

[david] In addition to the standard ways of coding OR, here's a
special one for searching the subject and the body for a given word
in either:

* HB ?? ^^(.+$)*(Subject:(.*[^a-z0-9])?|$(.*\<)*)remove\>

If the string doesn't have to be preceded by a word border, it gets
a little simpler:

* HB ?? ^^(.+$)*(Subject:.*|$(.|$))*string

8.18 ORing and score recipe

Once any of the conditions match, the score gets a positive value and
the recipe succeeds. Idea by Erik Selke

[era comments] ...allegedly the scoring system is going to cost you
more than plain old regex matching. Floating-point math and all that,
even if you use extremely simple scoring. Thus, it would probably be
slightly more efficient to do it the De Morgan way.

* 1^0 condition1
* 1^0 condition2

We can now write the previous case stydy (HB ORing traditionally)
with scores. I was tempted to write it like this, when [david]
told me the following.

* 1^0 H ?? match-it
* 1^0 B ?? match-it

[david] That will work, but it isn't the best way to do ORing,
because if a match is found to the first condition procmail still
takes the trouble to test the second one. Better, use the supremum
score on each condition:

$SUPREME = 9876543210

*$ $SUPREME^0 first_condition_to_be_ORed
*$ $SUPREME^0 second_condition_to_be_ORed
* ... etc. ...
*$ $SUPREME^0 last_condition_to_be_ORed

Upon reaching the supreme score, procmail will skip all remaining
weighted conditions on the recipe, deeming them matched. Since all
conditions on this recipe are weighted, once procmail finds one
matched condition it will skip the rest and execute the action.

8.19 ORing by using De Morgan rules

[Tim Pickett ] I thought I'd point out that
there are a few ways to do a logical OR of conditions. Someone posted
a solution here that involved using procmail's scoring system, but I
figured you could do it without scoring by taking advantage of De
Morgan's rule:

a or b is same as not(not a and not b)

or mathematically:

a || b <=> !( !a && !b )

Here's a way to do ORing

:0
* ! condition1
* ! condition2
{ } # official procmail no-op
:0 E
action_on_condition1_or_condition2

9.0 Variables

9.1 Setting and unsetting variables

You have already set variables with the "=" syntax. Variable names
are case sensitive: `var' is different from `VAR'

VAR = /var/tmp # directory
VAR = "this" # literal
VAR = 1
VAR = $FOO # another.
VAR = "$VAR at" # combined with previous value

Unsetting a variable is done like this

VAR # kill variable.
VAR= # same but with old style
VAR = "" # Variable is said to be "null" now

And you can put multiple assignments on the same line

VAR=1 VAR=2 VAR=3

Examine the following, which are all equivalent. The backticks will not
require a shell in the absence of any `SHELLMETAS' so neither of
these will spawn a shell

# _case1_: We Don't care if file exists this time...

VAR = `cat file`

# _case2_: The use of {} is considered "modern"

:0
* condition
{
VAR = `cat file`
}

# _case3_: oldish, and procmail specific and errors have
# been reported if you use this construct.
# Note: There must be no space in "VAR=|"

:0
* condition
VAR=| cat file

9.2 Variable initialisation and sh syntax

Procmail borrows some sh syntax for variable initialisation.
Note that sh's ${var:=default} and ${var=defaultvalue}
syntaxes are not available in a procmail rcfile.

o VAR1 = ${VAR2:-value}
sets VAR1 to VAR2 if VAR2 is set and non-null, and sets VAR1 to
default "value" otherwise
o VAR1 = ${VAR2-value}
sets VAR1 to VAR2 if VAR2 is set, and sets VAR1 to default
otherwise
o VAR1 = ${VAR2:+value}
sets VAR1 to "value" if VAR2 is set and non-null, and sets VAR1
to VAR2 otherwise.
o VAR1 =${VAR2+value}
Sets VAR1 to "value" if VAR2 is set and sets VAR1 to VAR2
otherwise.

And here are the classic usage examples

VAR = ${VAR:-"yes"} # set VAR to default value "yes"
VAR = ${VAR+"yes"} # If VAR contains value, set "yes"

Ever wondered if this calls `date` in all cases?

VAR = ${VAR:-`date`}

No, procmail is smart enough to skip calling `date' if VAR already
had value. It doesn't evaluate the whole line. Below you see what
each initialising operator does. Study it carefully

VAR = "" # Define variable
VAR = ${VAR:-"value1"} # VAR = "value1"
VAR = ""
VAR = ${VAR-"value2"} # VAR = ""

VAR = ""
VAR = ${VAR:+"value3"} # VAR = ""
VAR = ""
VAR = ${VAR+"value4"} # VAR = "value4"

# Note these:
VAR = "val"
VAR = ${VAR:+"value3"} # VAR = "value3"
VAR = "val"
VAR = ${VAR+"value4"} # VAR = "value4"

VAR # kill the variable
VAR = ${VAR:-"value1"} # VAR = "value1"
VAR
VAR = ${VAR-"value2"} # VAR = "value2"

VAR
VAR = ${VAR:+"value3"} # nothing is assigned
VAR
VAR = ${VAR+"value4"} # nothing is assigned

And if you want to choose from several initial values,
you might use the recipe below
instead of the standard var = ${var:-"value"}.

:0
* VAR ?? ^^^^
{
# no value (or was empty), set default value here based on
# some guesses

VAR = "base-default"

:0
* condition
{
VAR = "another-default"
}

...more conditions..
}

You could also use equivalent, but less readable condition line in
previous recipe:

*$ ${VAR:+!}

It works, because if variable contains a value the line expands to

* !

Where "!" is the procmail "false" operation. One more way to do the
same would be, that we require at leastone character to be present.
You could use also regexp (.), which would require at least one
character to be present, but you might not like matching pure spaces.

* ! VAR ?? [a-z]

9.3 Testing variables

If possible, perform positive tests, rather than negative, like below:

* ! TEST_FLAG ?? yes

Alternative with a positive test:

* TEST_FLAG ?? no
To my opinion, this is more
readable. You're free to disagree with me at this point, but all in
all, it's nicer to look at code that has as few ! flags as
possible, especially in variable tests.

[philip] The following fails if the variable is unset or null.

* variable ?? (.)

That was why I'd be better off to use

*$ variable ?? $NSPC

* variable ?? (.|$)

to require that *variable* contain at least one character. But
neither is a way to check whether a variable is set or not, because
each treats a null variable the same as an unset one. This is the
best way I know to check whether a variable is set or not:

*$ ! ${VAR+!}

[] Here is yet another way to test if variable
is set and if it isn't, sets it to a default value.

:0
*$ ! VAR^0
{
VAR = "value"
}

9.4 What does $\VAR mean?

[era and david] Procmail 3.11, $\VAR will escape regex metacharacters.
It should produce a suitably backslash-escaped expression for
Procmail's own use. In addition $\VAR will always begin with leading
empty parentheses.

You can't pass the $\VAR construct to shell programs, because there
is that leading parenthesis. Here's a recipe to standardize the regexp.
You can pass SAFE_REGEXP to an external programs like `sed'.

PROCMAIL_REGEXP = "$\VAR"

:0
* PROCMAIL_REGEXP ?? ^^\/.*
{
SAFE_REGEXP = "$MATCH"
}

[era] Note that this is slightly inexact; Procmail will
backslash-escape according to Procmail's needs, not sed's. For
example, Procmail doesn't think braces are magic (although that would
be nice to have in Procmail as well) whereas many modern variants of
sed do.

9.5 Common pitfalls when using variables

Procmail is picky and forgives nothing. Here are some of the favourite
mistakes one can make:

$EMAIL = "foo@site.com" # Done Perl lately? Remove that $

# Erm, this is ok, but many procmail recipe writers want to
# take extra precautions and include the regexps in parentheses.
# So, maybe (yabba|dabba|doo) would be more safe

REGEXP = "yabba|dabba|doo"

* Subject:.*$REGEXP # Hey, you need the "*$ Subject..."

*$ $REGEXP ?? hello # surely you meant '* REGEXP ?? hello'

9.6 Quoting: Using single or double quotes

Pay attention to this:

VAR = "you"
NEW = 'hey "$VAR"' # won't extrapolate $VAR; you get literal
NEW = "hey '$VAR'" # extrapolates to: hey 'you'

You can even combine separate words together

VAR = "1 ""and"" 2" # same as "1 and 2"

Don't let these many quotes disturb you, just count the beginning
and ending quotes. Superfluous here, but you may need some similar
construct somewhere else.

VAR = '1 '"'"'and'"'"' 2' # same as: 1 'and' 2

[david] Beware forgetting quotes, like when you'd do

SENDMAILFLAGS = -oQ/var/mqueue.incoming -odq

Procmail translates `!' into | "$SENDMAIL" "$SENDMAILFLAGS" as the
procmailrc(5) man page warns us. By the rules of sh quoting, that
means that shell sees only the first switch

% sendmail -oQ/var/mqueue.incoming

My suggestion: since you need a soft space inside `$SENDMAILFLAGS',
use the quotes when you define `$SENDMAILFLAGS' but do this instead
of using the `!' operator for forwarding:

SENDMAILFLAGS = "-oQ/var/mqueue.incoming -odq"

[Walter Haidinger ] Here's yet another
approach: deliver messages from procmail directly to mailboxes in
all those users' homes. No sendmail involved, _much_ lower loads.

:0:
*
/var/spool/mail/someuser

[philip] Assuming that "someuser" is an actual user in the
password file (I haven't been following this thread, some maybe
that isn't true here), then the following is probably better:

Walter Haidinger comments on this recipe: I'm happy to announce that
this works *really* well. No harm is done to the system-load
anymore. What a relief!

:0 w
* conditions
|procmail -d someuser

That lets procmail's very tricky "screenmailbox()" routine take
care of bogus mailboxes in a secure fashion.

Is that as safe as forwarding? Does another sendmail delivering
to /var/spool/mail/someuser use the same locking mechanism and notice
that mailbox is already locked? I don't want to risk a corrupt
mailbox.

[philip] Sendmail only delivers directly to files through
aliases that say things like:
whatever: /some/local/file
Under normal circumstances, sendmail calls the local mailer to actually
store mail in a file, and since that's procmail (right?), there
shouldn't be a problem. Also, sendmail 8 does kernel-level locking
when it delivers directly.

9.7 Quoting: Passing values to an external program

Remember to include the double quotes when you send variables'
values to the shell programs. Below you see a mistake,
because the content of the SUBJECT is not quoted and
thus not available from perl variable $ARGV[1].

:0
* condition
| perl-script $SUBJECT # mistake; use "$SUBJECT"

There is also another way. If your script can access environment
variables (almost all programs can), then you do not need to pass
the variables on the command line. Above, the SUBJECT is already
in the environment and in perl you can get it.

$SUBJECT = $ENV{SUBJECT}; # or "use Env;" and you see $SUBJECT

Next, do you know what is the difference between these two recipes?

:0
| "command arg1 arg2 arg3"

:0
| command "arg1" "arg2" "arg3"

You guessed it. The first one quotes the entire command and does not do
the right thing, the latter is correct and depending on the content of
argN variables. Anyway, play safe and always add quotes.

Sometimes you need trickier quoting to to get single quotes around
the `arg'. Pay attention to this, because this may be the reason
why your grep command doesn't seem to succeed as you expect.

# If $GREP "$arg" doesn't seem to work

* ? $GREP "'"$arg"'" $DATABASE

9.8 Passing values from an external program

External programs cannot set procmail variables directly. Programs
must write the values to external files and then read the values
from these files. Capturing only one value is easy:

var = `command` # capture STDOUT

But if a program modifies the body and exports some status
information it is trickier. We assume here that the script is
controlled by you and that you have added the switch
--export-status option which causes the program to print
information to a separate file.

LOCKFILE = $HOME/.run$LOCKEXT # protect external file writing
valueFile = $HOME/tmp/values

# modify body, and export status values to external file: one
# value in every line
#
# VALUE1
# VALUE2
# VALUE3

:0 fb
| $NICE script.pl --export-status $valueFile

values = `cat $valueFile`

# Derive values from each line

:0 # line 1
*$ values ?? ^^\/[^$NL]+
{
var1 = $MATCH
}

:0 # line 2
*$ values ?? ^^.*$\/[^$NL]+
{
var2 = $MATCH
}

:0 # line 3
*$ values ?? ^^.*$.*$\/[^$NL]+
{
var3 = $MATCH
}

LOCKFILE # Release lock

[richard] Alternatively write valueFile from your rc or external
program with lines like

PARAM1="value for param 1"
PARAM2="value for param 2"
PARAM3="value for param 3"

and read it with

INCLUDERC $valueFile

Now there is no need to worry about synchronizing the read with the
lines, or about adding new parameters, since each is labeled in
valueFile.

9.9 Incrementing a variable by a value N

[dan, phil and Richard] Here's a recipe for incrementing a variable
by a value N. If $VAR is not a number, we get an error. Note that
if $VAR + $N is not greater than 0, this recipe will not change the
value of VAR if the assignment happens inside braces. You must
place the assignment after the closing curly brace.

:0
*$ $VAR ^0
*$ $N ^0
{ } # procmail no-op
VAR = $=

9.10 Comparing values

It's too expensive to call the shell's `test' function to do
[-lt|-eq|-gt] because you can do the same with procmail. The
do-something below is run if SCORE <= MAXIMUM. The recipe simply
subtracts SCORE from MAXIMUM and determines if the result is
positive.

:0
*$ -$SCORE ^0
*$ $MAXIMUM ^0
{
.. do-something
}

[idea by era] it's getting slightly cumbersome if it's between MIN
and MAX:

:0
*$ $SCORE ^0
*$ -$MIN ^0
{
dummy # no-op, just for the LOG

:0
*$ -$SCORE ^0
*$ $MAX ^0
{
suitable
}
}

Eg. When values are MIN=1, MAX=5, SCORE=4

procmail: Assigning "SCORE=4"
procmail: Score: 4 4 ""
procmail: Score: -1 3 ""
procmail: Assigning "dummy"
procmail: Score: -4 -4 ""
procmail: Score: 5 1 ""
procmail: Assigning "suitable"

9.11 Strings: How many characters are there in a given string?

# 1998-06-23 PM-L [walter]

:0
* 1^1 VAR ?? .
{ }
LENGTH = $

9.12 Strings: How to strip trailing newline.

Suppose you have used regexp, which left newline($) in the `MATCH'.
If you wonder why the recipe works, remind yourself that regexp
operator "." never matches a newline.

:0
* VAR ?? ^^\/.+
{
VAR = $MATCH
}

9.13 Strings: deriving the last N characters of a string.

# 1998-06-23 PM-L [walter] Note the use of
# the $ sign below to anchor to end-of-string...
#
# For last 2 characters use * VAR ?? ()\/..$
# For last 5 characters use * VAR ?? ()\/.....$

:0 # Last character
* VAR ?? ()\/.$
{
TAIL = $MATCH
}

9.14 Strings: Getting partial matches from a string.

[dan] Getting a match to the right is quite easy with procmail's
`match' operator.

VAR = "1234567890"

:0
* VAR ?? ()\/3.*
{
result = $MATCH # now 34567890
}

but deleting 2 characters from the end is nearly impossible without
forking an outside process. The cheapest might be `expr' because it
doesn't need a shell to pipe `echo' to it (as `sed' would and I
believe `perl' would):

# by resetting the shellmetas, this will only call
# `expr'. If we wouldn't have fiddled with shellmetas,
# this would have called two processes: sh + expr

saved = $SHELLMETAS
SHELLMETAS
result = `expr "$VAR" : '$.*$..'` # now 12345678
SHELLMETAS = $saved

ksh or bash could do it as well:

# semicolon to force invoking a shell, actually
# first question mark will force a shell already.

saved = $SHELL
SHELL = /bins/sh
result = `echo ${VAR%??} ;`
SHELL = $saved

Now, if you know that the last two characters will be "90", that's
different. Of course, this totally screws up if the third-to-last
character is a 9.

:0
* VAR ?? ()\/.*[^0]
* MATCH ?? ()\/.*[^9]
{
result = $MATCH # now 12345678
}

[jari] Comments: If a shell must be used, then `awk' is a good tool for
simple string manipulation. Its startup time is faster that perl's
whose overhead is due to internal compilation. `awk' also consumes
less recourses overall than `perl'. Following will only work if VAR
is a string of continuous block of characters. (ARGV[1] can be used)

saved = $SHELLMETAS
SHELLMETAS

VAR = ` awk 'BEGIN{ v = ARGV[1]; \
print substr(v,1,length(v)-2); exit }' \
"$VAR" \
`

SHELLMETAS = $saved

This version requires _some_ file, any file, so that we get awk
started. In the previous code all the work was done in the BEGIN
block and no file was ever opened.

saved = $SHELLMETAS
SHELLMETAS

VAR = ` awk '{print substr(v,1,length(v)-2); exit }' \
v="$VAR" /etc/passwd \
`

SHELLMETAS = $saved

[dan] comments awk: `expr' is sure to be a smaller binary than awk
for procmail to fork, and it needs much less command-line code to
do this job. Note also that one still has to diddle with SHELLMETAS
to avoid a shell, because the awk code contains brackets; thus it
doesn't replace all.

There is also a way to remove words from the end of string by
procmail means if the strings are separated by same separator. Let's
use the word this-mailing-list-request which we would like to shorten
to this-mailing-list. [david] presented the recipe 1998-06-16 in PM-L.

VAR = "this-mailing-list"

# 1) if there is match at the end ending to these words
# 2) Get everything up till last match and store it to MATCH
# 3) Read MATCH, but exclude last dash "-"

:0
* VAR ?? -(owner|request|help)^^
* VAR ?? ^^\/.*-
* MATCH ?? ^^\/.*[^-]
{
VAR = $MATCH
}

9.15 Strings: Procmail string manipulation example

[1998-06-23 PM-L walter] ... Now we get to apply these formulas
to strip the last character off a string. It gets a bit ugly for
special cases. I've deliberately chosen a worst-case scenario.

VAR = "Testing 012301230111"
RC_APPEND = $PMSRC/pm-myappend.rc

:0
* VAR ?? ()\/.$
{
TAIL = $MATCH # last character of VAR "1"

# Get the longest match that does not end in the TAIL character

:0
*$ VAR ?? ()\/.*[^$TAIL]
{
HEAD = $MATCH # now "Testing 012301230"

# if the last two or more characters in VAR are
# identical, they all get chopped, oops

:0
* -1^0
* 1^1 VAR ?? (.)
* -1^1 HEAD ?? (.)
{
dummy = "tooshort"
INCLUDERC = $RC_APPEND
}
}
}

result = $HEAD # "Testing 01230123011"

# ........................................ pm-myappend.rc
# LENGTH(HEAD) plus 1 SHOULD equal LENGTH(VAR). That is
# not the case when the last 2 (or more) ending
# characters are identical. in that case, call appendrc
# recursively to stick back an appropriate number of
# TAIL characters.

:0
* -1^0
* 1^1 VAR ?? (.)
* -1^1 HEAD ?? (.)
{
HEAD = "$HEAD$TAIL"
INCLUDERC = $RC_APPEND
}

9.16 How to raise a flag if the message was filed

FILED = ! # ! is procmail "false"

:0 c: # We process the message more
* condition
foo

:0 a
{
FILED # Kill variable
}

...

:0 # Stop if previous cases filed the message
*$ $FILED
{
HOST = "_done_"
}

Or alternatively: procmail automatically sets `LASTFOLDER' if
it delivers message to mailbox.

LASTFOLDER # kill variable

:0 c:
* condition
foo

:0 c:
* condition
bar

... et cetera ...

:0
* ! LASTFOLDER ?? ^^^^ # Or ${LASTFOLDER+!}!
{
HOST = "_done_" # Force procmail to stop
}

9.17 Dollar sign in condition lines.

#todo, check this recipe

This doesn't seem to work for me...

* ^TO()$\foo@bar.com

[david] An unescaped dollar sign later in the line represents a
newline, so what you have there is searching for the following:

. An expression that matches the expansion of the ^TO token (which
is anchored to the start of a line by its definition), followed
by
. A newline, followed at the start of the next line by
. "foo@bar" [the backslash escapes the f, which didn't need
escaping], followed by
. any character that is not a newline (the period is unescaped),
and finally
. "com".

Try this instead:

*$ ^TO()$\foo@bar\.com

#todo: the dollar seems exactly the same in the above two
#todo Examples: are you sure that this is correct?

In fact, to avoid matches to things like foo@bar.community.edu,
you might want to do it this way:

*$ ^TO()$\foo@bar\.com\>

9.18 Finding mysterious foo variable

I have my fellow worker's procmail code and he uses a variable FOO
that I can't find in his code anywhere. It's not a shell variable
either, because it's literal. Where does it come from?

Your procmail runs /etc/procmailrc when it starts, please check
that. It may define some common variables already for all users.

9.19 Storing code to variable

One way to run complex code in a procmail recipe is first to store
it in a variable. Idea by [era]. You could do this in a separate shell
script too. The following example reads URLs from the body of
a message: the URLs have been put to separate lines and some special
Subject is used to trigger the dumping of the html pages:

# Code by [era]
#
COMMAND='while read url; do
case "$url" in
*://*)
lynx -traversal -realm -crawl -number_links "$url" |
$SENDMAIL $LOGNAME
;;
esac
done'

# Notice the trailing semicolon after `eval' !
:0 bw
* ^Subject: xxxxx
| eval "$COMMAND" ;

If you want to run the code inside the nested block, then look
carefully, there are double quotes around the command in backticks.
If you leave double quotes out, then each word in SH_CMD would be
interpreted separately:

$SH_CMD = '$echo "$VAR" >> $HOME/test.tmp'

:0
* condition
{
# condition satisfied; run the given shell command
# and do something more.

dummy = `"$SH_CMD"`

..rest of the code..
}

A similar construct works for message echo-ing too:

MESSAGE='Thank you so much for your message.
Unfortunately, the volume of mail I receive .... (blah blah blah).
If your matter is urgent, try calling +358-50-524-0965.
'

:0 hw
* ! ^X-Loop: moo$
| ($FORMAIL -rt -A "$MY_XLOOP"; echo "$MESSAGE") | $SENDMAIL

9.20 Getting headers into a variable.

[david] Here are several ways to get the entire header into a variable:

HEADER = `$FORMAIL -X ""` # The space after the X is vital.
HEADER = `sed /^$/q` # also writable as HEADER=`sed /./!q`

:0 h
HEADER=|cat -

will save the entire header into one variable. It has to be smaller
than $BUFSIZE, though. This way might work as well, and will require no
outside processes if it does:

:0
* ^^\/(.+$)*$
{
HEADER = $MATCH
}

9.21 Converting value to lowercase

If you know that a word belongs to set of choices, you can do
this inside procmail

LIST = ":word1:word2:word3:word4" # Colon to separate words
WORD = "WORD1"

:0
*$ LIST ?? :\/$WORD
{
WORD = $MATCH
}

But if you don't know the word or string beforehand, then this is
the generalized way: [idea by era and david]

:0 D
* WORD ?? [A-Z]
{
WORD = `echo "$MATCH" | tr A-Z a-z`
}

10.0 Suggestions and miscellaneous

10.1 Speeding up procmail

o Use absolute paths to take the burden of searching binary along path
from shell: Use $FORMAIL variable abstraction.

$FORMAIL = "/usr/local/bin/formail"

:0 fhw
| $FORMAIL -I "X-My-Header: value"

o Multiple `echo' commands that spread many lines can be converted
to single echo command if \n escape is supported. You usually
see these in autoresponders

echo "........."; \
echo "........."; \
echo ".........";

-->

echo ".........\n" \
".........\n" \
".........\n";

o You can avoid multiple and possible expensive FROM_DAEMON tests
by caching the result at the top of your .procmailrc. You can
now use variable $from_daemon like the big brother FROM_DAEMON.
The same idea can be applied to FROM_MAILER regexp. If you have
*pm-javar.rc*, it already defines variables `$from_daemon' and
`from_mailer' exactly like here:

from_daemon = "!"

:0
* ^FROM_DAEMON
{
from_daemon = "!!" # double !! means "OK"
}

:0
*$ ! $from_daemon
{
..do-it..
}

o Count the backticks and you know how many shell calls procmail
has to launch. See if you can minimize them and use some procmail
code instead.
o ^TO and other macros are expensive, see if you can use simple
Header:.*\ instead. Well, it's not clear if this
gives you much speed advantage.
o Don't call "$FORMAIL -xHeader:" every time you need a header
value, consider if it suffices to use `match' operator \/.
o You can minimize the calls to only one `formail' if you add many
headers along the way: See formail usage tips in this document
o Searching body is expensive, simply because it contains more text.
There isn't much to do about this, because you use `B' anyway
when you need it.
o See if you can move some tasks to your .cron file. procmailrc is
not meant for those purposes. Instead of calculation daily
values every time in procmail, let cron do that at 04:00 or
21:00. Don't run cron at midnight if you can, because everybody
else is running their crons at the same time. If "logical" date
change time can be used (when you arrive to work, when you
leave the work), use it in cron jobs.
o [philip] Setting `LINEBUF' permanently to a big value slows
procmail down.
o Remove all calls to `perl' and use programs that are nicer to
the system (If you just call command line perl, there is
probably an equivalent alternative with `awk' `tr' `sed' `cut')
o Examine each shell command and see if you do need `SHELLMETAS.'
If you can set `SHELLMETAS' to empty, this saves calling "sh" for
each invocation of the external command.

10.2 See the procmail installation's examples

Did you remember to look at the examples that come with procmail? If
not, it's time to give them a chance to educate you. Here is one
possible directory you could take a look. Ask from your sysadm if you
can't find the directory where to look into.

% ls /usr/local/lib/procmail-3.11pre7/examples/

Or if you're really anxious to get on your own, try this. The directory
/opt/local is for HP-UX 10 machines and the *forward* contains example
how to define your `.forward' for procmail.

% find /opt/local/ -name "forward" -print

If the find succeeded and found the file, then you know where the
procmail files installation directory is.

10.3 Printing statistics of your incoming mail

If you keep the procmail log crunching, it will record to which
folder the messages was filed. There is program `mailstat' which
can process the procmail.log file and print nice summary out of it.
If you generate the summary at midnight and clear the log, you
get pretty nice per day/per folder traffic analysis.

# -m merges all error messages into a single line

% mailstat -km procmail.log

10.4 Storing UBE mailboxes outside of quota

I want to store spam outside disk space. Problem: if I tell
procmail to deliver to, say, /tmp/spam.box, it does so just fine
(according to the log). Unfortunately, it delivers to /tmp on the
mail host which I cannot access. spam.box doesn't appear in the
/tmp directory of the shell machine when procmail is invoked for
incoming mail.

[philip] Under the most likely configuration of sendmail in
this situation, it is impossible to have procmail invoked by
sendmail on the shell machine: sendmail is probably set to just
forward all mail to the designated mail delivery machine.

There are other options: you could temporarily store the mail in
your account, then have a cronjob on the shell machine that
reprocesses the message. That would probably be more efficient than
having each message trigger an rsh to the shell machine. If you
actually get enough spam that it's pushing against your quota, then
the rsh is too expensive -- use a cronjob that invokes something
like:

cd your-maildir &&
lockfile spam.lock &&
test -s spam &&
{
cat spam >> /tmp/spam.box && rm -f spam spam.lock || \
rm -f spam.lock;
}

WARNING: the above assumes the following:

o everything in your-maildir/spam is spam and belongs in
/tmp/spam.box
o no further filtering of the messages is necessary: they just need
to be moved (it actually treats everything in the
your-maildir/spam as a single message and uses procmail as a
reliable copy command, thus the `DEFAULT' assignment as the use
of /dev/null as a empty procmailrc)
o /tmp/spam.box is a not a directory

If the latter two of those conditions isn't true OR IF THEY MIGHT
CHANGE then you should use `formail' `-s' to break the message apart
and invoke procmail on each one separately.

[era] Many sites cross-mount directories for various reasons. /tmp
is always local but /var/tmp might be cross-mounted between the
login host and the mail host; another one to try is /scratch -- and
if all else fails, ask your admin to set up an NFS share for this
purpose.

10.5 Using first 5-30 lines from the message

[era] The regex to grab few lines (or all of them, if there are
less than fifty) is not going to be very pretty, but it saves launching
an extra process.

:0 B
* $ ^^$SPCNL*\/$NSPC.*$(.*$)?(.*$)? ... etc, the rest of the lines
{
toplines = $MATCH
}

The skipping of whitespace at the beginning of the message is of
course not necessary. You should probably set `LINEBUF' reasonably
high if you grab many lines, say 30: 80*30 = 2400 bytes; probably
setting it to 8192 or 16384 is a good idea, depending how much you
want to match. The above gets ugly quickly, so

# But if N=30, sed ${N}q if you don't have head

:0 Bi
{
toplines = `head -$N`
}

:0 a
* toplines ?? pattern
{
...do-it
}

10.6 Using cat or echo in scripts?

I have seen a lot of examples that use 'echo', i.e.,

:0
* condition
| echo "first line of message" \
"second ..." \
"et cetera"

I started out with spam.rc from "ariel" which got me into the
habit of

:0
* condition
| cat file_containing_message

although I note that spam.rc did have one recipe using the echo
method. What are the reasons for choosing each method over the
other?

Here is a comparison table. Choose the one you think is best for you

o Echos don't have dependency on an external file:
everything is contained in the .procmailrc file. Echos keep
all the relevant stuff in one file. Cat's make you
maintain multiple files. That's the main
reason I lean toward echo's; you may have accounts on
several machines. It is easier to be able to copy just one
generic .procmailrc between them without having to copy a bunch
of messages also. Mostly, though, there's no real difference
between the two methods.
o Echo is easier to use with variables.
o Echo starts many processes, cat only starts one, but this is
not always true: In most current Bourne shell implementations,
echo is a builtin. This holds true with tcsh too.
o The main problem I see with the use of cat is "what happens when
you forget the file or destroy it ?". I suggest to, at least,
test that the file is readable before catting it.
o [richard] An argument against echo is that it is not well
standardized, and different versions may exist on the same
machine. Some recognize -n, some don't; some recognize embedded
metacharacters, some don't.This is an argument in favor of
`print'. Print, however, is not a built-in on all systems. The
comment on built-ins is pertinent to situations when a shell is
spawned. When procmail handles the call directly, it will
always look for a stand-alone executable. I guess echo may be
better, as long as we are aware of any differences in behavior
between built-in and stand-alone versions.

10.7 How to run an extra shell command as a side effect?

[jari] I was once wondering what would be the wisest way to send
messages to my daily "biff" log file about the events that
happened during my .procmailrc execution. This is how [david]
commented on my ideas

# case 1: print to BiffLog

dummy = `echo "message: $FROM $SUBJECT" >> $biff `

[david] Problems you get no locking on the destination file, and
unless you put it inside braces you have to run it on every message
unconditionally. (Also procmail tries to feed the whole message to
a command that won't read it, but the remedies for that don't help
very much.)

# case 2: We consume delivering recipe and therefor have to use
# `c' flag.

:0 whic:
| echo "message: $FROM $SUBJECT" >> $biff

Here it locks the destination file and you can add conditions to
it, so it's probably the best. If the head or the body is less than
one bufferful, you can limit the unnecessarily written data with `h'
or `b', but I think that in most OSes a partial buffer and a full
one are the same amount of effort.

# case 3: We use side effect of "?" here. Cool, but this
# doesn't do $biff file locking thus message order may
# not be what you expect.

:0
* condition
* ? echo message: $FROM $SUBJECT >> $biff
{ } # procmail no-op

We have conditions possible, but there is no locking on the
destination file. I'd go with method #2 or a variation thereof:

:0 hic: # we don't necessarily need `w'
* condition
| echo message: $FROM $SUBJECT >> $biff

:0 hi: # Or you could use this
* condition
dummy=| echo message: $FROM $SUBJECT >> $biff

[jari] Now, when [david] has explained how various ways differ
from each other, I present the recipe where I used the case 3.
When I was dropping a message to a folder, I wanted to send a
message to my biff log too. The idea is that the drop-conditions
have already matched and then we run extra command by using side
effect of "?" token. As far as the recipe is concerned, the "?"
is a no-op. The pedantic way would have been to add the LOCKFILE
around to the recipe, but imagine 50 similar recipes like
this...and you understand why the LOCKFILE was left out. It's
only necessary if you worry about sequential writing to the biff
file.

:0 :
* drop-condition
* ? echo message: $FROM $SUBJECT >> $biff
$MBOX

10.8 Forcing "ok" return status from shell script

...the "?" trick only allows running some additional shell
commands (`true' command always succeeds) while conditions
above have already determined that drop will take place. And you
can always make condition to succeed if a misbehaving shell script
always returns a failure exit code.

* ? misbehaving-shell-script || true

[david] If the script *always* returns a failure code, just do this:

* ! ? misbehaving-shell-script

The more complex case is a script that can return either success or
failure but you don't care which; if the drop conditions passed,
you want to run the action line. `echo' can also fail if the
process lacks permission or opportunity to write to stdout. A more
reliable choice is true(1); its purpose in life is to do nothing
but exit with status 0.

The command `:' is a shell builtin which always returns true
status. Not exactly more readable than true(1) "|| :" will save the
invocation of true (unless true is built into $SHELL), but procmail
will still run a shell. On the other hand, as long as the command
itself has no characters from `SHELLMETAS' a weight of 1^1 and no
"|| anything" will avoid the shell process as well.

However, there is yet a better way to make sure that a failure by the
script doesn't make procmail abort the recipe:

:0 flags
* other conditions
* 1^1 ? shell-script
action

Regardless of the exit status of the script, the condition will score
1 and not interfere with procmail's decision about the action line of
the recipe. Weighted exit code conditions behave like this (see the
procmailsc(5) man page):

* w^x ? command

scores w on success or x on failure.

* w^x ! ? command

scores the same as this:

* w^x pattern_that_appears_in_the_search_area_$?_times

10.9 Make your own .procmailrc available to others

There is never too much to learn about procmail and the best source
is the rc files that people have done. Remember to comment
your .procmailrc file well before you put it available. Below
is a recipe
for sending your .procmailrc upon request. If you want to send
anything more that one or two files (many times you want to put
other files available too), then please do not use this code but a
general file server module.
(Note: #REF #procmail_module_list ;Procmail module list;)

:0
* ! ^Subject:.Re:
* ^Subject:.*send.*procmailrc
* ! ^FROM_DAEMON
{
:0 fhw:
| $FORMAIL -rt \
-A "Precedence: junk" \
-I "Subject: Requested .procmailrc"; \
-I "$MY_XLOOP"

:0 a hwic
| ( cat - $HOME/.procmailrc ) | $SENDMAIL

:0 # trash the "Send procmailrc" request
/dev/null
}

10.10 Using dates efficiently

_Note_: See module list, where you will find `date' and `time'
parsing modules. You can also parse the date from the first
`Received' or `From_' header if it is the same each time in your
system. That would be orders of magnitude faster and decreases
your system load if you receive lot of mail.

Calling `date' in your procmail script _many_ times is not a good
idea. Use the `MATCH' as much as possible to be efficient in
procmail, like below where we call `date' only once. If you are not
in the same time zone as your server, and you want an accurate
report of the date, you might amend the invocation to the following:

date = `TZ="KDT9:30KST10:00,64/5:00,303/20:00";date "+%Y %m %d"`

The basic recipe is here

# By [richard] add %H:%M%S if you want these as well

:0
* date ?? ^^()\/....
{
YYYY = $MATCH
}

:0
* date ?? ^^..\/..
{
YY = $MATCH
}

:0
* date ?? ^^.....\/..
{
MM = $MATCH
}

:0
* date ?? ()\/..^^
{
DD = $MATCH
}

TODAY = "$YYYY-$MM-$DD" # ISO std date: like 1997-12-01

10.11 Keep simple header log

Here is a simple strategy: Record all what comes in and record all what
happened to that message. See how brief info is constantly recorded to
*BIFF* folder. You can now check the *BIFF* log every day to
see if the messages were sunk to right folders: Remember to add *BIFF*
rule to every recipe, so that the sink message [sunk-somewhere] is
recorded after incoming message headers.

I use this one-liner log in my Emacs window which is updated by
`live-mode' process all the time (See the Emacs tools section
later). It gives a nice overview of email messages the I'm receiving:
it's my biff(1) equivalent in Emacs.

# this requires that HH and MM have been setup before,
# see pm-jadate.rc

NOW = "$HH:$MM" # the time only
TODAY = "$YY-$MM-$DD $NOW" # ISO 8601: date and time

$NULL = $SPOOL/junk.null.spool # /dev/null is dangerous
BIFF = $PMSRC/pm-biff.log

# or if you prefer a log per day (easy for cleanup):
# BIFF = $PMSRC/pm-biff.log.$YYYY$MM$DD

# .............................................. headers ...

# DON'T USE THESE: they call shell
#
# FROM = `$FORMAIL -zxFrom:`
# SUBJECT = `$FORMAIL -zxSubject:`

:0 # Use procmail match feature
* ^From:\/.*
{
FROM = "$MATCH"
}

:0 # Use procmail match feature
* ^Subject:\/.*
{
SUBJECT = "$MATCH"
}

# ............................................. incoming ...
# record log of incoming mail

# or if you use a biff file per day, you could have:
# echo "$NOW $FROM $SUBJ" >> $BIFF

:0 hwic:
| echo "$TODAY $FROM $SUBJ" >> $BIFF

# ......................................... null recipe ...
# spam-like addresses - let friends@planetall.com fall through

:0 :
* From:.*(remove|delete|free|friend@)
* ? echo " [null-AddrReject]" >> $BIFF
$NULL

10.12 Gzipping messages

[Sean B. Straw ] On the recipe
delivery line where you'd normally be tossing it into a folder do
this instead:

:0 c:
|gzip -9fc >> $MAILDIR/mail.mbox.gz

This will compress each message as it comes in (and since most are
TEXT, it does a fine job - MIME, OTOH is one of the best ways to
mailbomb someone since it doesn't compress well - but the indirect
bombing via mailing lists doesn't do this), reducing the disk space
required, usually dramatically. Done in conjunction with something
like the following at the end of your .procmailrc, you could have a
header file you could quickly rummage through looking for valid
messages to add to a procmail recipe, then run:

gzip -d -c mail.mbox.gz | formail -s procmail -m recipe.rc

(note that if the recipe delivers into the mail.mbox.gz file on any
condition, then you should look to MOVE the file before running
this process, and use the moved version. In fact, this would be a
good idea anyway, as newly delivered mail may appear in the end of
the gzip file while you're doing this - and since your ultimate
goal is to be able to eliminate junk, you'll want to know that
after you've processed a gzipped mail file, you can delete it
without accidentally whacking new mail).

:0
* LASTFOLDER ?? ^^^^
{
# Save the message in case we need to retrieve it.

:0 c:
|gzip -9fc >> $MAILDIR/mail.mbox.gz

# copy headers for easy browsing - including being able to
# identify lists you're being subscribed to.

:0 h:
header.log
}

10.13 Emergency stop for your .procmailrc

[jari] If I have a bad luck while I am testing a new recipe, it may
run in a loop and and it may send me continuously email messages. I
then have to quickly recall .procmailrc and start disabling my
individual "control" recipe files. Yet I figure, in situations like
this where every second is important, there must be a better way.
[alan] This is quite easy already; put this at the top of your
.procmailrc:

# instead of leading dotfile, you may prefer
# stopFile = $HOME/procmailrc.stop which shows up in default ls.
# In the other hand you can do ls ~/.procmail* to see both...

stopFile = $HOME/.procmailrc.stop

:0
*$ $IS_EXIST $stopFile
{
EXITCODE = $EX_TEMPFAIL # Means: retry later; requeue
HOST = "_stopped_by_external_request_"
}

Then, when testing your procmailrc and disaster happens, you can
simply do following to disable your procmailrc filtering.

% touch $HOME/.procmailrc.stop

[richard] This is also a candidate recipe for including in
an INCLUDERC. Combining the two ideas, we have a file
procmailrc.stop which contains the recipe and is included near the
top of .procmailrc, When you don't want it, mv it to procmailrc.go.
Procmail complains about missing INCLUDERCs, but it does not
complain about them if they exist and are empty. Another reason to
not use dotted file names, but to use cp instead of mv.

Home page Services provided Software available Site licenses Systems status Local Documentation Windows 2000 Reporting problems Links Contact information