# Help! New bug in procmail or fetchmail?

## jkcunningham

Last weekend I updated portage and emerged -u world. Since then I've been receiving a flood of duplicate spam emails. For awhile I thought I was under some kind of attack. But I finally realized that it is the same couple emails, somehow being duplicated by my machine. These emails are different than other emails in that they have no ID and no date. Maybe there are other differences, but that's all I've noticed. 

Here's what happens. One of these spam emails arrives at my ISP. Fetchmail polls for new email every 30 seconds. It picks it up and passes it to procmail for processing. Procmail processes its recipes and bogofilter correctly identifies it as spam and puts it in the spam folder. Then procmail seems to repeat the entire process with the same email. Over and over. The processing time is about 10 seconds, so it looks like an email is arriving every ten seconds. But if I go to the ISP via webmail I can see that there is only one and it  is not being removed. 

Here is the verbose output of the procmail.log:

procmail: Assigning "ADMINFOLDER=/home/jcunningham/Mail/admin"

procmail: Assigning "BULKFOLDER=/home/jcunningham/Mail/bulk"

procmail: Assigning "FORMAIL=/usr/bin/formail"

procmail: Assigning "SENDMAIL=/usr/sbin/sendmail"

procmail: Executing "bogofilter,-u,-e,-p,-l"

procmail: Match on "^X-Bogosity: Yes"

procmail: Locking ".caught-bogo"

procmail: Executing "rcvstore,+spam/caught-bogo"

procmail: Assigning "LASTFOLDER=rcvstore +spam/caught-bogo"

procmail: Unlocking ".caught-bogo"

procmail: Notified comsat: "jcunningham@:/home/jcunningham/Mail/rcvstore +spam/caught-bogo"

From jcunningham  Wed Jul 30 07:57:01 2003

 Subject: thank-you

  Folder: rcvstore +spam/caught-bogo					    793

procmail: Assigning "ADMINFOLDER=/home/jcunningham/Mail/admin"

procmail: Assigning "BULKFOLDER=/home/jcunningham/Mail/bulk"

procmail: Assigning "FORMAIL=/usr/bin/formail"

procmail: Assigning "SENDMAIL=/usr/sbin/sendmail"

procmail: Executing "bogofilter,-u,-e,-p,-l"

procmail: Match on "^X-Bogosity: Yes"

procmail: Locking ".caught-bogo"

procmail: Executing "rcvstore,+spam/caught-bogo"

procmail: Assigning "LASTFOLDER=rcvstore +spam/caught-bogo"

procmail: Unlocking ".caught-bogo"

procmail: Notified comsat: "jcunningham@:/home/jcunningham/Mail/rcvstore +spam/caught-bogo"

From jcunningham  Wed Jul 30 07:57:35 2003

 Subject: thank-you

  Folder: rcvstore +spam/caught-bogo					    793

procmail: Assigning "ADMINFOLDER=/home/jcunningham/Mail/admin"

procmail: Assigning "BULKFOLDER=/home/jcunningham/Mail/bulk"

procmail: Assigning "FORMAIL=/usr/bin/formail"

procmail: Assigning "SENDMAIL=/usr/sbin/sendmail"

procmail: Executing "bogofilter,-u,-e,-p,-l"

procmail: Match on "^X-Bogosity: Yes"

procmail: Locking ".caught-bogo"

procmail: Executing "rcvstore,+spam/caught-bogo"

procmail: Assigning "LASTFOLDER=rcvstore +spam/caught-bogo"

procmail: Unlocking ".caught-bogo"

procmail: Notified comsat: "jcunningham@:/home/jcunningham/Mail/rcvstore +spam/caught-bogo"

From jcunningham  Wed Jul 30 07:58:09 2003

 Subject: thank-you

  Folder: rcvstore +spam/caught-bogo					    793

procmail: Assigning "ADMINFOLDER=/home/jcunningham/Mail/admin"

procmail: Assigning "BULKFOLDER=/home/jcunningham/Mail/bulk"

procmail: Assigning "FORMAIL=/usr/bin/formail"

procmail: Assigning "SENDMAIL=/usr/sbin/sendmail"

procmail: Executing "bogofilter,-u,-e,-p,-l"

procmail: Match on "^X-Bogosity: Yes"

procmail: Locking ".caught-bogo"

procmail: Executing "rcvstore,+spam/caught-bogo"

procmail: Assigning "LASTFOLDER=rcvstore +spam/caught-bogo"

procmail: Unlocking ".caught-bogo"

procmail: Notified comsat: "jcunningham@:/home/jcunningham/Mail/rcvstore +spam/caught-bogo"

From jcunningham  Wed Jul 30 07:58:43 2003

 Subject: thank-you

  Folder: rcvstore +spam/caught-bogo					    793

Here is the .procmailrc:

VERBOSE=on                  

SHELL=/bin/sh               

LINEBUF=4096                

PATH=$HOME/bin:/bin:/usr/bin:/usr/local/bin

MAILDIR=${HOME}/Mail        

LOCKEXT=.lock

DEFAULT=${MAILDIR}/inbox

LOGFILE=${HOME}/procmail.log

ADMINFOLDER=${MAILDIR}/admin

BULKFOLDER=${MAILDIR}/bulk  

FORMAIL=/usr/bin/formail    

SENDMAIL=/usr/sbin/sendmail 

## First run stuff through bogofilter

:0fw

| bogofilter -u -e -p -l

## If bogofilter fails, throw the mail back into the pipe

## e=only execute if preceeding recipe executed and failed

:0e

{ EXITCODE=75 HOST }

## if its spam, put it in the caught folder

:0:.caught-bogo

* ^X-Bogosity: Yes

| rcvstore +spam/caught-bogo

## If it passed, try it through spamassassin.

:0 fw

* < 256000

| /usr/bin/spamassassin

## Mails with a score of 15 or higher are almost certainly spam (with 0.05%

## false positives according to rules/STATISTICS.txt). Let's put them in a

## different mbox. (This one is optional.)

:0:.caught-sa

* ^X-Spam-Level: \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*

| rcvstore +spam/caught-sa

## All mail tagged as spam (eg. with a score higher than the set threshold)

## is moved to "probably-spam".

:0:.caught-sa-maybe

* ^X-Spam-Status: Yes

| rcvstore +spam/caught-sa-maybe

###

## Accept all the rest to default mailbox

:0:.inbox

| rcvstore +inbox

And, here is my .fetchmailrc:

# Configuration created Fri May 24 08:02:19 2002 by fetchmailconf

set postmaster "jcunningham"

set bouncemail

set no spambounce

set properties ""

# set check for mail interval in number of seconds 

set daemon 30

poll <my.isp> with proto POP3

       user 'jeffrey' there with password '<password>' is 'jeffrey' here

       options fetchall stripcr warnings 3600 mda '/usr/bin/procmail -f -'

And finally, here is the full header for one of the problem emails:

Return-Path: <adlung7@hotmail.com>

Delivered-To: cunningham.net%jeffrey@cunningham.net

Received: from mail.cunningham.net.criticalpath.net [209.231.81.83]

	by localhost with POP3 (fetchmail-6.2.3)

	for jeffrey@localhost (single-drop); Wed, 30 Jul 2003 07:58:43 -0700 (PDT)

Received: (cpmta 10901 invoked from network); 21 Jul 2003 22:51:37 -0700

Received: from 81.248.149.186 (HELO ALagny-110-1-7-186.w81-248.abo.wanadoo.fr)

  by smtp.c000.snv.cp.net (209.228.33.183) with SMTP; 21 Jul 2003 22:51:37 -0700

X-Received: 22 Jul 2003 05:51:37 GMT

From: "Rupert" <adlung7@hotmail.com>

To: <jeffrey@cunningham.net>

Subject: thank-you

Content-Type: text/html;

	charset="windows-1251"

X-Bogosity: Yes, tests=bogofilter, spamicity=0.999647, version=0.13.7.2

I can't figure out if the problem is with fetchmail, procmail, or what. Before the system update I received these same type of spam emails without having this problem, and I did not change any settings after the update. 

Help! Any ideas?

-Jeff

----------

## jkcunningham

Anybody?  Any ideas?

----------

## BackSeat

A few suggestions. Try removing 'fetchall' from the fetchmail file. That way you can find out if it is fetchmail somehow not deleting the message on the server. Also, have you checked the 'jcunningham' account to see if fetchmail is sending anything there? (And is 'jcunningham' actually a completely separate account, rather than an alias?).  Finally, if all else fails, a re-emerge of the relevant applications may be in order, in case you're stuck with an old library or similar.

Be interested to know how you get on.

BS

----------

## Genone

That's probably not related to your problem, but I'd recommend increasing the poll interval, polling every 30 seconds is not nice for the servers (and I doubt you check your mail every 30 seconds). Checking every 5 minutes should be more than sufficient.

----------

## jkcunningham

Thanks, guys. Removing the 'fetchall' seems to have gotten around this problem...so far, at least. When I moved one of the offending emails (via webmail) from a temp directory to the pop3 inbox, fetchmail picked it up only once. So, I guess I will need to go look at the inbox up there periodically and clean out these crap emails that fetchmail can't download. 

Or isn't that what 'flush' is all about? Presumably, that should delete it next time it checks, right? (I don't want to try it until I'm pretty sure). 

I increased my polling interval as you suggested. 30 seconds was the way it was set up by default, apparently. 

jcunningham is an alias to myself, so I guess fetchmail didn't send me any messages. 

I suppose I could re-emerge these apps, but they are fresh from my Sunday update. 

I posted a bug on this one to sourceforge, so maybe someone on the project will figure out how to make it work right. 

[edit]: I just tried running fetchmail from the command line with -v and it returned the following:

fetchmail: incorrect header line found while scanning headers

fetchmail: SMTP< 220 apollo.olympus.net ESMTP Postfix

fetchmail: SMTP> EHLO localhost

fetchmail: SMTP< 250-apollo.olympus.net

fetchmail: SMTP< 250-PIPELINING

fetchmail: SMTP< 250-SIZE 10240000

fetchmail: SMTP< 250-VRFY

fetchmail: SMTP< 250-ETRN

fetchmail: SMTP< 250-XVERP

fetchmail: SMTP< 250 8BITMIME

fetchmail: SMTP> MAIL FROM:<adlung7@hotmail.com> SIZE=3287

fetchmail: SMTP< 250 Ok

fetchmail: SMTP> RCPT TO:<root@localhost>

fetchmail: SMTP< 250 Ok

fetchmail: SMTP> DATA

fetchmail: SMTP< 354 End data with <CR><LF>.<CR><LF>

#fetchmail: message delimiter found while scanning headers

fetchmail: SMTP>. (EOM)

fetchmail: SMTP< 250 Ok: queued as A9689C2BD3

 flushed

fetchmail: POP3> DELE 1

fetchmail: POP3< Status: RO

fetchmail: POP3> QUIT

fetchmail: POP3< X-UIDL: PxzQ8dHkIbcq1gE

fetchmail: client/server protocol error while fetching from getmail.cunningham.net

fetchmail: 6.2.3 querying getmail.cunningham.net (protocol POP3) at Thu, 31 Jul 2003 18:38:13 -0700 (PDT): poll completed

fetchmail: Query status=4 (PROTOCOL)

fetchmail: normal termination, status 4

return=4

root@apollo.olympus.net: /home/jcunningham

Regards,

Jeff

----------

## BackSeat

 *jkcunningham wrote:*   

> jcunningham is an alias to myself, so I guess fetchmail didn't send me any messages. 

 OK, let's see. Fetchmail tries to deliver a mail to you, but has problems of some sort. So it does what you have told it to do in such circumstances: it tries to deliver it to jcunningham. Of course, it will have the same problem now, so it will fail again...

I would suggest that you put a real alternate recipient in as postmaster (if you really don't want it to go to postmaster). If you can't to that for some reason you may as well set postmaster to "", which will cause the mail not to be delivered, but that is no worse that what you have already done.

BS

----------

## jkcunningham

Mail gets delivered to me just fine - either as jcunningham or as postmaster aliased to jcunningham. This particular screwed up email isn't causing fetchmail to deliver mail anywhere - its rejecting it. And, its rejecting it in such away that it breaks its own subsequent attempts to read more mail from the POP3 account. Sounds like a bug to me.  But I can change the postmaster to another account just to rule it out. 

If fetchmail wanted to send me a message, I believe it would have no difficulty. 

-Jeff

----------

## fidler

I actually have the same problem, and was trying to figure out the version of fetchmail that actually worked, does anyone know...?

----------

## jkcunningham

I'm running fetchmail 6.2.3+RPA+NTLM+SDPS+SSL (according to fetchmail -V). I've managed to bypass the problem through the following changes to my .fetchmailrc:

## fetchall should work, but causes certain stalled invalid emails to sit on 

## server and prevents fetchmail from getting any emails beyond, 

## but downloads the same defective email every time it is called.

# options fetchall stripcr warnings 3600 mda '/usr/bin/procmail -f -'

options stripcr warnings 3600 mda '/usr/bin/procmail -f-'

These problem emails continue to sit on the server, but now I only see them once.  I hope this helps.

-Jeff

----------

