# Dspam problem

## bludger

I have tried following this howto as closely as possible:

http://gentoo-wiki.com/HOWTO_Spam_Filtering_with_DSPAM_and_Postfix

Whenever mails are received by the system, I see the following in /var/log/messages:

```

Oct 18 13:13:21 mailserver postfix/smtpd[10776]: initializing the server-side TLS engine

Oct 18 13:13:21 mailserver postfix/smtpd[10776]: connect from mailrelay.blah[aaa.bbb.ccc.dd]

Oct 18 13:13:21 mailserver postfix/smtpd[10776]: NOQUEUE: filter: RCPT from mailrelay.blah[aaa.bbb.ccc.dd]: <user@mydomain.com>: Recipient address triggers FILTER dspam:unix:/var/run/dspam/dspam.sock; from=<user.surname@mail-sender.com> to=<user@mydomain.com> proto=ESMTP helo=<mailrelay.blah>

Oct 18 13:13:21 mailserver postfix/smtpd[10776]: 8E54219E: client=mailrelay.blah[aaa.bbb.ccc.dd]

Oct 18 13:13:21 mailserver postfix/cleanup[10780]: 8E54219E: message-id=<OF7F6F7C7E.17990480-ONC125720B.003D01AA-C125720B.003DA0E0@noreply.mailsender.com>

Oct 18 13:13:21 mailserver postfix/qmgr[15129]: 8E54219E: from=<user.surname@mail-sender.com>, size=4362, nrcpt=1 (queue active)

Oct 18 13:13:21 mailserver postfix/smtpd[10776]: disconnect from mailrelay.blah[aaa.bbb.ccc.dd]

Oct 18 13:13:21 mailserver dspam[27784]: bailing on error 0

Oct 18 13:13:21 mailserver dspam[27784]: received invalid result (! DSR_ISSPAM || DSR_INNOCENT) : 0

Oct 18 13:13:21 mailserver dspam[27784]: process_message returned error 1953239040.  delivering.

Oct 18 13:13:21 mailserver postfix/smtpd[10783]: initializing the server-side TLS engine

Oct 18 13:13:21 mailserver postfix/smtpd[10783]: connect from localhost[127.0.0.1]

Oct 18 13:13:21 mailserver postfix/smtpd[10783]: D26C450812: client=localhost[127.0.0.1]

Oct 18 13:13:21 mailserver postfix/cleanup[10780]: D26C450812: message-id=<OF7F6F7C7E.17990480-ONC125720B.003D01AA-C125720B.003DA0E0@noreply.mailsender.com>

Oct 18 13:13:21 mailserver postfix/qmgr[15129]: D26C450812: from=<user.surname@mail-sender.com>, size=4542, nrcpt=1 (queue active)

Oct 18 13:13:21 mailserver postfix/local[10784]: D26C450812: to=<user@mydomain.com>, relay=local, delay=0, status=sent (delivered to maildir)

Oct 18 13:13:21 mailserver postfix/qmgr[15129]: D26C450812: removed

Oct 18 13:13:21 mailserver postfix/smtpd[10783]: disconnect from localhost[127.0.0.1]

Oct 18 13:13:21 mailserver postfix/lmtp[10781]: 8E54219E: to=<user@mydomain.com>, relay=/var/run/dspam/dspam.sock[/var/run/dspam/dspam.sock], delay=0, status=sent (250 2.6.0 <user@mydomain.com> Message accepted for delivery)

Oct 18 13:13:21 mailserver postfix/qmgr[15129]: 8E54219E: removed
```

All mail is delivered, nothing is filtered.  In the dspam web interface I see nothing, no spam or ham-count.

Any clues?

----------

## magic919

 *Quote:*   

> 
> 
> Oct 18 13:13:21 mailserver dspam[27784]: bailing on error 0
> 
> Oct 18 13:13:21 mailserver dspam[27784]: received invalid result (! DSR_ISSPAM || DSR_INNOCENT) : 0
> ...

 

Usually a permissions problem.  Have you done the tidy ups at the bottom of the HOWTO?

----------

## bludger

Thanks for the reply. I re-did the steps at the end of the howto and still get the same error.  Hmm.

Any idea what this message means?

----------

## magic919

It means the response it got from DSPAM was neither it is nor it is not spam.

Usual causes are ownership/permissions but you could check you have added the trusted user.  I'm thinking you are processing as user filter.

Did you build DPSAM with debug?  If not, then do so for further clues.

----------

## bludger

I get the following when I enable debug:

```
16377: [10/25/2006 07:52:20] parsing message body

16377: [10/25/2006 07:52:20] _ds_getall_spamrecords: unable to _mysql_drv_getpwnam(user@mydomain.org)

16377: [10/25/2006 07:52:20] _ds_getall_spamrecords() failed

16377: [10/25/2006 07:52:20] _mysql_drv_get_spamtotals: unable to _mysql_drv_getpwnam(user@mydomain.org)

16377: [10/25/2006 07:52:20] delivering message

16377: [10/25/2006 07:52:20] Establishing connection to 127.0.0.1:10025

```

Strangely, I set up a similar configuration on a test system and it works without a problem.

http://dspamwiki.expass.de/Installation/Postfix/NealesSetup says that a virtual file has to be set up, but his setup seems to be slightly different to the one in the gentoo wiki.  

Any ideas?

----------

## bludger

Ok I found my problem. I had a typo in dspam.conf.  I had 

ServerParameters       "--user-filter --deliver=innocent" instead of "--user filter"

Now the emails seem to being filtered.   Now to test whether training works.

Thanks for your help.

----------

## bludger

Ok the mail is being delivered via dspam, but when I forward some spam to spam@mydomain.org it gets delivered to the ~dspam/.maildir directory, but is not counted by the spamfilter.  Is the line in .procmailrc correct:

```
| /usr/bin/dspam --user dspam --class=spam --source=error
```

Should it not be "--user filter"?   If not, what else could be the problem?

----------

## bludger

It seems to me that it is not even using procmail to deliver to the dspam user.

/var/log/messages:

Oct 25 19:01:07 modoserver postfix/qmgr[20712]: F087F5347C: from=<user@mydomain.org>, size=4821, nrcpt=1 (queue active)

Oct 25 19:01:07 modoserver postfix/local[21595]: F087F5347C: to=<dspam@mydomain.org>, orig_to=<spam@mydomain.org>, relay=local, delay=1, status

=sent (delivered to maildir)

This seems to be using the postfix "local" mail delivery program.  How can I tell it to pipe the mail through dspam for this user, without converting the whole system over to procmail?

----------

## magic919

You can generally stick in the home dir a .forward file to invoke procmail

```

     "|IFS=' ';exec /usr/bin/procmail"

```

Or just set the retraining command as an alias of spam@ in Postfix.

----------

## bludger

Is that then an error/shortcoming in the wiki?

----------

## bludger

Ok, I am getting closer now. I entered what you suggested in ".forward" and now get the following in ~dspam/procmail.log:

procmail: [27819] Thu Oct 26 22:28:20 2006

procmail: Assigning "LOGABSTRACT=all"

procmail: Assigning "MAILDIR=/var/spool/dspam/.maildir"

procmail: Assigning "DEFAULT=/var/spool/dspam/.maildir/"

procmail: Executing "/usr/bin/dspam,--user,dspam,--class=spam,--source=error"

procmail: Assigning "LASTFOLDER=/usr/bin/dspam --user dspam --class=spam --source=error"

procmail: Notified comsat: "dspam@:/usr/bin/dspam --user dspam --class=spam --source=error"

From user@myserver.com  Thu Oct 26 22:28:20 2006

 Subject: [Fwd: Re: VlhAGRA]

  Folder: /usr/bin/dspam --user dspam --class=spam --source=error          2295

27820: [10/26/2006 22:28:20] Signature retrieval for '4540d50a262975965111896' failed

27820: [10/26/2006 22:28:20] Unable to find a valid signature. Aborting.

What could this mean?

----------

## bludger

After a bit of googling, it seems that the signature cannot be found in the db.  I searched for a recent message and found:

```
select uid,signature from dspam_signature_data where signature = '454f6e2566535965111896';

+------+------------------------+

| uid  | signature              |

+------+------------------------+

| 1003 | 454f6e2566535965111896 |

+------+------------------------+
```

1003 is the user "spamfilter" (instead of filter as in the wiki).  Now when spam is being retrained, it is retrained with the command 

```
/usr/bin/dspam --user dspam --class=spam --source=error
```

Could it be that this is the wrong user and "spamfilter" (or "filter") is required here?

Another thing I might try is to enable MySQLUIDInSignature.  This causes '1003, 454f6e2566535965111896' instead of just the signature to be stored in the message.  

Does the howto need correcting here, or did am I missing something again?  I find all of these various users rather confusing.

----------

## magic919

You need to match the retrain user to the spam user, for sure.  They must be the same for it to work or it looks for spam under wrong user.

I'll take a look at the WiKi later on.

----------

## magic919

Have corrected the user in the HOWTO.  Originally was dspam and then I'd changed it...

These days I mostly use IMAP (Dovecot) and have a folder called Spam for each user.  Then I run a cron job script to retrain any emails found there and then delete them.

----------

## bludger

Now I get:

```
procmail: Executing "/usr/bin/dspam,--user,spamfilter,--class=spam,--source=error"

8928: [11/08/2006 09:37:21] Option --user requires special privileges when user does not match current user, e.

g.. root or Trusted User [uid=26(dspam)]
```

(spamfilter=filter in your case). Do I have to add "Trust dspam" to /etc/mail/dspam/dspam.conf?  

BTW, I like your idea of automatically forwarding the mails from a spam folder.  It would be great if you could also post this script to the forum.

Thanks for all of your help.

----------

## magic919

The user you are using must be put down as the Trusted user in the DSPAM conf file So Trust spamfilter in your case.

The script - note this is one I found rather than wrote.  I run it from a cronjob.  Folder must be called Spam and messages must be read messages.

```

0,15,30,45 * * * * /usr/sbin/dspam_retrain.sh -d=/home -u=YOUR_SPAM_USER -s=Spam -i=false

```

```

#!/usr/bin/perl

#

# Train DSPAM from imap folders

# Norman Maurer <nm@byteaction.de> or <nm@spam-box.de>

##################################################

use Getopt::Mixed;

Getopt::Mixed::getOptions("d:s h v u:s s:s i:s user>u domain-dir>d spam-dir>s innocent-dir>i help>h verbose>v");

my $spam_dir = $opt_s;

my $innocent_dir = $opt_i;

my $domain_dir = $opt_d;

my $user = $opt_u;

my $spam_opts = "--class=spam --source=error";

my $innocent_opts = "--class=innocent --source=error";

my $spam_count = 0;

my $innocent_count = 0;

&help if (defined $opt_h || !defined $opt_u || !defined $opt_s || !defined $opt_i || !defined $opt_d);

&train_spam;

&train_innocent;

exit 0;

sub help

{

        print "\nUsage:\n";

        print "-d\t--domain-dir\tdirecotry where the domains are keept.\n";

        print "-u\t--user\tuser which should use for train dspam.\n";

        print "-s\t--spam-dir\tname of the directory where the users store their missing spam\n";

        print "-i\t--innocent-dir\tname of the firectory where the users store their false positives\n";

        print "-v\t--verbose\tgive verbose output\n";

        print "-h\t--help\tshow this help\n";

        print "\n";

        print "Example:\n";

        print "$0 -d=/var/qmail/vpopmail/domains -u=byteaction.de -s=spam-missing -i=false-positive -v\n\n";

        exit 0;

}

sub train_spam

{

        my @spam_array = split(/\n/, `find $domain_dir -regex '.*\.$spam_dir/cur/.*'`);

        foreach my $spam (@spam_array)

        {

                system("dspam $spam_opts --user $opt_u --client < $spam");

                system("rm -f $spam");

                $spam_count++;

        }

        if (defined $opt_v)

        {

                print "$0: Trained $spam_count missed spams\n";

        }

}

sub train_innocent

{

        my @innocent_array = split(/\n/, `find $domain_dir -regex '.*\.$innocent_dir/cur/.*'`);

        foreach my $innocent (@innocent_array)

        {

                system("dspam $innocent_opts --user $opt_u --client < $innocent");

                system("rm -f $innocent");

                $innocent_count++;

        }

        if (defined $opt_v)

        {

                print "$0: Trained $innocent_count false positives\n";

        }

}

```

----------

## bludger

Thanks for all of your help. I got it very close to working and then realised that with this configuration you have to have the spam signature in the message body and not in the headers (as wanted by my users).  I also found the system of logging on to the web server rather cumbersome.  

I have thus switched to what is for me a much simpler setup.  I followed a similar setup to http://www.lalkaka.com/dspam/ or http://splodge.fluff.org/docs/single-user-dspam.  Procmail is set up for each user to pipe through dspam before delivery.  If spams are found, they are put in the "spam" folder. The user trains by dragging and dropping to a "retrain-spam" or "retrain-nonspam" folder, from which they are automatically processed and then moved as appropriate.  

To me this was a much simpler and cleaner solution. Perhaps it is more difficult for a larger setup though.  Thanks a lot anyway for all your help.

----------

## magic919

I'm glad you found a method that works for you.  There are many ways to implement it and it's good you stuck with DSPAM.

----------

