# dspam-web and virtual users [SOLVED]

## NotExcessive

I've finally been able to get a virtual mail server up and running with Postfix, Dovecot, MySQL, Maildrop, Dspam, Dspam-web, and squirrelmail. I've followed the instructions at http://gentoo-wiki.com/HOWTO_Spam_Filtering_with_DSPAM_and_Postfix, spent a few days head-banging getting problems sorted out (https://forums.gentoo.org/viewtopic-t-490464-highlight-.html), and finally have a spiffy system.

With one small niggly, that might not be that small.

When in the quarantine page of the gui, you can check a message and tell dspam to deliver it anyway. When I did this, the quarantine count decreased, but the message never appeared. Looking at the log files, I see

```
Aug 25 15:35:16 [postfix/smtpd] disconnect from unknown[127.0.0.1]

Aug 25 15:35:17 [dspam] Option --user requires special privileges when user does not match current user, e.g.. root or Trusted User [uid=65534(nobody)]

Aug 25 15:35:17 [dspam] Unable to initialize agent context

Aug 25 15:35:17 [postfix/local] CC520EE3A9: to=<filter@mailserver.mydomain.net>, orig_to=<filter>,relay=local, delay=1, status=SOFTBOUNCE (Command died with status 1: "/usr/bin/dspam --process --class=spam --source=error --user filter". Command output: 19888: [08/25/2006

15:35:17] Option --user requires special privileges when user does not match current user,e.g.. root or Trusted User [uid=65534(nobody)] 19888: 08/25/2006 15:35:17] Unable to initialize agent context )
```

So the cgi runs as nobody and dspam doesn't want to play. Fair enough, so I add "Trust nobody" to dspam.conf and that problem goes away.

Now doing this again, I see that the email leaves quarantine but still doesn't arrive at the original intended destination. Looking at the log file again:

```
Aug 25 16:32:12 [postfix/local] CC520EE3A9:to=<filter@mailserver.mydomain.net>, orig_to=<filter>, relay=local, delay=3416, status=sent (delivered to command: /usr/bin/dspam --process --class=spam --source=error --user filter)

Aug 25 16:32:12 [postfix/qmgr] CC520EE3A9: removed
```

Whoops - it's trying to use Postfix's local, which of course won't do anything for a virtual user stored in a MySQL database, so after changing local_transport = local to local_transport = maildrop in /etc/postfix/main.cf, I try again.

Looking at the log:

```
Aug 25 16:46:15 [postfix/pipe] EBEFB23460: to=<filter@mailserver.mydomain.net>, orig_to=<filter>, relay=maildrop, delay=1, status=deferred (temporary failure. Command output: /usr/bin/maildrop: Temporary authentication failure. ) 
```

It then dawned on me what was wrong. Seeing as how dspam is being used as a content filter before any delivery takes place, it only knows user "filter" to store its data under. All emails for all virtual users in all virtual domains get processed under the one solitary user, "filter". 

No problem, except when it comes time to say "Hey this is a false positive, it's not spam and shouldn't be here in quarantine! Deliver it anyway". The email will then be sent to user "filter", and after maildrop tries to do the authentication against courier-authlib, will attempt to deliver it to a non-existent user filter@mydomain.net where, of course, the thing will fall over.

Not good. What the cgi needs to do is send the mail onto the appropriate virtual user, which means that maildrop needs to be fed user@domain.tld for whichever user the original recipient was supposed to be.

So the question I'm asking is: is there something in the configuration of dspam-web that I've missed, that will read the recipient field out of the quarantined email so that when it passes the email onto the local transport agent (maildrop in my case as I have virtual users), it has the correct address, or... is it the case that the cgi has to be modified in order for it to be able to do this?

In other words, has the scenario of a totally virtual server running a single dspam user for content filtering at the beginning of the mail loop been overlooked? I really don't think so, maybe somebody out there knows of a solution to this. I'm sure it's not  a new problem and just my lack of experience playing with mail server setups.Last edited by NotExcessive on Mon Aug 28, 2006 9:01 am; edited 1 time in total

----------

## magic919

Didn't want you to think I'd ignored or overlooked this.  With about 1 FP in 1000 messages it's just not pressing enough for me.  I release the message and it delivers to my local user filter.  I use Mutt and send it on to the intended recipient.

For the fully virtual set-up you'd need t run DSPAM as the virtual user and not the one user, filter.  You could look at using DSPAM groups to just have the one shared database.  Then you'll get a number of quarantines - 1 per user.  If Filter was an admin user it could review the individual quarantines and it would release to the appropriate virtual user.

I'll be interested to see if anyone has a better plan though.  Maybe SteveB will drop by.  He does plenty of DSPAM.

----------

## NotExcessive

I was looking at the option of modifying dspam.cgi so that it reads the "for <user@domain.tld>" string out of the email it's about to release, and then send that information out to maildrop. This is a bit of a mixed fudge (after all, one, real, user for the web interface to run under, but quarantine releases are sent to virtual users) but it's probably less work than my trying to get dspam running as the virtual user.

Purer approach, running it virtually, but hell, it took all this effort just to get it to run as a single real user!

Unless of course you've got a 5-minute-quickie on how to convert what I have dspam set up as now to a virtual user.        :Very Happy: 

----------

## magic919

Hmmm.  You can rebuild DSPAM with virtual-users in USE.  This creates an extra table in the database for the dspam_virtual_uids.  The 'old' stuff in the database seems to vanish when you convert if you build and then run emerge config.

Then you add filter to the admins if you haven't already.

Edit dspam.conf 

```

ServerParameters        "--user filter --deliver=innocent"
```

becomes

```

ServerParameters        "--deliver=innocent"
```

That's it.

You'll have as many quarantines as virtual users.  But you can wander around them all as filter is an admin.

Tony

----------

## NotExcessive

Ah. I already compiled dspam with virtual-users set from the beginning, and so the table dspam_virtual_uids is already in there with just one user, 'filter', as UID #1. So all I've done then is changed the line in dspam.conf to 

```
#ServerParameters        "--user filter --deliver=innocent"

ServerParameters        "--deliver=innocent"
```

 and restarted.

I'll see what happens now with the web interface: whether if you change to a virtual user, it releases to that same virtual user. If that's the case then of course the problem goes away.

I'll let you know how I get on.

----------

## magic919

I'll be watching with interest  :Smile: 

----------

## NotExcessive

OK, that's odd. With the changes I've just made, yes you can see the virtual users in the "User Statistics" tab, but now dspam is totally ineffective at stopping spam. I did the usual by sending junk from outside, moving it to the spam folder (for the respective user), running the cron job, and sending the email again to see what happens.

When I just had the single, real user, 'filter', I'd only have to tell it twice that the message was spam. On the third go, it'd grab it. Now, after repeating it over 6 times, it just lets it through.

Looking at the "User Statistics" tab:

```
Name                 Q.Size     TP     TN     FP     FN     SC     IC     Mode     On Spam     BNR Whitelist     Sed     Sig Loc

dean@mydomain1.net   --         0      1      0      0      0      0      TEFT    Quarantine    ON    ON          5       headers

filter               --         1    258      6      14     0      4      TEFT    Quarantine    ON    ON          5       headers

fred@mydomain2.net   --         0      9      0      0      0      0      TEFT    Quarantine    ON    ON          5       headers

Total    0.0 KB                 1    268     14      6      0      4                              
```

'filter' has a TP of 1, which is the number of emails it currently has in quarantine, which is correct. TN=258 is also correct - number of emails through that are OK.

Now looking at fred, every time I tell it that's spam and resend the message, TN goes up, and TP never gets incremented: dspam is now allowing everything through - retraining it by dropping the email into the spam folder and running the CRON job is telling it it's OK?   :Confused: 

Why has it suddenly gone arse- backwards?

----------

## magic919

You need to think about what's happening here.  You now have new users with no spam database.  You are retraining messages but using a cron job with user filter - not the user that got the spam message.

Look at sticking them in a group so they all use the filter training data.

----------

## NotExcessive

I'm not sure what's happening here - it's still broken. I created a group file /var/spool/dspam/group and placed the following line in it:

```
group1:shared:filter,*@domain1.net,*@domain2.com,*@domain3.net
```

filter is the real user and is the one that's working fine from the original configuration. The User Stats tab now looks like this:

```
The following table shows the number of messages processed for each user along with their current preference settings.

Name    Q.Size    TP    TN    FP    FN    SC    IC    Mode    On Spam    BNR    Whitelist    Sed    Sig Loc

dean@domain1.net    --    0    18    0    0    0    0    TEFT    Quarantine    ON    ON    5    headers

filter    --    1    258    6    14    0    4    TEFT    Quarantine    ON    ON    5    headers

group1    --    0    1    0    6    0    0    TEFT    Quarantine    ON    ON    5    headers

fred@domain3.net    --    0    9    0    0    0    0    TEFT    Quarantine    ON    ON    5    headers

Total    0.0 KB    1    286    20    6    0    4                              
```

I have these settings for all users and the group:

TEFT 	Quarantine 	ON 	ON 	5 	headers

After spam arrives at the inbox:

```
Name     Q.Size     TP     TN     FP     FN     SC     IC

group1    --         0      1      0      5      0      0
```

After I run the CRON job:

```
Name     Q.Size     TP     TN     FP     FN     SC     IC

group1    --         0      0      0      6      0      0

```

When the next spam arrives at the inbox:

```
Name     Q.Size     TP     TN     FP     FN     SC     IC

group1    --         0      1      0      6      0      0
```

So every time spam arrives, TN goes up by one. I retrain, TN goes down by one, FN up by one (so this seems to be what it should be doing) but it still doesn't block the spam. Only the statistics for group1 change - all others stay static. I thought that by declaring this group, all users would use filter's data folder, but  I notice there's a new folder under /data/local/group1 but there's no mbox as nothing's being trapped.

Do I need to redefine the group in any way? Should I keep my CRON job as /usr/sbin/dspam_retrain.sh -d=/vmail/domains -u=filter -s=Spam -i=false -v  ?

What I'll try is nuke the dspam database, re-create it, and see what happens.

----------

## NotExcessive

Nuked database, reconfigured, virtual_id_table just has group1 and filter in it (tried running CRON under both, no difference).

Whenever I retrain it from CRON, I see this:

```
Name     Q.Size     TP     TN     FP     FN     SC     IC     Mode     On Spam     BNR     Whitelist     Sed     Sig Loc

group1    --         0      2      0      6      0      0    TEFT    Quarantine    ON    ON    5    headers

Total    0.0 KB      0      2      6      0      0      0    
```

TN for group1 and total agree, but values for FP and FN are transposed.

----------

## NotExcessive

Hoooooookay. We're up and running. Leave the CRON job as user "filter". Go into /data/local/group1, do a touch group1.mbox, set chown to mail:root, set chmod to 660, restart dspam. Looks like I was bitten by the no-permissions-no-reporting-errors-nyah-nyah bug (technical term, patent applied for) again. 

As soon as I did that to the group1.mbox file, I restarted, and it seems to work. Retrain once, catches identical spam on the very next go. All users now have their own data structures in /data/domain.tld/user, and we also have /data/local/filter and /data/local/group1.

Log into the web interface as any user@domain.tld, you can see your individual quarantine (which does work as it does catch spam) and if you mark for release, then the email does go to the user's mailbox, so no worries now about release any more (I decided not to make group1:share, managed)

Whew. 

Now. Questions about cleaning this up, because I've probably got shrapnel spread all over the place, mucking around, and I'm not sure which bits are needed and which bits aren't.

1) For this setup as it is now, do I need to keep /data/local/filter and /data/local/group1 at all, because every user has their own .mbox instead?

2) If I decide to change over to "managed quarantines" (group1:share, managed) which .mbox gets used? filter's or group1's?

3) Which user is the CRON job supposed to run as, for both (1) and (2)? I suppose that affects the choice of .mbox in (2)?

4) Table dspam_token_data in database dspam: UID is set by the user name set in the CRON job? In other words, if I change the CRON user, does this mean the thing throws its existing knowledge out the window?

Asking the above because I don't want to break anything, but I'd like to clean out the unused stuff (if there is any) at the same time.

----------

## NotExcessive

Never mind, I had an attack of "I gotta see if I can do it over again without breaking it."

Dumped the entire database. Nuked all data structure directories. Edited the file group to be 

```
global:shared,managed:filter,*@domain1.net,*@domain2.com,*@domain3.net
```

 so I can surf all accounts without having to log in as all the different users (that was working on the last round anyway).

Did emerge --config =dspam-3.6.6, went through the motions. Restart everything.

Sent spam, retrained several times. Numbers on stats page change but never see anything for TP. Hence it's not catching anything - all mail gets through.

Look at /var/spool/dspam/data/local/global. All the files are there but there's nothing about global.mbox. 

Do:

```
touch global.mbox
```

Sent spam, retrained twice. On the third send, it got snapped up. Check quarantine for user global, it's there. Click on message to read - no problems. Check and release - permission denied error. The cgi can't write back into global.mbox. 

Do:

```
chmod 660 global.mbox
```

And try to release from quarantine again. No problems. Try with different users, all go into same mbox, and all get delivered properly when released. Previously thought I had to chown global.mbox to mail:root but it seems happy as root:root so leaving that as it is.

Everything seems to work. At last, we're DONE.

I'm documenting all my trials and tribulations in case it helps anyone that might come across this thread. I've learned a hell of a lot, as well as the fact that if you create a group, you've got to go in and touch the .mbox file for spam trapping to happen at all (never mind what the stats tell you) and chmod it to 660 for dspam-web to release properly. Oddly enough when I had individual /domain.tld/user directories from my last experiment, the respective .mboxes got created and populated without any manual intervention.

Something to watch out for.

The only questions I have left are: should I remove user filter from the group definition, and ditto from the dspam and cron setups.

----------

## magic919

Sounds like you have been having 'fun'.  Glad to see it's all under control.

I'm always having to follow up changes with a quick chown and/or chmod as ownership and permissions just keeps tripping up the DSPAM set-up.  I'd get rid of filter now you are using global as the main user.

----------

## NotExcessive

OK so my CRON job will be /usr/sbin/dspam_retrain.sh -d=/vmail/domains -u=global -s=Spam -i=false -v , right?

But I'll still need to keep /home/filter as the cgi files have to live somewhere.

----------

## magic919

Sounds good on both counts.  Bear in mind you are the test pilot/crash test dummy on this one.  My virtual set-up is a step behind your one.

----------

