# net-www/dspam-web-3.1.0 config steps to access web app?

## jeffk

I've emerged net-www/dspam-web-3.1.0 and dependencies. I want to check out the console a little, and I'm up against that wall of not-knowing-what-webapp-config-ebuilds-do. It doesn't appear that there are any steps connecting the cgi application in

/usr/share/webapps/dspam-web/3.1.0 

The einfo message:

```
 * The CGIs need to be executed as group dspam in order to write

 * to the dspam data directory. You will need to configure apache

 * manually to do this. Another option is to add the user apache

 * to the dspam group. You can do this automatically by running:

 * ebuild /var/db/pkg/net-www/dspam-web-3.1.0/dspam-web-3.1.0.ebuild config

 * This app requires basic auth in order to operate properly.

 * You will need to add dspam users to the .htpasswd file or

 * configure a different authentication mechanism for the user

 * accounts.

 * Caching service dependencies...

 *  Service 'sendmail' already provide 'mta'!;

 *  Not adding service 'postfix'...
```

is helpful, and I did perform the ebuild config step. What URL will this CGI app be found at on the server, and what steps are necessary to configure apache to point to it? Thanks.

----------

## steveb

normaly it should be installed under the /dspam on the server. i don't know what steps are needed to configure the application, since i have not used the ebuild for dspam web interface (i installed it manually).

cheers

SteveB

----------

## mallchin

It should, but it isn't. Looks like the ebuild is broken?

----------

## steveb

 *mallchin wrote:*   

> It should, but it isn't. Looks like the ebuild is broken?

 then fill in a bug report at bugs.gentoo.org

cheers

SteveB

----------

## mallchin

I will shortly, along with some other dspam related bugs I want to post.

A simple symlink from /var/www/localhost/dspam -> /usr/share/webapps/dspam-web/3.1.0/htdocs/ and editing .htacess fixes the web component.

Has anyone had problems with dspam & mysql? Mine doesn't shove any data into the tables?...

----------

## steveb

 *mallchin wrote:*   

> I will shortly, along with some other dspam related bugs I want to post.
> 
> A simple symlink from /var/www/localhost/dspam -> /usr/share/webapps/dspam-web/3.1.0/htdocs/ and editing .htacess fixes the web component.
> 
> Has anyone had problems with dspam & mysql? Mine doesn't shove any data into the tables?...

 YES! I had massive troubles with DSPAM and MySQL!

switched after the troubles to postgresql and I am very happy with it. currently i have over 200'000 spam messages feeded into the corpus and postgresql is still working without any troubles. vacuuming the data takes some time, but it works. in mysql I could not optimize the tables. it was to much data. anyway... this is currently the output for the globaluser i created:

```
 / # dspam_stats -H globaluser

globaluser:

                TS Total Spam:                  0

                TI Total Innocent:              0

                SM Spam Misclassified:          0

                IM Innocent Misclassified:      0

                SC Spam Corpusfed:         232855

                IC Innocent Corpusfed:      35331

                TL Training Left:               0

 / #    
```

i have as well submited some info to bugs.gentoo.org: bug #57838

the package maintainer and me have not the same viewpoint on some topics, but that's normal  :Wink: 

cheers

SteveB

btw: if dspam is not writing any data to your mysql database, then something is wrong with your setup. did you configure dspam to use mysql and did you change the config files?

----------

## mallchin

Hehe. I guess we can't all see eye to eye  :Smile: 

Yep; I emerged dspam from portage like this...

```

[ebuild   R   ] mail-filter/dspam-3.1.0  -cyrus -debug -exim -maildrop +mysql -neural -postgres -procmail  0 kB

```

...and think I have everything configured correctly. I've checked locations and permissions of important files and can use dspam_stats to generate this for my user...

```

martin:

                TS Total Spam:                  0

                TI Total Innocent:              0

                SM Spam Misclassified:          0

                IM Innocent Misclassified:      0

                SC Spam Corpusfed:              0

                IC Innocent Corpusfed:          0

                TL Training Left:            2500

```

...but as you can see it doesn't seem to have done much yet, also dspam_stats breaks if I change the settings in mysql.data so I am assuming it can connect to mysql okay, but the tables are all blank.

I think I have the mysql user and tables setup with correct permissions so I'm confused   :Rolling Eyes: 

I see a lot of reports of people using the download from the site and success reports, maybe the emerge does something differently?

----------

## steveb

yes. the ebuild is much diffrend then the the way described in the diffrend how-to's or the other guide lines you find on the net.

but the ebuild is not wrong or does not install dspam in such a way that it does not work.

the ebuild just installs the stuff diffrendly. that's all  :Wink: 

anyway... what exactly is your problem with the dspam ebuild? did you do all the steps needed to finish the emerge process?

cheers

SteveB

----------

## mallchin

Checked out your ebuild, nice job  :Smile: 

I used that instead and have successfully logged data to mysql, and can read it back using dspam_stats as follows:

```

martin@dopesmoker.net:

                TS Total Spam:                 54

                TI Total Innocent:             17

                SM Spam Misclassified:          0

                IM Innocent Misclassified:      0

                SC Spam Corpusfed:              0

                IC Innocent Corpusfed:          0

                TL Training Left:            2483

```

My two niggles now are...

1) How is spam delivered?

I am using maildrop under courier to call dspam , this seems to let proper mail through and it arrives in my Inbox, and has a section on deliverying spam to .Spam, but it doesn't seem to appear. I am using the maildroprc script here (modified below):

```

import RECIPIENT

import SENDER

import HOME

if ($SENDER ne "")

{

        FROM=$SENDER

}

else

{

        FROM="unknown"

}

CLEAN_FROM=escape($FROM);

# Spam Filter

xfilter "/usr/bin/dspam --user $RECIPIENT --mode=teft"

if (/^X-DSPAM-Result:/:h)

{

}

else

{

        xfilter "/usr/bin/spamc -u $RECIPIENT"

}

if (/^X-Spam-Status: Yes/:h || /^X-DSPAM-Result: Spam/:h)

{

        # See if the Spam folder exists.

        `test -d "./Maildir/.Spam/`

        if ($RETURNCODE != 0)

        {

                # If not, copy one from the pre-existing skel directory.

                `cp -Rp /usr/local/etc/courier/skel/Maildir/.Spam ./Maildir/`

        }

        to "./Maildir/.Spam/."

}

```

2) dspam cgi doesn't show any data

I've copied over the cgi script to my server and setup .htaccess and .htpasswd, but when I login it doesn't show any data.  :Sad: 

----------

## steveb

you need to edit the various cgi scripts. everything inbetween Configuration and End Configuration needs maybe to be adjusted. my admin.cgi config looks like this:

```
## Configuration

#

$CONFIG{'DSPAM_HOME'}   = "/var/dspam";

$CONFIG{'ME'}           = "admin.cgi";

$CONFIG{'TEMPLATES'}    = "./templates";        # Location of HTML templates

$CONFIG{'ALL_PROCS'}    = "ps -deaf";

$CONFIG{'PREFIX'}       = "/usr";               # DSPAM prefix

$CONFIG{'LARGE_SCALE'}  = 1;

$CONFIG{'DOMAIN_SCALE'} = 0;

#

## End Configuration
```

and my dspam.cgi looks like this:

```
## Configuration

#

$CONFIG{'DSPAM_HOME'}   = "/var/dspam";

$CONFIG{'ME'}           = "dspam.cgi";

$CONFIG{'DSPAM'}        = "/usr/bin/dspam";

$CONFIG{'DSPAM_ARGS'}   = "--deliver=innocent --mode=teft --class=innocent " .

                          "--source=error --user $ENV{'REMOTE_USER'} -d %u";

$CONFIG{'LARGE_SCALE'}  = 1;                    # --enable-large-scale

$CONFIG{'DOMAIN_SCALE'} = 0;                    # --enable-domain-scale

$CONFIG{'TEMPLATES'}    = "./templates";        # Location of HTML templates

#

## End Configuration
```

i have as well modified the templates to show to the enduser where to send mail for spam and not-spam. but this is another storry.

cheers

SteveB

----------

## steveb

btw: because i don't like to update .htaccess and the password all the time i have written for me a cron job to do it:

```
#!/bin/bash

db_user="mailsql"

db_password="secret"

db_name="mailsql"

db_table="users"

db_field="\`email\` , \`clear\`"

db_where="and \`access\` = 'y' AND \`postfix\` = 'y'"

dspam_htaccess_path="/var/www/localhost/htdocs/dspam/htpasswd"

dspam_htaccess_tmp="/tmp/a$$.$$.$$"

dspam_htaccess_tmp_sorted="/tmp/b$$.$$.$$"

mysql -h localhost -u ${db_user} --password=${db_password} -e "select ${db_field} from ${db_table} where 1 ${db_where} order by ${db_field} desc" ${db_name}|grep -i "\."|while true

do

        read sql_result

        [[ "${sql_result}" == "" ]] && break

        email_address="$(echo ${sql_result}|sed "s/^\([^ ]*\)[ ]*\(.*\)$/\1/g")"

        email_password="$(echo ${sql_result}|sed "s/^\([^ ]*\)[ ]*\(.*\)$/\2/g")"

        htpasswd2 -nb ${email_address} ${email_password} 1>>${dspam_htaccess_tmp} 2>&1

done

if [[ -f "${dspam_htaccess_tmp}" ]]

then

        htpasswd2 -nb MyUltraSecretAdminAccount secretAdminPassword 1>>${dspam_htaccess_tmp} 2>&1

        htpasswd2 -nb root secretRootPassword 1>>${dspam_htaccess_tmp} 2>&1

        htpasswd2 -nb globaluser secretGlobalUserPassword 1>>${dspam_htaccess_tmp} 2>&1

        grep -v ^$ ${dspam_htaccess_tmp} | sort | uniq>${dspam_htaccess_tmp_sorted}

        if [[ "$(diff ${dspam_htaccess_tmp_sorted} ${dspam_htaccess_path} 1>/dev/null 2>&1 ; echo $?)" != "0" ]]

        then

                mv -f ${dspam_htaccess_tmp_sorted} ${dspam_htaccess_path} 1>/dev/null 2>&1

        fi

        [[ -f "${dspam_htaccess_tmp}" ]] && rm -f ${dspam_htaccess_tmp_sorted} 1>/dev/null 2>&1

        [[ -f "${dspam_htaccess_tmp_sorted}" ]] && rm -f ${dspam_htaccess_tmp} 1>/dev/null 2>&1

fi

exit 0
```

i know... i know... it is quick and dirty, but i did not wanted to invest to much time.

cheers

SteveB

----------

## mallchin

Thanks.

Mine seems a little different and includes most options from configure.pl; My adming.cgi looks like this:

```

# Read configuration parameters common to all CGI scripts

require "configure.pl";

#

# The current CGI script

#

$CONFIG{'ME'}               = "admin.cgi";

```

I've setup configure.pl like so:

```

$CONFIG{'DSPAM_HOME'}   = "/etc/mail/dspam";

$CONFIG{'DSPAM_BIN'}    = "/usr/bin";

$CONFIG{'DSPAM'}        = $CONFIG{'DSPAM_BIN'} . "/dspam";

$CONFIG{'DSPAM_STATS'}  = $CONFIG{'DSPAM_BIN'} . "/dspam_stats";

$CONFIG{'DSPAM_ARGS'}   = "--deliver=innocent --mode=tum --class=innocent " .

                          "--source=error --user $ENV{'REMOTE_USER'} -d %u";

$CONFIG{'LARGE_SCALE'}  = 0;                    # --enable-large-scale

$CONFIG{'DOMAIN_SCALE'} = 0;                    # --enable-domain-scale

$CONFIG{'TEMPLATES'}    = "./templates";        # Location of HTML templates

$CONFIG{'ALL_PROCS'}    = "ps -deaf";

$CONFIG{'MAIL_QUEUE'}   = "mailq | grep '^[0-9,A-F]' | wc -l";

# Default DSPAM display

$CONFIG{'HISTORY_SIZE'} = 200;          # Number of items in history

$CONFIG{'MAX_COL_LEN'}  = 50;           # Max chars in list columns

$CONFIG{'SORT_DEFAULT'} = "Date";       # Show quarantine by "Date" or "Rating"

$CONFIG{'3D_GRAPHS'}    = 1;

#$CONFIG{'LOCAL_DOMAIN'}        = "ganymede";

# Add customized settings below

$CONFIG{'LOCAL_DOMAIN'} = "dopesmoker.net";

# Configuration was successful

1;

```

The bin paths are okay but I don't know if it's looking for /var/spool/dspam or /etc/mail/dspam, I guess the latter as it contains default.prefs etcetera which are editable in admin.cgi.

----------

## mallchin

Like that quick and dirty hack btw  :Wink: 

I'm only using it for a few users, though I was thinking of something similar to add users on corporate installs.

----------

## steveb

btw: did you fix your maildrop problems?

i just have today quickly looked into maildrop and i am now looking at maildrop as a delivery agent instead of the setup i have right now.

i have installed maildrop but have not used it with dspam. but i would probably use something like this as my maildroprc (i have NOT tested it! this is just the script i am quickly prototyping in kwrite. maybe it works, maybe not):

```
import RECIPIENT

import SENDER

import HOME

import EXTENSION

import LOGNAME

if ( $EXTENSION eq "spam")

{

   xfilter "/usr/bin/dspam --user $LOGNAME --mode=teft --source=error --class=spam --process --feature=chained,noise,whitelist --stdout --deliver=innocent,spam"

   `test -d $HOME/.Trash/`

   if ( $RETURNCODE != 0 )

   {

      `/usr/bin/maildirmake $HOME/.Trash`

   }

   to "$HOME/.Trash/"

}

if ( $EXTENSION eq "ham")

{ 

   xfilter "/usr/bin/dspan --user $LOGNAME --mode=teft --source=error --class=innocent --process --feature=chained,noise,whitelist --stdout --deliver=innocent,spam"

   `test -d $HOME/.Trash/`

   if ( $RETURNCODE != 0 )

   {

      `/usr/bin/maildirmake $HOME/.Trash`

   }

   to "$HOME/.Trash/"

}

if ( $EXTENSION ne "spam" && $EXTENSION ne "ham" )

{

   xfilter "/usr/bin/dspam --user $LOGNAME --deliver=innocent,spam --stdout --process --mode=teft --feature=chained,noise,whitelist"

   if (/^X-DSPAM-Result: Spam/:h)

   {

      `test -d $HOME/.Spam/`

      if ( $RETURNCODE != 0 )

      {

         `/usr/bin/maildirmake $HOME/.Spam`

      }

      to "$HOME/.Spam/"

   }

}
```

right now i have modified my master.cf to have a entry for "ham-user" and "spam-user", where i reroute the spam/ham reports of the users. but if i am not completly wrong reading the documentation of maildrop, then i could save some trouble by using maildrop and filter there the spam/ham reports.

what do you think? does the above script make sense?

cheers

SteveB

----------

## mallchin

Yes thanks! I used your ebuild (nice work) and it all started to work as expected. I've done much reading and it all makes sense now  :Smile: 

Your maildroprc script looks good but how do you pass those $extensions? I see your using --source=error to train dspam on an error, so I'm guessing your feeding missed and innocents...

Here's mine...

/etc/maildroprc

```

import RECIPIENT

import SENDER

import HOME

logfile "$HOME/.maildrop.log"

# replace blank From: line

if ($SENDER ne "")

{

  FROM=$SENDER

}

else

{

  FROM="unknown"

}

CLEAN_FROM=escape($FROM);

#is recipient spam lover?

if ( $RECIPIENT ne "spam-lover@mydomain.net" )

{

  # check for spam using dspam

  xfilter "/usr/bin/dspam --user $RECIPIENT --mode=teft --feature=chained,noise,tb=0,whitelist --deliver=innocent,spam --stdout"

}

# deliver mail

if ( /^X-DSPAM-Result: Spam/)

{

  # try to deliver to SPAM maildir

  exception {

              to "$HOME/Maildir/.Spam/"

            }

}

else

{

  exception {

              to "$HOME/Maildir/"

            }

}

```

I have a spam-lover address that wants to bypass dspam, else other mail is checked, and put into .Spam if it's spam, or Inbox if it's ham.

I handle missed & innocents with two IMAP folders. I simply drag'n'drop mail into the relevant folder, then have dspam check using a cronjob:

crontab -e martin

```

*/5 * * * * find $HOME/.maildir/.Spam-innocent/cur/ -name "*" -type f -exec /etc/mail/dspam/spam_innocent.sh martin@mydomain.net {} \;

*/5 * * * * find $HOME/.maildir/.Spam-missed/cur/ -name "*" -type f -exec /etc/mail/dspam/spam_missed.sh martin@mydomain.net {} \;

```

/etc/mail/dspam/spam_innocent.sh

```

#!/bin/sh

cat $2 | dspam --user $1 --mode=teft --class=innocent --source=error --feature=chained,noise,tb=5,whitelist && mv $2 $HOME/.maildir/cur/

```

/etc/mail/dspam/spam_missed.sh

```

#!/bin/sh

cat $2 | dspam --user $1 --mode=teft --class=spam --source=error --feature=chained,noise,tb=5,whitelist && mv $2 $HOME/.maildir/.Spam/cur/

```

I need some decent corpus to train with, else its all working pretty sweet! I think maildrop is an excellent way to implement dspam and like the extra scripting available through maildroprc...

----------

## mallchin

Forgot to mention I have dspam-web working too, love those graphs hehe  :Razz: 

----------

## steveb

you could use http://www.spamarchive.org/ and download the archives from there and then feed the corpus.

to process each message from the archives, you could use mboxgrep (just emerge net-mail/mboxgrep). a script to automaticly process each *.r2.gz from a driectory could look like this:

```
#!/bin/bash

for foo in *.r2.gz

do

        echo "Processing $(mboxgrep --mailbox-format=zmbox --headers --regexp=. --count ${foo}) SPAM messages from ${foo}"

        mboxgrep --mailbox-format=zmbox --headers --regexp=. --pipe="nice --adjustment=-15 dspam --class=spam --source=corpus --inoculate --user 'globaluser' --mode=teft --feature=chained,noise" ${foo}

        echo

        echo "Runing dspam_clean -p0 globaluser"

        dspam_clean -p0 globaluser

        echo

done
```

using folders in imap to track spam/not spam is okay, but what do you do with pure pop3 users? they need another way to submit spam/not-spam messages.

cheers

SteveB

btw: i again started to change/modify the current dspam ebuild (if you want, i could post it here). and to my surprice the mysql way is not that bad, if you choose to use the speed optimized tables instead of the space optimized tables. i still need to feed serval thoused messages into the corpus, to be 100% sure about dspam and mysql. postgres on the other hand is rock solid.

----------

## mallchin

Thanks for the link, in all my searching I had yet to find a good spam test resource, great stuff!

I've tested the SA-Corpus but once I'd trained dspam it denounced all mail as spam, though I think with some more recent spam, and some ham of my own, I will get better results.

My users are using Outlook and I can move them to IMAP easily. For those with POP3 I'd suggest the forward method; It's not as neat as the drag'n'drop but equally viable. Some aliases like this should do the trick:

```

spam-martin: |/usr/bin/dspam --user martin@mydomain.net --mode=teft --class=spam --source=error --feature=chained,noise,tb=5,whitelist

fp-martin: |/usr/bin/dspam --user martin@mydomain.net --mode=teft --class=innocent --source=error --feature=chained,noise,tb=5,whitelist

```

I noticed your ebuild had made it into portage; If you've an updated one I'd be happy to try it out  :Smile: 

----------

## steveb

this is exactly what i don't want to do! aliases for each user is a hell to maintain.

about my ebuild: well.... found again a error. don't know if i had that error in my original ebuild or not, but i opend again a bug report on bugs.gentoo.org:

https://bugs.gentoo.org/show_bug.cgi?id=61564

the new ebuild i am working on allows you to add your custom --enable-xxx --disable-xxx compile flags when you emerge dspam. but the ebuild is not 100% tested. i need some more time.

cheers

SteveB

----------

## steveb

btw: i have set up postfix to handle spam/ham messages without the need to specify for each user a alias. if you want me, i could post the way i did it (it is not done with maildrop, but maybe i will replace it with maildrop).

cheers

SteveB

----------

## mallchin

Do you want to train each user individually?

You could always use globaluser to incorporate missed/innocents, but I guess maildrop is probably the best solution. You could check to see if the recipient started spam-/fp- and then pipe to dspam instead...

----------

## Snaketails

Hi, I have a question for mallchin

Why in those scripts you have posted, are you moving the SPAM back into the .maildir ?  I can imagine the non-spam getting returned, but why the spam too?

 *mallchin wrote:*   

> 
> 
> I handle missed & innocents with two IMAP folders. I simply drag'n'drop mail into the relevant folder, then have dspam check using a cronjob:
> 
> /etc/mail/dspam/spam_innocent.sh
> ...

 

----------

## tomaw

 *Snaketails wrote:*   

> Hi, I have a question for mallchin
> 
> Why in those scripts you have posted, are you moving the SPAM back into the .maildir ?  I can imagine the non-spam getting returned, but why the spam too?
> 
> 

 

It doesn't.  spam_innocent.sh moves ham mails that were classified as spam back into the inbox after dspam training.  spam_missed.sh moves false negatives into the Spam folder after dspam training.

----------

