# awstats + postfix

## pelacables

Hi,

I'm having problems on setting awstats to display postfix graphs.

I followed http://awstats.sourceforge.net/docs/awstats_faq.html#MAIL to configure awstats, but I have a problem:

Awstats provides a script (/usr/bin/maillogconvert.pl) which converts mail log format to standard format that is comprensible by awstats.

My problem comes from this script, which is not converting anything:

here's an example:

#perl /usr/bin/maillogconvert.pl standard < /var/log/mail/test

#

It returns nothing. (oh! test is a file copy of mail log).

If I execute awstats script:

# /usr/share/webapps/awstats/6.5-r1/hostroot/cgi-bin/awstats.pl -config=mail.emergetux.net -update

Update for config "/etc/awstats/awstats.mail.emergetux.net.conf"

With data in log file "perl /usr/bin/maillogconvert.pl standard < /var/log/mail/current|"...

Phase 1 : First bypass old records, searching new record...

Searching new records from beginning of log file...

Jumped lines in file: 0

Parsed lines in file: 0

 Found 0 dropped records,

 Found 0 corrupted records,

 Found 0 old records,

 Found 0 new qualified records.

So, there's a problem with the script. It is executed in awstas conf file like:

LogFile="perl /usr/bin/maillogconvert.pl standard < /var/log/mail/current|"

and here's some log output:

Oct 19 16:17:24 [postfix/smtpd] NOQUEUE: reject: RCPT from 61-230-68-145.dynamic.hinet.net[61.230.68.145]: 554 <candy59839@yahoo.com.tw>: Relay access denied; from=<michael78694@MyMainServer.com> to=<candy59839@yahoo.com.tw> proto=SMTP helo=<www.MyMainServer.com>

Oct 19 16:17:25 [postfix/smtpd] lost connection after RCPT from 61-230-68-145.dynamic.hinet.net[61.230.68.145]

Oct 19 16:17:25 [postfix/smtpd] disconnect from 61-230-68-145.dynamic.hinet.net[61.230.68.145]

Oct 19 16:19:16 [postfix/smtpd] connect from lists.gentoo.org[140.105.134.102]

Oct 19 16:19:17 [postfix/smtpd] 45A257D943: client=lists.gentoo.org[140.105.134.102]

Oct 19 16:19:17 [postfix/cleanup] 45A257D943: message-id=<20061019141352.GA25140@pce151.ram.siemens.at>

Oct 19 16:19:17 [postfix/qmgr] 45A257D943: from=<gentoo-user+bounces-53313-arnau=emergetux.net@gentoo.org>, size=2226, nrcpt=1 (queue active)

Oct 19 16:19:17 [postfix/pipe] 45A257D943: to=<arnau______en______emergetux.net>, relay=cyrus, delay=1, status=sent (afrodita.emergetux.net)

Oct 19 16:19:17 [postfix/qmgr] 45A257D943: removed

Oct 19 16:19:17 [postfix/smtpd] disconnect from lists.gentoo.org[140.105.134.102]

So, I don't know what wrong with my logs....

Any idea about what is happening?

Thanks in advance!

Cheers,

Arnau

----------

## Narusegawa

Could you attach the configuration file your using for awstats?

I have awstats 6.5 and postfix working fine myself and I didn't follow that FAQ but it's a pretty standard way of working anyway.

----------

## pelacables

```

LogFile="perl /usr/bin/maillogconvert.pl standard < /var/log/mail/current|"

LogType=M

LogFormat="%time2 %email %email_r %host %host_r %method %url %code %bytesd"

LogSeparator=" "

SiteDomain="localhost"

HostAliases="localhost 127.0.0.1 REGEX[myserver\.com$]"

DNSLookup=2

DirData="/var/www/mail.emergetux.net/htdocs/awstats/"

DirCgi="/cgi-bin/awstats"

DirIcons="/awstats/icons"

AllowToUpdateStatsFromBrowser=0

AllowFullYearView=2

EnableLockForUpdate=0

DNSStaticCacheFile="dnscache.txt"

DNSLastUpdateCacheFile="dnscachelastupdate.txt"

SkipDNSLookupFor=""

AllowAccessFromWebToAuthenticatedUsersOnly=0

AllowAccessFromWebToFollowingAuthenticatedUsers=""

AllowAccessFromWebToFollowingIPAddresses=""

CreateDirDataIfNotExists=0

BuildHistoryFormat=text

BuildReportFormat=html

SaveDatabaseFilesWithPermissionsForEveryone=0

PurgeLogFile=0

ArchiveLogRecords=0

KeepBackupOfHistoricFiles=0

DefaultFile="index.html"

SkipHosts=""

SkipUserAgents=""

SkipFiles=""

OnlyHosts=""

OnlyUserAgents=""

OnlyFiles=""

NotPageList="css js class gif jpg jpeg png bmp ico swf"

ValidHTTPCodes="200 304"

ValidSMTPCodes="1 250"

AuthenticatedUsersNotCaseSensitive=0

URLNotCaseSensitive=0

URLWithAnchor=0

URLQuerySeparators="?;"

URLWithQuery=0

URLWithQueryWithOnlyFollowingParameters=""

URLWithQueryWithoutFollowingParameters=""

URLReferrerWithQuery=0

WarningMessages=1

ErrorMessages=""

DebugMessages=0

NbOfLinesForCorruptedLog=50

WrapperScript=""

DecodeUA=0

MiscTrackerUrl="/js/awstats_misc_tracker.js"

UseFramesWhenCGI=1

DetailedReportsOnNewWindows=1

Expires=0

MaxRowsInHTMLOutput=1000

Lang="auto"

DirLang="./lang"

ShowMenu=1

ShowSummary=HB

ShowMonthStats=HB

ShowDaysOfMonthStats=HB

ShowDaysOfWeekStats=HB

ShowHoursStats=HB

ShowDomainsStats=0

ShowHostsStats=HBL

ShowAuthenticatedUsers=0

ShowRobotsStats=0

ShowWormsStats=0

ShowEMailSenders=HBML

ShowEMailReceivers=HBML

ShowSessionsStats=0

ShowPagesStats=0

ShowFileTypesStats=0

ShowFileSizesStats=0

ShowOSStats=0

ShowBrowsersStats=0

ShowScreenSizeStats=0

ShowOriginStats=0

ShowKeyphrasesStats=0

ShowKeywordsStats=0

ShowMiscStats=0

ShowHTTPErrorsStats=0

ShowSMTPErrorsStats=1

ShowClusterStats=0

AddDataArrayMonthStats=1

AddDataArrayShowDaysOfMonthStats=1

AddDataArrayShowDaysOfWeekStats=1

AddDataArrayShowHoursStats=1

IncludeInternalLinksInOriginSection=0

MaxNbOfDomain = 10

MinHitDomain  = 1

MaxNbOfHostsShown = 10

MinHitHost    = 1

MaxNbOfLoginShown = 10

MinHitLogin   = 1

MaxNbOfRobotShown = 10

MinHitRobot   = 1

MaxNbOfPageShown = 10

MinHitFile    = 1

MaxNbOfOsShown = 10

MinHitOs      = 1

MaxNbOfBrowsersShown = 10

MinHitBrowser = 1

MaxNbOfScreenSizesShown = 5

MinHitScreenSize = 1

MaxNbOfWindowSizesShown = 5

MinHitWindowSize = 1

MaxNbOfRefererShown = 10

MinHitRefer   = 1

MaxNbOfKeyphrasesShown = 10

MinHitKeyphrase = 1

MaxNbOfKeywordsShown = 10

MinHitKeyword = 1

MaxNbOfEMailsShown = 20

MinHitEMail   = 1

FirstDayOfWeek=1

ShowFlagLinks=""

ShowLinksOnUrl=1

UseHTTPSLinkForUrl=""

MaxLengthOfShownURL=64

HTMLHeadSection=""

HTMLEndSection=""

Logo="awstats_logo6.png"

LogoLink="http://awstats.sourceforge.net"

BarWidth   = 260

BarHeight  = 90

StyleSheet=""

ExtraTrackedRowsLimit=500

```

What do you have if you execute the script against the log file?Last edited by pelacables on Fri Oct 27, 2006 7:40 am; edited 1 time in total

----------

## pelacables

Other thing I asked in some list, and forgot to mention here, is the LogFormat value:

```

LogFormat="%time2 %email %email_r %host %host_r %method %url %code %bytesd" 

```

I understand that this is standard mail log format. I did not see any line like this in my log, but I suppose script does.

I turned off the -v option in master.cf:

```

smtp      inet  n       -       n       -       -       smtpd (-v)

```

And I haven't done any other change, so Postfix uses default values...

Cheers!

----------

## Narusegawa

Give me an hour and I'll check this against my own servers config when I get home from work.

----------

## pelacables

I give you more cause till tomorrow I won't check this issue again!

thanks for your help!

----------

## Narusegawa

the only issues I can see are

```
SiteDomain="localhost"

HostAliases="localhost 127.0.0.1 REGEX[myserver\.com$]" 
```

in yours.

In mine I have SiteDomain as the main domain for this server. And each mail domain in the aliases list.

```
SiteDomain="mailserver.co.uk"

HostAliases="domain2.com domain3.co.uk domain4.org" 
```

I think thats your problem. It's looking for mail to and from "localhost" which is 0.

And just for clarification my Logfile is

```
LogFile="perl /var/www/mailserver.co.uk/mail/cgi-bin/maillogconvert.pl standard < /var/log/mail.log |"
```

My master.cf has

```
smtp      inet  n       -       n       -       -       smtpd
```

And running the perl line on a command prompt as root, gives me a few lines which I expect. So I'm hazarding a guess the logfile your using when running this on the command line is empty.

----------

## pelacables

Hi,

You're right in part, this conf value is wrong. I've changed it. But it still does not find any record

```

# /usr/share/webapps/awstats/6.5-r1/hostroot/cgi-bin/awstats.pl -config=mail.emergetux.net -update

Update for config "/etc/awstats/awstats.mail.emergetux.net.conf"

With data in log file "perl /usr/bin/maillogconvert.pl standard < /var/log/mail/test|"...

Phase 1 : First bypass old records, searching new record...

Searching new records from beginning of log file...

Jumped lines in file: 0

Parsed lines in file: 0

 Found 0 dropped records,

 Found 0 corrupted records,

 Found 0 old records,

 Found 0 new qualified records.

```

I still think that problem comes from LogFormat & MailLogConverter.pl:

# perl /usr/bin/maillogconvert.pl standard < /var/log/mail/test

#

Could you please try to execute this scipt agains your log file?

Could you please paste some mail log file and  LogFormat value?

Thanks for your reply.

----------

## Narusegawa

What's the output of

```
perl /usr/bin/maillogconvert.pl standard < /var/log/mail/test
```

(Without the trailing |)

And whats the contents of the /var/log/mail/test file aswell. 

A few lines from my log file are 

```
Oct 27 12:05:15 hinata postfix/smtpd[5619]: connect from unknown[222.124.97.77]

Oct 27 12:05:21 hinata postfix/smtpd[5619]: NOQUEUE: reject: RCPT from unknown[222.124.97.77]: 554 Service unavailable; Client

 host [222.124.97.77] blocked using sbl-xbl.spamhaus.org; http://www.spamhaus.org/query/bl?ip=222.124.97.77; from=<cuffbabylon

@rrbilimsel.com.tr> to=<tenchi@eclipticdreams.com> proto=ESMTP helo=<bkielud.0niie.rr.com>

Oct 27 12:05:21 hinata postfix/smtpd[5619]: NOQUEUE: reject: RCPT from unknown[222.124.97.77]: 554 Service unavailable; Client

 host [222.124.97.77] blocked using sbl-xbl.spamhaus.org; http://www.spamhaus.org/query/bl?ip=222.124.97.77; from=<cuffbabylon

@rrbilimsel.com.tr> to=<naru@eclipticdreams.com> proto=ESMTP helo=<bkielud.0niie.rr.com>
```

The output of me running maillogconvert.pl against it is a nicer format  :Smile:  (maillogconvert doesnt look at configuration files so this should always work).

My version of postfix is "postfix-2.2.10" however my awstats was installed manually as I've used it for 4 years now and have no confidence in the webapps installer version of this.

Incidentally since installing awstats and looking at the stats I'm rather worried about the amount of mail travelling around my server (or potential mail)

 *Quote:*   

> 554	Requested mail action rejected: access denied	2366	43.8 %	0
> 
> 504	Command parameter not implemented	1665	30.8 %	0
> 
> 999	Unknown error	810	15 %	6.41 MB
> ...

 

That's one helluva lot of failed mail! And sadly awstats doesn't give much more information pertaining to these mail errors.

----------

## pelacables

Hi,

forum is not sending me an email!! WTH!

```

afrodita ~ # perl /usr/bin/maillogconvert.pl standard < /var/log/mail/test

afrodita ~ # perl /usr/bin/maillogconvert.pl standard < /var/log/mail/log-2006-10-25-01\:02\:24

afrodita ~ # perl /usr/bin/maillogconvert.pl standard < /var/log/mail/current

```

Nothing! I can run the script agains whatever log file and I see nothig...

test file is a concatenation of log files... (metalog rotates maillog every day, or when it reaches 2MB)

```

Oct 13 14:56:23 [postfix/smtpd] ctable_locate: leave existing entry key kikiuki88a@yahoo.com.tw

Oct 13 14:56:23 [postfix/smtpd] NOQUEUE: reject: RCPT from 218-161-126-85.dynamic.hinet.net[218.161.126.85]: 554 <kikiuki88a@yahoo.com.tw>: Relay access denied; from=<kikiuki88@yahoo.com.tw> to=<kikiuki88a@yahoo.com.tw> proto=SMTP helo=<80.37.233.41>

Oct 13 14:56:23 [postfix/smtpd] generic_checks: name=reject_unauth_destination status=2

Oct 13 14:56:23 [postfix/smtpd] > 218-161-126-85.dynamic.hinet.net[218.161.126.85]: 554 <kikiuki88a@yahoo.com.tw>: Relay access denied

[...]

And this is from "current" file:

Oct 30 01:54:10 [postfix/smtpd] connect from lists.gentoo.org[140.105.134.102]

Oct 30 01:54:11 [postfix/smtpd] 7D41F7CD4D: client=lists.gentoo.org[140.105.134.102]

Oct 30 01:54:11 [postfix/cleanup] 7D41F7CD4D: message-id=<200610300145.04041.bo.andresen@zlin.dk>

Oct 30 01:54:11 [postfix/qmgr] 7D41F7CD4D: from=<gentoo-user+bounces-53935-arnau=emergetux.net@gentoo.org>, size=3158, nrcpt=1 (queue active)

Oct 30 01:54:11 [postfix/pipe] 7D41F7CD4D: to=<arnau@emergetux.net>, relay=cyrus, delay=0, status=sent (afrodita.emergetux.net)

Oct 30 01:54:11 [postfix/qmgr] 7D41F7CD4D: removed

Oct 30 01:54:14 [postfix/smtpd] disconnect from lists.gentoo.org[140.105.134.102]

```

I think this is postfix standard...

I agree with you, maillogconverted should work always! but not in my case... I'm gonna look for older version...

I have same postfix version, and awstats installed from portage, but I believe it does not matter...

If you look for complet mail info, you could try net-mail/pflogsumm or mailgraph.

Thanks for your help...

I'll continue looking for the source of my problem!

----------

## Narusegawa

I can see that our log formats are slightly different.

I have

  servername process/sub-process[PID]: command from postfix

where as you have

  [process/sub-process] command from postfix

I believe this is why yours is failing. Sadly I'm at work, but as soon as I get home I'll dig out my log format line from postfix (it's been awhile since I installed and looked at the postfix stuff myself)

----------

## pelacables

Yes,

I use metalog.

I asked in spanish postfix mailing list for where is the postfix log format defined, but no one answered me...

Maillogconverter is looking for some special format...

That could be a possible solution ,gonna look for it in postfix site.

----------

## Narusegawa

I myself just use a basic syslog-ng setup. But I believe the root of this problem is indeed the log format. I hope you find a solution.

----------

