# Amavisd Segfaults

## hanj

Hello All

Starting Nov 8, I've been seeing fairly consistent segfaults from amavisd. I'm unable to determine what is causing it. Below is a listing of occurances in /var/log/messages. The server has been running fine (normal load - mail traffic is normal). Also I checked uptime (reboot was Nov.1) and recent emerges. The only emerges close Nov.8 was rrdtool and cacti (11/5 and 11/6)

```
Nov  8 04:12:34 comp amavisd[32519]: segfault at 5f1e9f40 ip 0813622a sp 5f1e9f40 error 6 in perl5.8.8[8048000+144000]

Nov  8 04:20:42 comp amavisd[914]: segfault at 5f1e9f40 ip 0813622a sp 5f1e9f40 error 6 in perl5.8.8[8048000+144000]

Nov  8 04:30:50 comp amavisd[32641]: segfault at 5f1e9f40 ip 0813622a sp 5f1e9f40 error 6 in perl5.8.8[8048000+144000]

Nov  8 04:52:01 comp amavisd[2720]: segfault at 5f1e9f40 ip 0813622a sp 5f1e9f40 error 6 in perl5.8.8[8048000+144000]

Nov  8 05:35:41 comp amavisd[5006]: segfault at 5f1e9f40 ip 0813622a sp 5f1e9f40 error 6 in perl5.8.8[8048000+144000]

Nov  8 06:45:37 comp amavisd[16413]: segfault at 5f1e9f40 ip 0813622a sp 5f1e9f40 error 6 in perl5.8.8[8048000+144000]

Nov  8 07:55:40 comp amavisd[23476]: segfault at 5f1e9f40 ip 0813622a sp 5f1e9f40 error 6 in perl5.8.8[8048000+144000]

Nov 12 17:18:18 comp amavisd[12686]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 12 17:26:55 comp amavisd[14167]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 12 17:36:52 comp amavisd[14891]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 12 17:56:53 comp amavisd[18258]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 12 18:36:53 comp amavisd[21351]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 12 19:46:54 comp amavisd[28790]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 12 20:56:53 comp amavisd[7876]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 12 22:06:55 comp amavisd[16099]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 12 23:16:55 comp amavisd[20916]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 13 00:06:17 comp amavisd[27597]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 13 00:11:55 comp amavisd[28923]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 13 00:21:51 comp amavisd[26905]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 13 00:26:54 comp amavisd[30292]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 13 00:41:53 comp amavisd[28971]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 13 01:21:54 comp amavisd[30683]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 13 01:37:01 comp amavisd[4601]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 13 02:31:51 comp amavisd[7793]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 13 02:46:51 comp amavisd[9828]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 13 03:41:53 comp amavisd[32020]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 13 03:56:55 comp amavisd[2010]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 13 04:51:55 comp amavisd[6267]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 13 05:06:54 comp amavisd[6250]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 13 06:01:51 comp amavisd[12024]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]

Nov 13 06:16:54 comp amavisd[15511]: segfault at 5de9bea0 ip 0813622a sp 5de9bea0 error 6 in perl5.8.8[8048000+144000]
```

```
[ebuild   R   ] mail-filter/amavisd-new-2.6.1  USE="mysql -courier -dkim -ldap -milter -postgres -qmail" 891 kB

[ebuild   R   ] dev-lang/perl-5.8.8-r5  USE="berkdb gdbm -build -debug -doc -ithreads -perlsuid" 9,887 kB

[ebuild   R   ] mail-mta/postfix-2.5.5  USE="mysql pam sasl ssl vda -cdb -dovecot-sasl -hardened -ipv6 -ldap -mailwrapper -mbox -nis -postgres (-selinux)" 3,097 kB
```

```
Portage 2.1.4.5 (default-linux/x86/2007.0, gcc-4.1.2, glibc-2.6.1-r0, 2.6.25-hardened-r9 i686)

=================================================================

System uname: 2.6.25-hardened-r9 i686 AMD Duron(TM)

Timestamp of tree: Thu, 13 Nov 2008 10:00:01 +0000

app-shells/bash:     3.2_p33

dev-java/java-config: 1.3.7, 2.1.6

dev-lang/python:     2.5.2-r7

dev-python/pycrypto: 2.0.1-r6

sys-apps/baselayout: 1.12.11.1

sys-apps/sandbox:    1.2.18.1-r2

sys-devel/autoconf:  2.13, 2.61-r2

sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.1-r1

sys-devel/binutils:  2.18-r3

sys-devel/gcc-config: 1.4.0-r4

sys-devel/libtool:   1.5.26

virtual/os-headers:  2.6.23-r3

ACCEPT_KEYWORDS="x86"

CBUILD="i686-pc-linux-gnu"

CFLAGS="-O3 -march=i686 -funroll-loops -pipe "

CHOST="i686-pc-linux-gnu"

CONFIG_PROTECT="/etc"

CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/terminfo /etc/udev/rules.d"

CXXFLAGS="-O3 -march=i686 -funroll-loops -pipe "

DISTDIR="/usr/portage/distfiles"

FEATURES="distlocks metadata-transfer sandbox sfperms strict unmerge-orphans userfetch"

GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/linux/distributions/gentoo"

MAKEOPTS="-j2"

PKGDIR="/usr/portage/packages"

PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"

PORTAGE_TMPDIR="/var/tmp"

PORTDIR="/usr/portage"

PORTDIR_OVERLAY="/usr/local/portage"

SYNC="rsync://rsync.gentoo.org/gentoo-portage"

USE="apache2 berkdb cli cracklib crypt dri fortran gdbm gpm iconv innodb isdnlog maildir midi mudflap mysql ncurses nptl nptlonly openmp openssh pam pcre perl php pppd pwdb python readline reflection sasl session snortsam spl ssl tcpd unicode x86 xorg zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" APACHE2_MPMS="prefork" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" USERLAND="GNU" VIDEO_CARDS="apm ark chips cirrus cyrix dummy fbdev glint i128 i740 i810 imstt intel mach64 mga neomagic nsc nv r128 radeon rendition s3 s3virge savage siliconmotion sis sisusb tdfx tga trident tseng v4l vesa vga via vmware voodoo"

Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LANG, LC_ALL, LDFLAGS, LINGUAS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
```

Any suggestions on how isolate and fix this problem?

Thanks!

hanji

----------

## cach0rr0

what all plugins are you using with amavisd? 

I haven't seen this on my test box that runs amavis/SA/pyzor. Heck, blind guess, sa-updates pulling down a new set of rules, one of which contains a shoddy regex that causes perl to upchuck? random guess.

----------

## hanj

 *cach0rr0 wrote:*   

> what all plugins are you using with amavisd? 

 

Here is output on start up:

```
Nov 13 17:32:37 comp amavis[16898]: starting.  /usr/sbin/amavisd at comp amavisd-new-2.6.1 (20080629), Unicode aware

Nov 13 17:32:37 comp amavis[16898]: Perl version               5.008008

Nov 13 17:32:38 comp amavis[16900]: Net::Server: Group Not Defined.  Defaulting to EGID '1062 1062'

Nov 13 17:32:38 comp amavis[16900]: Net::Server: User Not Defined.  Defaulting to EUID '105'

Nov 13 17:32:38 comp amavis[16900]: Module Amavis::Conf        2.103

Nov 13 17:32:38 comp amavis[16900]: Module Archive::Zip        1.23

Nov 13 17:32:38 comp amavis[16900]: Module BerkeleyDB          0.34

Nov 13 17:32:38 comp amavis[16900]: Module Compress::Zlib      2.015

Nov 13 17:32:38 comp amavis[16900]: Module Convert::TNEF       0.17

Nov 13 17:32:38 comp amavis[16900]: Module Convert::UUlib      1.09

Nov 13 17:32:38 comp amavis[16900]: Module DBD::mysql          4.005

Nov 13 17:32:38 comp amavis[16900]: Module DBI                 1.601

Nov 13 17:32:38 comp amavis[16900]: Module DB_File             1.815

Nov 13 17:32:38 comp amavis[16900]: Module Digest::MD5         2.33

Nov 13 17:32:38 comp amavis[16900]: Module Digest::SHA1        2.11

Nov 13 17:32:38 comp amavis[16900]: Module MIME::Entity        5.420

Nov 13 17:32:38 comp amavis[16900]: Module MIME::Parser        5.420

Nov 13 17:32:38 comp amavis[16900]: Module MIME::Tools         5.420

Nov 13 17:32:38 comp amavis[16900]: Module Mail::Header        1.77

Nov 13 17:32:38 comp amavis[16900]: Module Mail::Internet      1.77

Nov 13 17:32:38 comp amavis[16900]: Module Mail::SpamAssassin  3.002001

Nov 13 17:32:38 comp amavis[16900]: Module Net::DNS            0.63

Nov 13 17:32:38 comp amavis[16900]: Module Net::Server         0.97

Nov 13 17:32:38 comp amavis[16900]: Module Razor2::Client::Version 2.82

Nov 13 17:32:38 comp amavis[16900]: Module Time::HiRes         1.9715

Nov 13 17:32:38 comp amavis[16900]: Module URI                 1.35

Nov 13 17:32:38 comp amavis[16900]: Module Unix::Syslog        0.100

Nov 13 17:32:38 comp amavis[16900]: Amavis::DB code      loaded

Nov 13 17:32:38 comp amavis[16900]: Amavis::Cache code   loaded

Nov 13 17:32:38 comp amavis[16900]: SQL base code        loaded

Nov 13 17:32:38 comp amavis[16900]: SQL::Log code        NOT loaded

Nov 13 17:32:38 comp amavis[16900]: SQL::Quarantine      NOT loaded

Nov 13 17:32:38 comp amavis[16900]: Lookup::SQL code     loaded

Nov 13 17:32:38 comp amavis[16900]: Lookup::LDAP code    NOT loaded

Nov 13 17:32:38 comp amavis[16900]: AM.PDP-in proto code NOT loaded

Nov 13 17:32:38 comp amavis[16900]: SMTP-in proto code   loaded

Nov 13 17:32:38 comp amavis[16900]: Courier proto code   NOT loaded

Nov 13 17:32:38 comp amavis[16900]: SMTP-out proto code  loaded

Nov 13 17:32:38 comp amavis[16900]: Pipe-out proto code  NOT loaded

Nov 13 17:32:38 comp amavis[16900]: BSMTP-out proto code loaded

Nov 13 17:32:38 comp amavis[16900]: Local-out proto code loaded

Nov 13 17:32:38 comp amavis[16900]: OS_Fingerprint code  NOT loaded

Nov 13 17:32:38 comp amavis[16900]: ANTI-VIRUS code      loaded

Nov 13 17:32:38 comp amavis[16900]: ANTI-SPAM code       loaded

Nov 13 17:32:38 comp amavis[16900]: ANTI-SPAM-SA code    loaded

Nov 13 17:32:38 comp amavis[16900]: Unpackers code       loaded

Nov 13 17:32:38 comp amavis[16900]: DKIM code            NOT loaded

Nov 13 17:32:38 comp amavis[16900]: Tools code           NOT loaded

Nov 13 17:32:38 comp amavis[16900]: Found $file            at /usr/bin/file

Nov 13 17:32:38 comp amavis[16900]: No $dspam,             not using it

Nov 13 17:32:38 comp amavis[16900]: No $altermime,         not using it

Nov 13 17:32:38 comp amavis[16900]: Internal decoder for .mail

Nov 13 17:32:38 comp amavis[16900]: Internal decoder for .asc

Nov 13 17:32:38 comp amavis[16900]: Internal decoder for .uue

Nov 13 17:32:38 comp amavis[16900]: Internal decoder for .hqx

Nov 13 17:32:38 comp amavis[16900]: Internal decoder for .ync

Nov 13 17:32:38 comp amavis[16900]: Found decoder for    .F    at /usr/bin/unfreeze

Nov 13 17:32:38 comp amavis[16900]: Found decoder for    .Z    at /usr/bin/uncompress

Nov 13 17:32:38 comp amavis[16900]: Found decoder for    .gz   at /bin/gzip -d

Nov 13 17:32:38 comp amavis[16900]: Found decoder for    .bz2  at /bin/bzip2 -d

Nov 13 17:32:38 comp amavis[16900]: No decoder for       .lzo  tried: lzop -d

Nov 13 17:32:38 comp amavis[16900]: No decoder for       .rpm  tried: rpm2cpio.pl, rpm2cpio

Nov 13 17:32:38 comp amavis[16900]: Found decoder for    .cpio at /usr/bin/pax

Nov 13 17:32:38 comp amavis[16900]: Found decoder for    .tar  at /usr/bin/pax

Nov 13 17:32:38 comp amavis[16900]: Found decoder for    .deb  at /usr/bin/ar

Nov 13 17:32:38 comp amavis[16900]: Internal decoder for .zip

Nov 13 17:32:38 comp amavis[16900]: No decoder for       .7z   tried: 7zr, 7za, 7z

Nov 13 17:32:38 comp amavis[16900]: Found decoder for    .rar  at /usr/bin/unrar

Nov 13 17:32:38 comp amavis[16900]: Found decoder for    .arj  at /usr/bin/unarj

Nov 13 17:32:38 comp amavis[16900]: Found decoder for    .arc  at /usr/bin/arc

Nov 13 17:32:38 comp amavis[16900]: Found decoder for    .zoo  at /usr/bin/zoo

Nov 13 17:32:38 comp amavis[16900]: Found decoder for    .lha  at /usr/bin/lha

Nov 13 17:32:38 comp amavis[16900]: Found decoder for    .cab  at /usr/bin/cabextract

Nov 13 17:32:38 comp amavis[16900]: No decoder for       .tnef tried: tnef

Nov 13 17:32:38 comp amavis[16900]: Internal decoder for .tnef

Nov 13 17:32:38 comp amavis[16900]: Found decoder for    .exe  at /usr/bin/unrar; /usr/bin/lha; /usr/bin/unarj

Nov 13 17:32:38 comp amavis[16900]: Using primary internal av scanner code for ClamAV-clamd

Nov 13 17:32:38 comp amavis[16900]: Found secondary av scanner ClamAV-clamscan at /usr/bin/clamscan

Nov 13 17:32:38 comp amavis[16900]: Creating db in /var/amavis/db/; BerkeleyDB 0.34, libdb 4.5
```

hanji

----------

## cach0rr0

how intrusive can we be with this box? 

offhand I'd say

-crank up the log level if you havent already

-yank spamassassin out of the picture (temporarily), just to make sure a shoddy regex isnt making it upchuck. $sa_spawned =1 if I'm reading correctly, will fork SA...so you'll know if it's SA choking, or amavisd itself (which would subsequently point towards message decomposition as a culprit...but how?!?!?!?) Either that or comment it out of you amavisd conf

(EDIT: perl is choking, so...gut instinct tells me it's along these lines)

----------

## hanj

Thanks for the reply. I'll crank up the log level and take sa out for a bit. I want to wait for it to happen again. Of course, everything has been running peachy since this morning's segfault.

Thanks!

hanji

----------

## cach0rr0

hrmmm...makes me wonder if we had a dodgy SA rule, which was subsequently overwritten/fixed with an update

you auto-updating SA rules? Snagging the ones from SARE? Wondering if we can correlate an SA rules update to the start of the problem. 

might be worthwhile cranking up the score for a Razor trigger whilst waiting on this to happen again, purely to keep the inboxes tidy. 

I'll fiddle with this tomorrow at work where the test box is, doing this from memory at the moment.

EDIT: Would also be keen to fork SA so that when it happens again, we know if it's amavisd itself dying, or SA killing it

----------

## hanj

Segfault today.. with additional output dumped to console..

```
Nov 21 12:25:21 comp amavisd[6006]: segfault at de26c9c ip 080d1d96 sp 5df839c0 error 4 in perl5.8.8[8048000+144000]
```

Here is the output.. looks to be kernel/memory related?

```
comp Eeek! page_mapcount(page) went negative! (-1)

comp page pfn = 32fd7

comp page->flags = 40000090

comp page->count = 1

comp page->mapping = 00000000

comp ------------[ cut here ]------------

comp invalid opcode: 0000 [#1]

comp Process amavisd (pid: 6550, ti=f09a0000 task=d4e48bf0 task.ti=f09a0000)

comp Stack: c09a075f 00000000 c09a0748 00000001 c165fae0 c0832ba9 32fd7045 00000000

comp e8b1be8c f09a1cf4 00154efa 00000000 00000001 edc53080 ea30c3e0 c09e8994

comp fffffda2 ffffffff edc53080 0be3b000 00000000 f09a1cf4 f2c3a3e4 ea30c3e0

comp Call Trace:

comp [<c0832ba9>] <0> [<c08353e1>] <0> [<c080fadd>] <0> [<c0845a3c>] <0> [<c0866788>] <0> [<c081e654>] <0> [<c084047c>] <0> [<c0842245>] <0> [<c0845175>] <0> [<c0844fba>] <0> [<c0865dcf>] <0> [<c0833a9c>] <0> [<c0831802>] <0> [<c0844f6c>] <0> [<c0844fba>] <0> [<c0846268>] <0> [<c08012b6>] <0> [<c08029f2>] <0> =======================

comp Code: 89 da 25 00 40 02 00 3d 00 40 02 00 75 03 8b 53 0c ff 72 04 68 48 07 9a c0 e8 24 97 fd ff ff 73 10 68 5f 07 9a c0 e8 17 97 fd ff <0f> 0b 83 c4 10 eb fe 8b 53 10 8b 03 83 e2 01 c1 e8 1e f7 da 83

comp EIP: [<c083865b>]  SS:ESP 0068:f09a1c7c

comp Eeek! page_mapcount(page) went negative! (-1)

comp page pfn = 72a89

comp page->flags = 80010038

comp page->count = 1

comp page->mapping = c557d568

comp ------------[ cut here ]------------

comp invalid opcode: 0000 [#2]

comp Process amavisd (pid: 7317, ti=c3ab6000 task=f3a4c690 task.ti=c3ab6000)

comp Stack: c09a075f c557d568 c09a0748 00000001 c1e55120 c0832ba9 72a89045 00000000

comp c7a231d4 c3ab7d9c 0015effd 00000000 00000001 de9ab0b4 f411f760 c09e8994

comp fffffd60 ffffffff de9ab0b4 0bc8b000 00000000 c3ab7d9c c7a23334 f411f760

comp Call Trace:

comp [<c0832ba9>] <0> [<c08353e1>] <0> [<c080fadd>] <0> [<c0845a3c>] <0> [<c0866788>] <0> [<c0832913>] <0> [<c0833a9c>] <0> [<c0831802>] <0> [<c0844f6c>] <0> [<c0844fba>] <0> [<c0846268>] <0> [<c08012b6>] <0> [<c08029f2>] <0> =======================

comp Code: 89 da 25 00 40 02 00 3d 00 40 02 00 75 03 8b 53 0c ff 72 04 68 48 07 9a c0 e8 24 97 fd ff ff 73 10 68 5f 07 9a c0 e8 17 97 fd ff <0f> 0b 83 c4 10 eb fe 8b 53 10 8b 03 83 e2 01 c1 e8 1e f7 da 83

comp EIP: [<c083865b>]  SS:ESP 0068:c3ab7d24
```

I'm currently using 2.6.25-hardened-r10 kernel.

Here is what is available as far as memory, swap and load

```
top - 12:48:22 up 5 days,  2:55,  2 users,  load average: 1.09, 0.95, 0.96

Tasks: 131 total,   1 running, 130 sleeping,   0 stopped,   0 zombie

Cpu(s): 25.4%us,  4.4%sy,  0.4%ni, 67.3%id,  2.4%wa,  0.0%hi,  0.1%si,  0.0%st

Mem:   3117684k total,  2989688k used,   127996k free,   477816k buffers

Swap:  2994020k total,        0k used,  2994020k free,  1633472k cached
```

hanji

----------

