# OpenSSL eating A LOT of cpu

## Rainmaker

Hi all,

I have a problem with openssl, both on my "unstable" workstation as on my "stable" server. The problem is openssh is eating about 90% of cpu, even when I'm not logged in. It seems to happen at random times, sometimes about every 10 minutes. it keeps eating CPU for about 6-7 minutes on my 2500 athlon, and about an hour on my router (pentium 200).

I have no idea what openssh is doing eating away so much of my valuable CPU time. Does anyone have an idea how to fix this?

 *Workstation wrote:*   

> Portage 2.0.51.22-r2 (default-linux/x86/2005.0, gcc-4.0.1, glibc-2.3.5-r0, 2.6.12-nitro5 i686)
> 
> =================================================================
> 
> System uname: 2.6.12-nitro5 i686 AMD Athlon(tm) XP 2500+
> ...

 

 *Server wrote:*   

> Portage 2.0.51.22-r2 (default-linux/x86/2005.0, gcc-3.4.4, glibc-2.3.5-r0, 2.6.11-hardened-r1 i586)
> 
> =================================================================
> 
> System uname: 2.6.11-hardened-r1 i586 Pentium MMX
> ...

 

----------

## Suer7reus

Have you checked your logs?  Look for failed login attempts (server's hardened kernel should audit them - for kernel config info just ask my lazy ass once more =) ).  It might be some jackass trying to brute-force you.  If so, you can set your system (via the hardened kernel, among others - again ask me again for kernel config info) to lock the bastard out for a while after so many failed attempts.  Oh, and even if you don't have any evidence of such activity, change your passwords (just because =P).

Oh, and if you're really bored sometime, you could strace it =P.

If you're in a chatty mood, it sounds like we have similar setups - how's the nitro kernel workin out for you?  Would you recommend it for desktop/laptop use?

Good luck =)!

----------

## Rainmaker

nothing in my logs. My server is behind a firewall, not directly connected to the internet, let alone getting brute-forced. My workstation is behind 2 firewalls... Port 22 is cought by the first firewall and explicitly blocked. So I'm pretty sure that's not it. There's nothing in the logs, not for sshd or from cron. I'll strace it the next time it comes around. Stuid I didn't think of that myself. Thanks for the tip  :Smile: .

As far as nitro goes: IT ROCKS. Never had any real trouble with it. It runs fast, but I'm afraid I don't really have any other kernel I can compare performance to, because I use reiser4 for my / partition. It's a great patchset,. but why not try it yourself? You won't break anything (hardware-wise that is).

Compared to mm-sources, I do notice a slighty better reaction time while the system is under heavy load.

----------

## Rainmaker

well, that strace doesn't really help, or maybe I'm missing something:

```
time([1122256959])                      = 1122256959

write(2, ".", 1)                        = 1

time([1122256959])                      = 1122256959

time([1122256959])                      = 1122256959

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

time([1122256960])                      = 1122256960

write(2, ".", 1)                        = 1

time([1122256960])                      = 1122256960

time([1122256961])                      = 1122256961

write(2, ".", 1)                        = 1

time([1122256961])                      = 1122256961

```

and this goes on and on and on.

----------

## Rainmaker

managed to capture the end of the strace:

```
time([1122264083])                      = 1122264083

write(2, "+", 1)                        = 1

time([1122264083])                      = 1122264083

write(2, "+", 1)                        = 1

write(2, "*", 1)                        = 1

write(2, "\n", 1)                       = 1

getuid32()                              = 0

geteuid32()                             = 0

getgid32()                              = 0

getegid32()                             = 0

getuid32()                              = 0

geteuid32()                             = 0

getgid32()                              = 0

getegid32()                             = 0

stat64("//.rnd", {st_mode=S_IFREG|0600, st_size=1024, ...}) = 0

open("//.rnd", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3

chmod("//.rnd", 0600)                   = 0

fstat64(3, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0

mmap2(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xa7f80000

write(3, "\356F\242,\nz\271\332\273l\375\353,8\3~\222\265\333\253"..., 1024) = 1024

close(3)                                = 0

munmap(0xa7f80000, 65536)               = 0

open("/var/qmail/control/dh1024.pem.tmp", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3

fstat64(3, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0

mmap2(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xa7f80000

write(3, "-----BEGIN DH PARAMETERS-----\nMI"..., 245) = 245

close(3)                                = 0

munmap(0xa7f80000, 65536)               = 0

exit_group(0)                           = ?

```

Looks like it's generating some kind of security certificate for qmail and it's stuck on a near infinite loop acquiring random numbers.

Odd

```
Medusa% sudo find / -name ".rnd"

/.rnd

find: WARNING: Hard link count is wrong for /proc/16161: this may be a bug in your filesystem driver.  Automatically turning on find's -noleaf option.  Earlier results may have failed to include directories that should have been searched.

find: /proc/23279/task: No such file or directory

find: /proc/23279/fd: No such file or directory

/root/.rnd

```

I removed both .rnd scripts.I'll see if it comes back. Both scripts missed the "x" permission, so there's no way openssl could have executed them (a change in permission would have been visible in the strace)

the .rnd scripts seem to contain binairy code. Any idea on how to decompile this? hexdump gives me:

```
Medusa# hexdump /root/.rnd

0000000 a0e3 5bf3 4554 780e 6f69 f617 89ca 83fc

0000010 c9ca 6f01 1c7a 50ce 9b35 9a1f a668 51b6

0000020 3b39 8b8e 3e70 8337 6606 d9ef d1b4 5b7d

0000030 0832 8346 a8e1 c640 969d 1890 f36e 16cb

0000040 682c 6d61 ad80 1b46 3131 0405 fb8b 95ba

0000050 ca7d c3c4 d1fd 6416 4aee f298 0789 2cec

0000060 5588 b6d4 2010 d56a 8fef 3e56 74b2 465d

0000070 c8bc 5379 eaba 8ea8 6349 a1db 4041 7528

0000080 ec76 2d8b 694e f743 9008 c30c ffa8 861d

0000090 2acd 022c f96e ae5b f77f 59ad 8e51 e12d

00000a0 4786 8117 38ca da10 8daa bb7a 5bfc 20ea

00000b0 8bb0 7b04 dc5b c1c8 fb93 12a9 74f3 657d

00000c0 c2ad d01e 9909 bc7c 29df abc2 65f4 85b6

00000d0 66e7 d51d f8f2 9b1c 91c7 50d6 a61c 9956

00000e0 6d0c 67dc 0e10 0947 51c7 12e5 5f31 92f4

00000f0 01bf 23b8 b47c 76cf 3f26 73d4 0f6b e376

0000100 d3f5 d1e1 fd5d b9c3 f8a4 ed1e 4141 b35c

0000110 3d69 f108 baa1 79a4 b316 1998 a199 6c2a

0000120 5bec 0815 1158 2c28 eff9 de35 a785 93b8

0000130 7400 51e7 8afe e7eb 0454 14dc fa48 c35d

0000140 39e3 8629 131e 78f6 795a 120d b1c8 07f5

0000150 f2a8 40f6 3121 7d79 28a3 db70 d7fe 08c0

0000160 d29b 5d31 75b1 b5cd 28fa 29b0 df4f 2d58

0000170 d745 4cf6 37c8 a837 6c8e 2c9c 85e8 2396

0000180 a78b 9d42 388b 064d cd2c 9d58 6204 a365

0000190 5469 9440 8b11 febf ef5e 8b68 1045 6454

00001a0 5fdf 581d ed91 2cda 1646 51c5 70dc 85bd

00001b0 41ef e78c 3f2b 5af0 cc9a b7b2 695c 5d9d

00001c0 4c18 db31 a7a9 2d0a 6ad6 f2e7 e622 f78d

00001d0 3cf6 e9d3 4ddc 6738 d4b8 0240 8925 b0e5

00001e0 9f61 587a ba61 6d9f c182 d0ea f219 cd04

00001f0 a91a dd5f 4945 7ca8 a469 2809 fef4 0759

0000200 2370 337c 70ff b804 5035 fbdd c59e af1a

0000210 e397 8bf3 f4e3 ed83 7040 0584 ab87 31fa

0000220 d159 ee90 1bea e318 dbe1 9998 5007 e745

0000230 336e c7fc e1a7 c1d0 d0aa 4b57 2117 c06c

0000240 0a02 88b3 9651 39ac f5fc 3f36 8831 b33a

0000250 b84d 0c90 23b9 3c99 4fcb 9f73 f246 e28a

0000260 61e8 0b94 ae76 a6e4 2073 2b8d e0b2 38f4

0000270 befc 4194 af57 17ca af9e fe9a 0701 2687

0000280 b693 398b 30af f621 0d8f 1067 9075 76c1

0000290 1eb3 8e5c 301c c352 7044 386a cdd1 29cd

00002a0 7fbd 2a2d eea5 18c1 f7c7 30af 0f50 c43d

00002b0 1f68 a98f 19af d826 7b66 7e86 24bc 3c6f

00002c0 0bc7 cef6 1566 9040 5941 7abe dfcb 18a4

00002d0 4e04 ce7b 3cc2 0c13 19af c570 611a f920

00002e0 b80d b1f8 1929 8de8 806e 59a9 679c 500a

00002f0 82b0 b229 fa9f 2958 21d9 2c3e adf5 ebb8

0000300 22ed ece5 8a0a 875e c844 b1c7 b7e7 5c52

0000310 453f 80d4 2c53 ef82 4138 f9fb d5ef 4862

0000320 044a a7cd 9238 e06c 9160 3338 ebf1 7419

0000330 c9e2 9be0 6b7b f0ae fe5b b19f ac5c d673

0000340 8a4e 0478 0b55 ceed 08dc 19c8 226e 6599

0000350 e1b7 c1f6 824f 5f85 cf82 0b98 49fe a0fe

0000360 fe62 5114 b994 8ddb 543e 0028 cbfb a019

0000370 144c 627d 2a6d fe20 6640 e3e6 df83 8172

0000380 b5fc a5e7 c1fc b32d 3d12 9897 5fde 62b9

0000390 269e 576d a561 24a4 374b 0dc2 852a 050a

00003a0 8cce c8aa c80d ca81 9e05 dbd1 9a40 50cc

00003b0 f378 f57d 5357 5211 0cd6 352a b08d 697f

00003c0 4668 e3ae d11b 9ba1 228e 4324 1cfa 828b

00003d0 d0ea 4686 0b7c 1c0a 0b76 2a89 ac43 5141

00003e0 99a6 06b4 fae6 9793 fe71 5aa0 bbc9 e63c

00003f0 bc65 c680 e009 fcb6 7480 801d 9863 bcb1

0000400
```

But I don't think this'll ring a bell in anyone's head...

----------

## Rainmaker

ok, that wasn't it. There's still a openssl process spawning. After this, a new .rnd file was generated.

I did notice I read the strace output incorrectly: the file //.rnd is opened to write to, making it likely this is just some output file the key gets written to, for use by other processes. Somehow it does seem qmail related. Strange, because qmail is not even running on my workstation (though it is installed, version 1.03-r16 with SSL support in USE)

----------

## Suer7reus

Sorry for the late reply - I've been out of town.

Puzzling indeed...

I wish I knew what to tell you, but it sounds like you've tried everything I would (except maybe yelling at it - try that!).

I suppose its never a bad idea to up/down-grade openssl and/or qmail (maybe ~, if they aren't already).  Just remerging often restore my fuzzy feeling of okayness, and sometimes the software follows suit =).

Good luck![/quote]

----------

## xmwxd

I also the same problem

 *Rainmaker wrote:*   

> Hi all,
> 
> I have a problem with openssl, both on my "unstable" workstation as on my "stable" server. The problem is openssh is eating about 90% of cpu, even when I'm not logged in. It seems to happen at random times, sometimes about every 10 minutes. it keeps eating CPU for about 6-7 minutes on my 2500 athlon, and about an hour on my router (pentium 200).
> 
> I have no idea what openssh is doing eating away so much of my valuable CPU time. Does anyone have an idea how to fix this?
> ...

 

----------

## Rainmaker

I think I found a partial solution: re-emergeing openssl without --as-needed in my LDFLAGS seemed to speed it up a little bit. Now openssl only eats my CPU for a few seconds. Anoying when gaming, but acceptable.

No idea why my server is experiencing the same problems though (LDFLAGS are unset)

xmwxd: can you post emerge info for comparison?

----------

## xmwxd

 *Rainmaker wrote:*   

> I think I found a partial solution: re-emergeing openssl without --as-needed in my LDFLAGS seemed to speed it up a little bit. Now openssl only eats my CPU for a few seconds. Anoying when gaming, but acceptable.
> 
> No idea why my server is experiencing the same problems though (LDFLAGS are unset)
> 
> xmwxd: can you post emerge info for comparison?

 

My emerge info

```
Gentoo wxd # emerge info

Portage 2.0.51.22-r2 (default-linux/x86/2005.0, gcc-3.4.4, glibc-2.3.5-r0, 2.6.11-gentoo-r10 i686)

=================================================================

System uname: 2.6.11-gentoo-r10 i686 Intel(R) Celeron(R) CPU 1.70GHz

Gentoo Base System version 1.6.13

dev-lang/python:     2.3.5, 2.4.1

sys-apps/sandbox:    1.2.8

sys-devel/autoconf:  2.13, 2.59-r6

sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.5

sys-devel/binutils:  2.16-r1

sys-devel/libtool:   1.5.18-r1

virtual/os-headers:  2.6.11-r2

ACCEPT_KEYWORDS="x86"

AUTOCLEAN="yes"

CBUILD="i686-pc-linux-gnu"

CFLAGS="-O2 -march=pentium4 -fomit-frame-pointer"

CHOST="i686-pc-linux-gnu"

CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3.3/env /usr/kde/3.3/share/config /usr/kde/3.3/shutdown /usr/kde/3.4/env /usr/kde/3.4/share/config /usr/kde/3.4/shutdown /usr/kde/3/share/config /usr/lib/X11/xkb /usr/lib/mozilla/defaults/pref /usr/share/config /var/qmail/alias /var/qmail/control /var/vpopmail/domains /var/vpopmail/etc"

CONFIG_PROTECT_MASK="/etc/gconf /etc/splash /etc/terminfo /etc/env.d"

CXXFLAGS="-O2 -march=pentium4 -fomit-frame-pointer"

DISTDIR="/usr/portage/distfiles"

FEATURES="autoconfig distlocks sandbox sfperms strict"

GENTOO_MIRRORS="ftp://ftp.sjtu.edu.cn/gentoo ftp://ftp.tsinghua.edu.cn/mirror/gentoo"

LANG="zh_CN"

MAKEOPTS="-j2"

PKGDIR="/usr/portage/packages"

PORTAGE_TMPDIR="/var/tmp"

PORTDIR="/usr/portage"

PORTDIR_OVERLAY="/usr/local/portage"

SYNC="rsync://ftp.tsinghua.edu.cn/gentoo/gentoo-portage"

USE="x86 X a52 aac aalib acl acpi alsa apache apache2 arts audiofile authdaemond avi bash-completion berkdb bindist bitmap-fonts bmp bonobo boundschecking bzip2 bzlib calendar caps cdparanoia cdr cjk crypt cscope curl dga directfb divx4linux dvb dvd dvdread eds emacs emacs-w3 emboss encode esd exif fam fbcon ffmpeg flac foomaticdb fortran freetds freetype ftp gd gdbm geoip gif gnome gpm gstreamer gtk gtk2 gtkhtml hardened hardenedphp i8x0 icq imagemagick imlib ipv6 jabber jpeg ldap libg++ libwww linguas_zh_CN live livecd lm_sensors mad matrox mcal mikmod milter mng mod_php motif mozilla mp3 mpeg msn mysql nas ncurses network nls nocd nptl nvidia odbc ogg oggvorbis openal opengl osc oscar oss pam pdflib perl php png python quicktime readline real rtc samba sdl slang source speex spell sse ssl stroke svg svga tcpd tiff truetype truetype-fonts type1-fonts unicode usb valias vcd vorbis win32codecs wmv xine xinerama xml xml2 xmms xpm xv xvid zlib userland_GNU kernel_linux elibc_glibc"

Unset:  ASFLAGS, CTARGET, LC_ALL, LDFLAGS, LINGUAS

```

----------

## wll

Gentoo's qmail installs /etc/cron.hourly/qmail-genrsacert.sh, which contains the line:

```
/usr/bin/openssl genrsa -out ${tmpfile} ${bits} 2>/dev/null
```

This may be what you're watching. As to why it's eating your CPU, I don't know.

On my Sempron 1800, executing a similar openssl command, it's pretty fast:

```
www ~ # time /usr/bin/openssl genrsa 512 -out tempfile

Generating RSA private key, 512 bit long modulus

...

SNIP

...

real    0m0.022s

user    0m0.020s

sys     0m0.000s

```

Tried it a couple of times and it never took longer than a real 0.044 seconds.

----------

## Rainmaker

I'm afraid I was a little premature marking this as solved.

Thanks for the tip, but this command is probably not what's causing the high cpu usage:

```
Medusa% time /usr/bin/openssl genrsa 512 -out tempfile

Generating RSA private key, 512 bit long modulus

..................++++++++++++

......++++++++++++

e is 65537 (0x10001)

-----BEGIN RSA PRIVATE KEY-----

...

-----END RSA PRIVATE KEY-----

/usr/bin/openssl genrsa 512 -out tempfile  0,05s user 0,00s system 47% cpu 0,121 total

<ssh'ing to server>

Goofy ~ # time /usr/bin/openssl genrsa 512 -out tempfile

Generating RSA private key, 512 bit long modulus

.......++++++++++++

.....................................++++++++++++

e is 65537 (0x10001)

-----BEGIN RSA PRIVATE KEY-----

...

-----END RSA PRIVATE KEY-----

real    0m1.885s

user    0m0.888s

sys     0m0.075s
```

Thanks for your input though

----------

## nephros

I am getting this too.

I am pretty sure it is coming from that cronjob file, cause when you kill the cpu-hogging process you will get a mail from your cron daemon that the script failed.

I also can't reproduce the behaviour when running the command manually, and neither when running the cron script manually.

Worked around for now by removing it from cron.hourly (it is still in cron.daily), but I still would like to know what is causing this...

----------

## Rainmaker

OK, good enough for me.

Filed bug 102663

----------

