# Very slow boot with new kernel

## Provider

Hello, I have had a problem with my kernels ever since version 2.6.27 and more.  The last good kernel that I have working is version 2.6.26-gentoo-r3.

The problem is that the system take a very long time to boot up to the login prompt, at least 10 minutes.  But the boot process start at usual speed then stalls at the "Wiping /tmp directory" for a long time, then continues the boot process but very very slowly.   Samething with the shutdown process, it takes for ever to shutdown or reboot.

This week I have tried the latest stable version gentoo-sources 2.6.30-r4 and I have the same problem.  I configured the kernel manually from scratch, meaning I have not taken the .config file from another version to the 2.6.30-r4 version.  I have tried the Pappy's seeds I read in other threads in this forums, the only thing I have not tried is to use the genkernel tool to see if that will build a good kernel.

My system is a dual CPU Xeon 2.4GHz, Asus PCH-DL motherboard (Intel 875P chipset), 1 gig DDR ECC RAM with a IDE 40 HDD using ext3 file system.

Here is my emerge --info:

```

Portage 2.1.6.13 (default/linux/x86/2008.0, gcc-4.3.2, glibc-2.9_p20081201-r2, 2.6.26-gentoo-r3 i686)

=================================================================

System uname: Linux-2.6.26-gentoo-r3-i686-Intel-R-_Xeon-TM-_CPU_2.40GHz-with-gentoo-1.12.11.1

Timestamp of tree: Wed, 26 Aug 2009 14:15:01 +0000

app-shells/bash:     3.2_p39

dev-lang/python:     2.4.6, 2.5.4-r3, 2.6.2-r1

dev-python/pycrypto: 2.0.1-r8

dev-util/cmake:      2.6.4

sys-apps/baselayout: 1.12.11.1

sys-apps/sandbox:    1.6-r2

sys-devel/autoconf:  2.13, 2.63-r1

sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.2

sys-devel/binutils:  2.18-r3

sys-devel/gcc-config: 1.4.1

sys-devel/libtool:   1.5.26

virtual/os-headers:  2.6.27-r2

ACCEPT_KEYWORDS="x86"

CBUILD="i686-pc-linux-gnu"

CFLAGS="-O2 -march=pentium4 -pipe"

CHOST="i686-pc-linux-gnu"

CONFIG_PROTECT="/etc"

CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/udev/rules.d"

CXXFLAGS="-O2 -march=pentium4 -pipe"

DISTDIR="/usr/portage/distfiles"

FEATURES="distlocks fixpackages parallel-fetch protect-owned sandbox sfperms strict unmerge-orphans userfetch"

GENTOO_MIRRORS="ftp://mirrors.tera-byte.com/pub/gentoo/ ftp://gentoo.arcticnetwork.ca/pub/gentoo/ ftp://gentoo.mirrored.ca/ http://adelie.polymtl.ca/"

LANG="en_US.utf8"

LDFLAGS="-Wl,-O1"

LINGUAS="en fr"

MAKEOPTS="-j5"

PKGDIR="/usr/portage/packages"

PORTAGE_CONFIGROOT="/"

PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"

PORTAGE_TMPDIR="/var/tmp"

PORTDIR="/usr/portage"

SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage"

USE="X acl alsa berkdb bzip2 cli cracklib crypt cups dri dvd fortran gdbm gnome gpm gtk hal iconv ipv6 isdnlog logrotate mmx mmxext mudflap ncurses nls nptl nptlonly openmp pam pcre perl pppd python qt4 readline reflection samba session spl sse sse-filters sse2 ssl sysfs tcpd unicode x86 xorg xulrunner zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en fr" USERLAND="GNU" VIDEO_CARDS="mga"

Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LC_ALL, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY

```

Let me know if you need more info, like the .config file or the output of lspci or what ever.

Thank you

------------------

Eric

----------

## tony-curtis

Going to guess you don't have DMA on the disk(s) with an IDE driver.  If so...

maybe something in the kernel changed.  Check the newer kernel config to see if your chipset support has changed.

Try

```
hdparm /dev/hda
```

or whatever the disk is called to see if DMA is supported.

I had this issue and switched to SATA (/dev/sda etc).

----------

## Provider

hdparm reports the same in both kernel versions (2.6.26-r3 and 2.6.30-r4), DMA is on:

```

Linux # hdparm /dev/hda

/dev/hda:

 multcount     = 16 (on)

 IO_support    =  0 (default) 

 unmaskirq     =  0 (off)

 using_dma     =  1 (on)

 keepsettings  =  0 (off)

 readonly      =  0 (off)

 readahead     = 256 (on)

 geometry      = 77557/16/63, sectors = 78177792, start = 0

Linux / # 

```

But I do found it weird that the IO_support and unmaskirq are different than another system that I have running the excate same HDD.  That other system is not the same hardware, it is an Athlon XP

```

/dev/hda:

 multcount     = 16 (on)

 IO_support    =  1 (32-bit)

 unmaskirq     =  1 (on)

 using_dma     =  1 (on)

 keepsettings  =  0 (off)

 readonly      =  0 (off)

 readahead     = 256 (on)

 geometry      = 77557/16/63, sectors = 78177792, start = 0

```

----------

## jcat

I'm not saying this is the problem, but you should probably migrate to SATA drivers in the kernel at some point, I believe it's meant to handle older IDE drives now as well.

Is there nothing in the system logs or dmesg to suggest that there's an issue?

Maybe there's a drive issue that's no being reported. Have you done any drive performance tests?

What about a a simple

```
hdparm -tT /dev/hda
```

..what's the output?

Cheers,

jcat

----------

## energyman76b

do you use exactly the same configs? Have you twiddled with timer or debug options? Enabled useless stuff like namespaces and other stuff?

what about dmesg? errors there? Have you checked the filesystem?

----------

## Provider

I wanted to switch to the SATA based driver for a while so I am in the process of doing that.

But I just remembered something, since its has been a long time since I did some troubleshooting for this problem, I got the slow boot only when I enabled HIGHMEM support in the kernel with version 2.6.26.  I have 1 gig of RAM that is why i wanted to have highmem enabled.  Now it is not enabled as you can see in this dmesg output:

```

Linux ~ # dmesg | more

Linux version 2.6.26-gentoo-r3 (root@Linux) (gcc version 4.1.2 (Gentoo 4.1.2)) #1 SMP Wed Dec 3 22:33:31 EST 2008

BIOS-provided physical RAM map:

 BIOS-e820: 0000000000000000 - 000000000009e400 (usable)

 BIOS-e820: 000000000009e400 - 00000000000a0000 (reserved)

 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)

 BIOS-e820: 0000000000100000 - 000000003f960000 (usable)

 BIOS-e820: 000000003f960000 - 000000003f963000 (ACPI NVS)

 BIOS-e820: 000000003f963000 - 000000003f970000 (ACPI data)

 BIOS-e820: 000000003f970000 - 000000003f980000 (reserved)

 BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved)

Warning only 896MB will be used.

Use a HIGHMEM enabled kernel.

896MB LOWMEM available.

found SMP MP-table at [c00f5940] 000f5940

Entering add_active_range(0, 0, 229376) 0 entries of 256 used

Zone PFN ranges:

  DMA             0 ->     4096

  Normal       4096 ->   229376

Movable zone start PFN for each node

early_node_map[1] active PFN ranges

    0:        0 ->   229376

On node 0 totalpages: 229376

  DMA zone: 32 pages used for memmap

  DMA zone: 0 pages reserved

  DMA zone: 4064 pages, LIFO batch:0

  Normal zone: 1760 pages used for memmap

  Normal zone: 223520 pages, LIFO batch:31

  Movable zone: 0 pages used for memmap

DMI 2.3 present.
```

I remember that if i enabled highmem support with that version i got the same slow boot problem.

jcat, I will have the info requested tomorrow.

energyman, Yes namespaces is enable in my kernel and I can not disable it, is that a very bad thing?  It is also enabled in my working 2.6.26 kernel, but like i said with lowmem support.  I will have to compile a lowmem 2.6.30 kernel to test it out.

Thanks

-----

Eric

----------

## energyman76b

no, it should not make a difference  :Wink: 

namespaces, that is.

highmen is a different beast...

----------

## Provider

Ok, I did the hdparm -tT /dev/hda test that was suggested by jcat, and the results are the same with both version kernel.

```
Linux eric # hdparm -tT /dev/hda

/dev/hda:

 Timing cached reads:   1286 MB in  2.00 seconds = 643.11 MB/sec

 Timing buffered disk reads:  120 MB in  3.00 seconds =  39.94 MB/sec

Linux eric # 

```

I did not get the SATA drivers to work.  I think I do not have the right device for the root= parameter in Grub since the SATA drivers rename the device name and I did not troubleshoot it much also.  "MAXTOR3" is a udev rule i created that maps to /dev/hda3 which is my root partition.  I created that rule because I knew the device name would change with the new SATA drivers and I wanted a consistant name if I had to switch between old legacy PATA and new SATA drivers.   I get this error when I boot up: 

```
VFS: Cannot open root device  "MAXTOR3" or unknown-block(2,0)

Pleas append a correct "root=" boot option;

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
```

Anyway I don't think it is a hard drive issue, I think it is more a RAM configuration problem in the kernel. Because I have the problem when I enabled HIGHMEM in the kernel config.  But I did not try to disable the HIGHMEM support in the new 2.6.30 kernel yet.  I will do so now.

----------

## energyman76b

udev rules don't help you, because rootfs has to be mounted to read the udev rules.

So either replace hda with sda in fstab/grub or mount-by-uuid.

----------

## Provider

I tried replacing hda with sda, but thet did not work.  I already have 4 other SATA drive in the server and the PATA drive, so maybe everything got moved around and the PATA drive is not sda.  

Anyway, I recompiled a 2.6.30 kernel with highmem support  disabled and everything works fine.  I changed some other stuff also, I changed  CONFIG_CLASSIC_RCU=y, CONFIG_EMBEDDED=y and CONFIG_CPU_SUP_INTEL=y

I guess the new kernel changed something, or the option I enabled, because it does not complain about the HIGHMEM option anymore

```
    0.000000] ACPI: Local APIC address 0xfee00000

[    0.000000] 1017MB LOWMEM available.

[    0.000000]   mapped low ram: 0 - 3f960000

[    0.000000]   low ram: 0 - 3f960000

[    0.000000]   node 0 low ram: 00000000 - 3f960000

[    0.000000]   node 0 bootmap 00011000 - 00018f2c

[    0.000000] (8 early reservations) ==> bootmem [0000000000 - 003f960000]

[    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]

[    0.000000]   #1 [0000001000 - 0000002000]    EX TRAMPOLINE ==> [0000001000 - 0000002000]

[    0.000000]   #2 [0000006000 - 0000007000]       TRAMPOLINE ==> [0000006000 - 0000007000]

[    0.000000]   #3 [0000200000 - 00007b4b64]    TEXT DATA BSS ==> [0000200000 - 00007b4b64]

[    0.000000]   #4 [000009e400 - 0000100000]    BIOS reserved ==> [000009e400 - 0000100000]

[    0.000000]   #5 [00007b5000 - 00007b812d]              BRK ==> [00007b5000 - 00007b812d]

[    0.000000]   #6 [0000010000 - 0000011000]          PGTABLE ==> [0000010000 - 0000011000]

[    0.000000]   #7 [0000011000 - 0000019000]          BOOTMAP ==> [0000011000 - 0000019000]
```

It sees the full 1 gig RAM now.

I might try to isolate the option that was giving me trouble, if  I have time.

----------

## Provider

I think i may have found something.  If  I take a look at dmesg, i see that where i get the stall is when the swap is being added

```

[   19.308552]  md1: unknown partition table

[   20.944888] kjournald starting.  Commit interval 5 seconds

[   20.945146] EXT3 FS on hda1, internal journal

[   20.945155] EXT3-fs: mounted filesystem with writeback data mode.

[   20.997435] kjournald starting.  Commit interval 5 seconds

[   21.028080] EXT3 FS on md0, internal journal

[   21.028088] EXT3-fs: mounted filesystem with writeback data mode.

[   21.135870] kjournald starting.  Commit interval 5 seconds

[   21.165872] EXT3 FS on md1, internal journal

[   21.165881] EXT3-fs: mounted filesystem with writeback data mode.

[   21.398393] Adding 1959920k swap on /dev/hda2.  Priority:-1 extents:1 across:1959920k 

[  121.611419] e1000: eth0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX

[  177.161056] w83627hf: Found W83627THF chip at 0x290

[  198.879093] PPP generic driver version 2.4.2

[  201.464939] PPP BSD Compression module registered

[  207.718101] ip_tables: (C) 2000-2006 Netfilter Core Team

[  212.318242] ctnetlink v0.93: registering with nfnetlink.

[  213.074899] ClusterIP Version 0.8 loaded successfully

```

This output is from the 2.6.30 kernel I compiled today, it load faster than all the other 2.6.30 i compiled, but there is still a difference in boot speed compared to my 2.6.26 kernel.  The kernel 2.6.26 i used before booted much faster.

----------

## energyman76b

21.398393] Adding 1959920k swap on /dev/hda2.  Priority:-1 extents:1 across:1959920k 

 [  121.611419] e1000: eth0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX 

 [  177.161056] w83627hf: Found W83627THF chip at 0x290 

 [  198.879093] PPP generic driver version 2.4.2 

there is some serious crap going on.

mkswap - and remove e1000 - is boot still slow?

----------

## Provider

I have a question, that might sound stupid, but I have to ask it since I do not have the answer for it.  How can I run mkswap on a running system that already has a swap partition created and usable in the system?  If I try to run the command it says the device is busy.  Do I have to do it with a bootable CD?

----------

## energyman76b

swapoff -a

mkswap /dev/yourswappartition

swapon -a

----------

