# Inheritied old gentoo server and this is what I found...

## tekn0mage

Hello all I just found myself the proud new maintainer of a years-old Gentoo server (thankfully the previous admin chose Gentoo). It's a custom built box using all Intel server hardware so it feels stable. However, it was put into production in 2005, and hasn't had a kernel update in a while. The current version of the kernel in use is 2.6.21. I was planning on putting gentoo-sources (2.6.30) on it but am curious if there are any caveats I should be aware of before doing so.

The box is 3000 miles away and if something were to go wrong, I would have to be on a plane to fix it (at my own expense under the contract). I am in no particular hurry, although I would like to do some changes to provide a little better peace of mind.

I've attached the emerge --info just so the information would be available in the thread. I appreciate anyone who has some time to take a look. 

```
Portage 2.1.6.13 (default/linux/x86/2008.0, gcc-4.1.2, glibc-2.6.1-r0, 2.6.21-gentoo-r4 i686)

=================================================================

System uname: Linux-2.6.21-gentoo-r4-i686-Intel-R-_Xeon-TM-_CPU_3.00GHz-with-glibc2.0

Timestamp of tree: Wed, 16 Sep 2009 11:30:01 +0000

distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled]

ccache version 2.4 [disabled]

app-shells/bash:     3.2_p17-r1

dev-java/java-config: 1.3.7, 2.0.33-r1

dev-lang/python:     2.4.4-r6, 3.1.1

dev-python/pycrypto: 2.0.1-r6

dev-util/ccache:     2.4-r7

sys-apps/baselayout: 1.12.10-r5

sys-apps/sandbox:    1.2.18.1-r2

sys-devel/autoconf:  2.13, 2.61-r2

sys-devel/automake:  1.7.9-r1, 1.9.6-r2, 1.10.1

sys-devel/binutils:  2.18-r1

sys-devel/gcc-config: 1.4.0-r4

sys-devel/libtool:   1.5.26

virtual/os-headers:  2.6.23-r3

ACCEPT_KEYWORDS="x86"

CBUILD="i686-pc-linux-gnu"

CFLAGS="-O2 -mtune=prescott -march=prescott -pipe -fomit-frame-pointer -msse3"

CHOST="i686-pc-linux-gnu"

CONFIG_PROTECT="/etc /var/bind /var/qmail/alias /var/qmail/control /var/vpopmail/etc"

CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/terminfo /etc/udev/rules.d"

CXXFLAGS="-O2 -mtune=prescott -march=prescott -pipe -fomit-frame-pointer -msse3"

DISTDIR="/usr/portage/distfiles"

FEATURES="distlocks fixpackages parallel-fetch protect-owned sandbox sfperms strict unmerge-orphans userfetch"

GENTOO_MIRRORS="ftp://ftp.ucsb.edu/pub/mirrors/linux/gentoo/ ftp://ftplib.cc.gatech.edu/pub/gentoo"

LDFLAGS="-Wl,-O1"

MAKEOPTS="-j4"

PKGDIR="/usr/portage/packages"

PORTAGE_CONFIGROOT="/"

PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"

PORTAGE_TMPDIR="/var/tmp"

PORTDIR="/usr/portage"

PORTDIR_OVERLAY="/usr/local/portage"

SYNC="rsync://rsync.gentoo.org/gentoo-portage"

USE="acl apache2 berkdb bzip2 cli cracklib crypt curl dri fortran gd gdbm gocr gpm iconv imap ipv6 isdnlog jpeg mmx mudflap mysql ncurses nls nptl nptlonly objc openmp pam pcre perl pppd python readline reflection session spamassassin spl sse ssl sysfs tcpd threads tiff unicode vpopmail x86 xorg zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" USERLAND="GNU" VIDEO_CARDS="r128"

Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LANG, LC_ALL, LINGUAS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
```

Mahalo!

----------

## asturm

If you really want to risk a kernel upgrade when things go well, best idea is probably going with 2.6.27 - not quite fresh anymore you might think, but enjoying long-term support and thus already at version 2.6.27.34. So that's the conservative bulletproof server option.  :Wink: 

----------

## loki_val

If you switch to gcc-4.3*, go with, IIRC, >=kernel-2.6.27, there were some changes there that broke things in subtle ways.

----------

## CurtE

What version of Apache2?  Before or after the structure change?

----------

## Hu

Jumping that many kernel revisions means you will probably cross at least one internal restructuring of Kconfig options.  If so, there is a risk you might leave out a driver needed to boot successfully.  I think there are known local privilege escalation vulnerabilities in the version you are running, though.  Theoretically, all you need to recover from a failed boot is to have someone come to the console and boot the old kernel, so that you can get back into it remotely.  Assuming the system has a head, this is something you could talk any average adult through over the phone.  If you cannot even get that level of help, I would say skip the kernel upgrade.

----------

## bombcar

The safest thing to do would be to image the system locally, and try to upgrade it in a VM or some other similar method. At least you'd find out surprises, hopefully.

----------

## phoenixp

Hey tekn0mage, I seem to recall grub having an option to change the default for one boot (a '-once' flag or some such).

You could make your new kernel panic if it doesn't come up all the way(panic=X), thus rebooting to your default (old) kernel. Also, add a script to /etc/init.d to make sure you have network connectivity and sshd is running, and if not, reboot.

Obviously test this out on a local machine first, ie make it fail to bring up the kernel and see if it reboots to the old one and seperately make it boot up completely but fail to run ssh/connect to the internet and reboot to old kernel.

In the case of a faraway server you can't assume it will all go well, so rather make specific plans for when it doesn't and test them locally first. Also don't forget to change your default kernel once it's working. Nothing like having your system reboot to an old kernel months/years from now when you've forgotten all about it  :Wink: 

Hope this helps,

Phen.

----------

## dmpogo

I wonder why do you need a new kernel on 2005 hardware. Any particular subsystems that were critically improved since ?    

I have 2005 server, and it gets happily on 2.6.17 kernel. Does it job, when it stops will be decomissioned.

----------

## alacheesu

Some good suggestions here, but still.. if it ain't broke, don't fix it, imo.

----------

## baeksu

 *alacheesu wrote:*   

> Some good suggestions here, but still.. if it ain't broke, don't fix it, imo.

 

But if it can be broken (vmsplice local root exploit) by Jessica Biel, it's better to fix it in advance.

Of course, there was this too: Bypassing Linux' NULL pointer dereference exploit prevention (mmap_min_addr)

So maybe a kernel upgrade is in order...

----------

## tekn0mage

Thank you all, these are all very good reads. This is a great thread that contains some timeless strategies for dealing with some real-world issues. Inheriting servers, jumping kernel versions, weighing risk vs. reward are all necessary methods for successful system administrators.

I like the idea of backing up the server and restoring to a VM. This is obviously much easier today than it was 5 years ago. I can start downloading the files today and have a fully restored copy of the server locally within 4-5 days. Since rushing is not really required, this is the safest approach. 

With Gentoo, I've noticed there are more decisions besides new kernel or not. I've seen problems between new kernel versions and outdated udev. I'll have to proceed carefully and read the 2.6.27 kernel docs thoroughly. I like the idea of switching to a "tried and true" version of the 2.6 kernel that implements the least amount of changes. Obviously being remote (although in a data center with remote hands) I don't like losing a night of sleep because of sloppy upgrades.

One question I do have is with the linux headers. When upgrading the kernel, is it required to update the kernel headers? If so, what impacts would that have on binaries that are compiled already? 

At any rate, thank you for the comments they are extremely helpful to me and I'm sure anyone viewing this thread.

----------

## krinn

go for 2.6.31 or keep it as it, 2.6.30 is still vulnerable to the latest root exploit.

if you don't care, well, just don't update the kernel. But if you care to update the kernel, why choose a kernel like 2.6.27 or 30 that still have knows holes ?

----------

## asturm

The latest root exploit you're speaking of is (was) only present in kernel 2.6.30 and above. And even if 2.6.27 had been affected, long-term support would have meant it to be fixed as soon as 2.6.30 and 2.6.31, which happened actually two months ago: http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.30.y.git;a=commit;h=76f578b630347be522b6df7917013fd0712612e5

 *tekn0mage wrote:*   

> One question I do have is with the linux headers. When upgrading the kernel, is it required to update the kernel headers? If so, what impacts would that have on binaries that are compiled already? 

 

It is not required to upgrade the kernel headers, and at the same time shouldn't impact existing binaries. According to the description they are only used when compiling your system libc, so should be upgraded beforehand if you want it to benefit from latest linux-headers' features.

----------

## dj_farid

If you have a person at the server end, this is what you could do:

Buy a cheap terminal server. One that converts serial ports to ethernet.

Make grub and linux send output to the serial port.

Configure the terminal server with an IP and send it to the person at the server.

This way you will always be able to administer the server, even if it looses network connectivity. You will be able to choose another kernel in grub...

The only thing you would need help with is a reboot during a kernel panic.

I have used these devices from a company called Tibbo with good results. They are reasonably cheap and very small.

http://tibbo.com/products/controllers/ds203.html

----------

## tekn0mage

That's a pretty sweet little device, I'd love to figure out some sort of long-term out of band management solution. The server eventually will be migrated within a year's time frame. However, it is essential to keep functions running on the machine since it is a production server. With budget constraints being an issue, a whole new server and migrating the data is just not possible at this time. Having said that, a remote solution like the one you linked might be just the ticket. 

Question: would another server in the same rack be able to function like that small device? I've never looked into running linux console over serial. I could always drop-ship a null modem cable and have the techs connect it up, if that would be a feasible solution for getting console.

Would be kinda nice to see exactly what's on the screen when a server puked, too.

----------

## Hu

Yes, if you have root access on both of them.  It is fairly common to have a headless machine for things like gateways or PVRs, and control them via serial console.  Both Grub and the kernel fully support writing to a serial console.  As far as I know, you need a real serial cable, not a USB-to-serial cable.  The latter is not supported by legacy Grub, and is only supported by the kernel once USB initializes late in the boot process.  Preparing serial support is simple, and can be tested easily with an emulator locally.  It requires no patches, so any reasonably modern Linux system can be used as a testbed.  If the remote kernel already has serial support built-in, you can even test the agetty part before you reboot for the kernel upgrade.  Kernel level support for serial is a configurable option, so it might have been left out if your predecessor did not expect to need it.

You need to configure grub, the kernel command line, and init, if you want full access.  Configuring grub will let you choose which kernel to boot.  Configuring the kernel will let you see the kernel messages during boot, including any diagnostics if the boot fails.  Configuring init will put an agetty on the serial line, so you can log in over serial in case sshd does not recover.  Configure both grub and the kernel through grub.conf.  See info grub for syntax for configuring grub itself.  Add console=ttyS0,115200 to the kernel command line to make the kernel write to the first serial port at a baud rate of 115200.  Add s0:12345:respawn:/sbin/agetty 115200 ttyS0 vt100 to /etc/inittab to put an agetty on the serial line.

For a quick emulation test, you can use app-emulation/kvm to run a guest with a serial port, and verify that a kermit on the host can negotiate with the guest correctly.

Also, as phoenixp mentioned, you can use a panic=N option on the kernel command line to automatically reboot on panic.  If your serial console logs everything it receives to a file, you can consult the file after the fact to see the panic messages, even as the kernel reboots into a known usable configuration.  You can test this in the emulator as well, by specifying a known bad root= option to the guest kernel.

----------

## dmpogo

While serial console is a more solid and feature full setup, do you know if your server has IPMI interface ?

----------

## tekn0mage

Apparently I got lucky and found the make and model of the motherboard in use. It is an Intel SE7525GP2 motherboard. It comes with BIOS-supported console redirection. Couldn't I just enable that feature, then connect up a serial cable between the two (would I need a null modem or just serial?)?

I may not have to set up any kind of kernel support for this since the server motherboard clearly supports it.

Could be a lucky break.

----------

## pappy_mcfae

Your best bet would be to configure the kernels side by side, mirroring settings from one source to the next. That will give you the highest likelihood of success. Also, I am with the voices calling for either moving to the .27, or the .31 version kernels. Since .27 remains in production, and probably will for a little more time to come, it's a fairly safe bet. Also, the latest version was released last week, and has all current exploit fixes installed. The .31 has all the most recent goodies, most of which you probably don't need.

The minimum information needed is the results of lspci -n, lsusb, and cat /proc/cpuinfo as well as the /etc/fstab file and the .config itself. I have lots of experience setting up kernels, if you're interested. If you'd prefer to do it yourself, might I suggest one of my kernel seeds. See my sig for more details.

Blessed be!

Pappy

----------

## tekn0mage

I'd love to try one of your seeds. I'm a bit concerned obviously about hardware choices. Will your seeds be relatively safe to use with a server that's 5 years old? Fortunately everything on it is mostly intel based, so it should be fairly standard. Not sure how your seeds accommodate old gear, I'd be hard pressed to have kernel problems from 3000 miles away from the actual machine. But I'm willing to give it a try.

----------

## tekn0mage

 *Hu wrote:*   

> Yes, if you have root access on both of them.  It is fairly common to have a headless machine for things like gateways or PVRs, and control them via serial console.  Both Grub and the kernel fully support writing to a serial console.  As far as I know, you need a real serial cable, not a USB-to-serial cable.  The latter is not supported by legacy Grub, and is only supported by the kernel once USB initializes late in the boot process.  Preparing serial support is simple, and can be tested easily with an emulator locally.  It requires no patches, so any reasonably modern Linux system can be used as a testbed.  If the remote kernel already has serial support built-in, you can even test the agetty part before you reboot for the kernel upgrade.  Kernel level support for serial is a configurable option, so it might have been left out if your predecessor did not expect to need it.
> 
> You need to configure grub, the kernel command line, and init, if you want full access.  Configuring grub will let you choose which kernel to boot.  Configuring the kernel will let you see the kernel messages during boot, including any diagnostics if the boot fails.  Configuring init will put an agetty on the serial line, so you can log in over serial in case sshd does not recover.  Configure both grub and the kernel through grub.conf.  See info grub for syntax for configuring grub itself.  Add console=ttyS0,115200 to the kernel command line to make the kernel write to the first serial port at a baud rate of 115200.  Add s0:12345:respawn:/sbin/agetty 115200 ttyS0 vt100 to /etc/inittab to put an agetty on the serial line.
> 
> For a quick emulation test, you can use app-emulation/kvm to run a guest with a serial port, and verify that a kermit on the host can negotiate with the guest correctly.
> ...

 

Excellent post. I'm putting console over serial support on the second machine now, and using the actual production machine (from the original post) to test if I can get the legacy machine working first. If successful, I will add console over serial support to the production machine and then reboot it without upgrading kernels. This allows me to test my cable which is just a DB9 female to DB9 female -- hopefully it is a null modem cable and not some console cable for some obscure piece of hardware.

So the only option I'm going to need to add from what I can tell is under Device Drivers -> Character Devices -> Serial Drivers -> Console on 8250/16550 and compatible serial port. Is this the proper "kernel" driver in your above post?

Thanks again.

----------

## pappy_mcfae

 *tekn0mage wrote:*   

> I'd love to try one of your seeds. I'm a bit concerned obviously about hardware choices. Will your seeds be relatively safe to use with a server that's 5 years old? Fortunately everything on it is mostly intel based, so it should be fairly standard. Not sure how your seeds accommodate old gear, I'd be hard pressed to have kernel problems from 3000 miles away from the actual machine. But I'm willing to give it a try.

 

You make the hardware choices. I set up the standard stuff. Once you get the seed, you'll have to set it up for your machine. Check out the how-to. If you'd like, send me the info I listed, and I'll set you up.

Blessed be!

Pappy

----------

## tekn0mage

This is for the OLD (legacy) server which will be my console over serial testbed:

```
victory ~ # lspci -n

00:00.0 0600: 8086:7190 (rev 03)

00:01.0 0604: 8086:7191 (rev 03)

00:02.0 0604: 1011:0024 (rev 03)

00:07.0 0601: 8086:7110 (rev 02)

00:07.1 0101: 8086:7111 (rev 01)

00:07.2 0c03: 8086:7112 (rev 01)

00:07.3 0680: 8086:7113 (rev 02)

00:0d.0 0200: 10b7:9800 (rev 24)

01:00.0 0300: 1002:4757 (rev 7a)

02:0b.0 0100: 9005:001f (rev 01)

victory ~ # lsusb

Bus 001 Device 001: ID 0000:0000  

victory ~ # cat /proc/cpuinfo

processor       : 0

vendor_id       : GenuineIntel

cpu family      : 6

model           : 8

model name      : Pentium III (Coppermine)

stepping        : 3

cpu MHz         : 797.640

cache size      : 256 KB

fdiv_bug        : no

hlt_bug         : no

f00f_bug        : no

coma_bug        : no

fpu             : yes

fpu_exception   : yes

cpuid level     : 2

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse

bogomips        : 1597.42

processor       : 1

vendor_id       : GenuineIntel

cpu family      : 6

model           : 8

model name      : Pentium III (Coppermine)

stepping        : 3

cpu MHz         : 797.640

cache size      : 256 KB

fdiv_bug        : no

hlt_bug         : no

f00f_bug        : no

coma_bug        : no

fpu             : yes

fpu_exception   : yes

cpuid level     : 2

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse

bogomips        : 1595.22

victory ~ # cat /etc/fstab

# noatime turns off atimes for increased performance (atimes normally aren't

# needed; notail increases performance of ReiserFS (at the expense of storage

# efficiency).  It's safe to drop the noatime options if you want and to 

# switch between notail and tail freely.

/dev/sda1               /boot           reiserfs        noauto,noatime          1 1

/dev/sda2               /               reiserfs        noatime                 0 0

/dev/sda3               none            swap            sw                      0 0

/dev/sdb1               /home           ext3            noatime                 0 0

/dev/sdb2               /var            reiserfs        noatime                 0 0

/dev/sdc1               /newdrive       ext3            noatime                 0 0

/dev/cdroms/cdrom0      /mnt/cdrom      iso9660         noauto,ro               0 0

none                    /proc           proc            defaults                0 0

none                    /dev/shm        tmpfs           defaults                0 0

victory ~ # 
```

And the next set of code will be for the actual production server to be upgraded:

```
ark ~ # lspci -n 

00:00.0 0600: 8086:359e (rev 0a)

00:00.1 ff00: 8086:3591 (rev 0a)

00:02.0 0604: 8086:3595 (rev 0a)

00:03.0 0604: 8086:3596 (rev 0a)

00:04.0 0604: 8086:3597 (rev 0a)

00:1c.0 0604: 8086:25ae (rev 02)

00:1d.0 0c03: 8086:25a9 (rev 02)

00:1d.1 0c03: 8086:25aa (rev 02)

00:1d.4 0880: 8086:25ab (rev 02)

00:1d.5 0800: 8086:25ac (rev 02)

00:1d.7 0c03: 8086:25ad (rev 02)

00:1e.0 0604: 8086:244e (rev 0a)

00:1f.0 0601: 8086:25a1 (rev 02)

00:1f.1 0101: 8086:25a2 (rev 02)

00:1f.2 0101: 8086:25a3 (rev 02)

00:1f.3 0c05: 8086:25a4 (rev 02)

04:02.0 0100: 9005:8016 (rev 10)

04:02.1 0100: 9005:8016 (rev 10)

05:02.0 0300: 1002:4752 (rev 27)

05:03.0 0200: 8086:1076 (rev 05)

ark ~ # lsusb

Bus 003 Device 001: ID 0000:0000  

Bus 002 Device 001: ID 0000:0000  

Bus 001 Device 001: ID 0000:0000  

ark ~ # cat /proc/cpuinfo

processor       : 0

vendor_id       : GenuineIntel

cpu family      : 15

model           : 3

model name      : Intel(R) Xeon(TM) CPU 3.00GHz

stepping        : 4

cpu MHz         : 2993.009

cache size      : 1024 KB

physical id     : 0

siblings        : 2

core id         : 0

cpu cores       : 1

fdiv_bug        : no

hlt_bug         : no

f00f_bug        : no

coma_bug        : no

fpu             : yes

fpu_exception   : yes

cpuid level     : 5

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl cid xtpr

bogomips        : 5988.82

clflush size    : 64

processor       : 1

vendor_id       : GenuineIntel

cpu family      : 15

model           : 3

model name      : Intel(R) Xeon(TM) CPU 3.00GHz

stepping        : 4

cpu MHz         : 2993.009

cache size      : 1024 KB

physical id     : 0

siblings        : 2

core id         : 0

cpu cores       : 1

fdiv_bug        : no

hlt_bug         : no

f00f_bug        : no

coma_bug        : no

fpu             : yes

fpu_exception   : yes

cpuid level     : 5

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl cid xtpr

bogomips        : 5985.57

clflush size    : 64

processor       : 2

vendor_id       : GenuineIntel

cpu family      : 15

model           : 3

model name      : Intel(R) Xeon(TM) CPU 3.00GHz

stepping        : 4

cpu MHz         : 2993.009

cache size      : 1024 KB

physical id     : 3

siblings        : 2

core id         : 0

cpu cores       : 1

fdiv_bug        : no

hlt_bug         : no

f00f_bug        : no

coma_bug        : no

fpu             : yes

fpu_exception   : yes

cpuid level     : 5

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl cid xtpr

bogomips        : 5985.60

clflush size    : 64

processor       : 3

vendor_id       : GenuineIntel

cpu family      : 15

model           : 3

model name      : Intel(R) Xeon(TM) CPU 3.00GHz

stepping        : 4

cpu MHz         : 2993.009

cache size      : 1024 KB

physical id     : 3

siblings        : 2

core id         : 0

cpu cores       : 1

fdiv_bug        : no

hlt_bug         : no

f00f_bug        : no

coma_bug        : no

fpu             : yes

fpu_exception   : yes

cpuid level     : 5

wp              : yes

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl cid xtpr

bogomips        : 5985.60

clflush size    : 64

ark ~ # cat /etc/fstab

# /etc/fstab: static file system information.

# $Header: /home/cvsroot/gentoo-src/rc-scripts/etc/fstab,v 1.14 2003/10/13 20:03:38 azarah Exp $

#

# noatime turns off atimes for increased performance (atimes normally aren't

# needed; notail increases performance of ReiserFS (at the expense of storage

# efficiency).  It's safe to drop the noatime options if you want and to 

# switch between notail and tail freely.

# <fs>                  <mountpoint>    <type>          <opts>                  <dump/pass>

# NOTE: If your BOOT partition is ReiserFS, add the notail option to opts.

/dev/sda1               /boot           ext2            noauto,noatime          1 1

/dev/sda3               /               reiserfs        noatime                 0 0

/dev/md2                /home           reiserfs        noatime                 0 0

/dev/md3                /var            reiserfs        noatime                 0 0

/dev/sdg1               /backup         reiserfs        noatime                 0 0

/dev/sdh1               /backup2        reiserfs        noatime                 0 0

/dev/sda2               none            swap            sw                      0 0

/dev/cdroms/cdrom0      /mnt/cdrom      iso9660         noauto,ro               0 0

none                    /proc           proc            defaults                0 0

none                    /dev/shm        tmpfs           defaults                0 0

ark ~ # 
```

Is that what you were looking for?

----------

## pappy_mcfae

I'd also need the .config for the machine needing update, unless that was the .config you already sent. If not, send the .config for the other machine.

Blessed be!

Pappy

----------

## tekn0mage

.config for the production server:

http://pastebin.com/m43bb7db4

.config for the old legacy server:

http://pastebin.com/m370e3717

The production server will ultimately get the upgraded kernel. I'm using the legacy system (which needs console over serial enabled) to test the console over serial so that I can gain access to the production server in case something goes horribly wrong while testing the new kernel.

At any rate, thanks again, this is looking quite doable.

----------

## tekn0mage

 *Hu wrote:*   

> Yes, if you have root access on both of them.  It is fairly common to have a headless machine for things like gateways or PVRs, and control them via serial console.  Both Grub and the kernel fully support writing to a serial console.  As far as I know, you need a real serial cable, not a USB-to-serial cable.  The latter is not supported by legacy Grub, and is only supported by the kernel once USB initializes late in the boot process.  Preparing serial support is simple, and can be tested easily with an emulator locally.  It requires no patches, so any reasonably modern Linux system can be used as a testbed.  If the remote kernel already has serial support built-in, you can even test the agetty part before you reboot for the kernel upgrade.  Kernel level support for serial is a configurable option, so it might have been left out if your predecessor did not expect to need it.
> 
> You need to configure grub, the kernel command line, and init, if you want full access.  Configuring grub will let you choose which kernel to boot.  Configuring the kernel will let you see the kernel messages during boot, including any diagnostics if the boot fails.  Configuring init will put an agetty on the serial line, so you can log in over serial in case sshd does not recover.  Configure both grub and the kernel through grub.conf.  See info grub for syntax for configuring grub itself.  Add console=ttyS0,115200 to the kernel command line to make the kernel write to the first serial port at a baud rate of 115200.  Add s0:12345:respawn:/sbin/agetty 115200 ttyS0 vt100 to /etc/inittab to put an agetty on the serial line.
> 
> For a quick emulation test, you can use app-emulation/kvm to run a guest with a serial port, and verify that a kermit on the host can negotiate with the guest correctly.
> ...

 

I was excited about the prospect of testing this serial connectivity on the machine itself, however, the kernel vm does not appear to work on hardware that does not have Intel or AMD virtual machine extensions. Thus, I will not be able use KVM to test on the machine itself.

Is there another method where I could connect up to ttyS0 on the local machine to see if the console redirect is working?

----------

## tekn0mage

Ok - I can see the agetty running on the legacy machine:

```
root      4538  0.0  0.0   1676   692 ttyS0    Ss+  14:14   0:00 /sbin/agetty 115200 ttyS0 vt100
```

I also found this in the kernel log:

```
Sep 20 06:40:30 victory [    0.000999] console [ttyS0] enabled
```

Near as I can tell, the legacy box is up and waiting for a serial connection. On the production machine, I'm using minicom to connect via serial. The port I've used is ttyS0 and ttyS1 but after pressing enter a few times, I don't ever get a login prompt. 

Is there anything else I can do to check?

----------

## pappy_mcfae

I've already done stuff like it.  :Smile:  I'll get to it in a few.

Blessed be!

Pappy

----------

## pappy_mcfae

tekn0mage,

Here is the production server .config. I set you up with 2.6.31-gentoo. It's the latest gentoo-sources as of this writing. I checked them side by side, and the new .config reflects the old, even in the obvious mistakes. Once you establish that this will work, you can get rid of some of the crap, and tweak it up. Of course, you could always just find out it boots, be happy, and never bother with it again...until the next time.

As far the machine you're going to use, I'm going to take a look as soon as I post this message.

Click here for your new .config. Compile as is.

If you're using grub, you can use fallback in case the above won't boot. If you're using lilo, it's a bit more interesting, but doable. Google "lilo fallback" for details.

Blessed be!

Pappy

----------

## tekn0mage

Thank you kindly. Now once I can get this serial console stuff looked at... I'll be ready to give it a whirl.

Turns out the production server does have a console redirect option in BIOS, which I am assuming is the IPMI feature an earlier poster was talking about. So I'm going to tinker around with it and perhaps reboot the system to give it a whirl.

----------

## pappy_mcfae

Here is the .config for the legacy server. It is also done in 2.6.31-gentoo. Have fun.

Blessed be!

Pappy

----------

## Hu

 *tekn0mage wrote:*   

> So the only option I'm going to need to add from what I can tell is under Device Drivers -> Character Devices -> Serial Drivers -> Console on 8250/16550 and compatible serial port. Is this the proper "kernel" driver in your above post?

 

Yes.  I believe that should be sufficient, but I have not checked that.  It looks from your later posts like you already got this working, though.

 *tekn0mage wrote:*   

> I was excited about the prospect of testing this serial connectivity on the machine itself, however, the kernel vm does not appear to work on hardware that does not have Intel or AMD virtual machine extensions. Thus, I will not be able use KVM to test on the machine itself.
> 
> Is there another method where I could connect up to ttyS0 on the local machine to see if the console redirect is working?

 

I meant for you to run KVM on your local desktop that you are using for day-to-day work, since it does not take long to set up and proves the concept cheaply.  Hopefully, HVM extensions are common enough by now that your primary machine has them.  HVM extensions make the KVM experience much more pleasant, but you can use app-emulation/qemu or KVM with the -no-kvm option if the host system lacks HVM extensions.  KVM is derived from Qemu, and either of these new choices will result in pure software emulation of the guest.  It is slow, but it is accurate enough to prove that you have everything configured correctly.

I forgot to mention one other change.  After you get access to the agetty over serial, verify that you can login as root.  Some systems ship with a /etc/securetty that does not permit root login over serial.  Add ttyS0 to this file if you find that to be a problem.

----------

## tekn0mage

Thank you for your reply but I did NOT get this to work. Null modem cable is connected, agetty is running but when I use minicom on the adjacent server, I cannot see anything inside of minicom.

I re-checked all my settings and still do not see anything happening.

----------

## dj_farid

 *tekn0mage wrote:*   

> Thank you for your reply but I did NOT get this to work. Null modem cable is connected, agetty is running but when I use minicom on the adjacent server, I cannot see anything inside of minicom.
> 
> I re-checked all my settings and still do not see anything happening.

 

Did you try with different speeds? 9600 baud is usually the de facto standard. I would recommend you to set it up to use 9600, if you set anything up (even if it seems rather slow).

Minicom is at least in my opinion a bit hard to configure the first time you use it. Minicom is perfect when you know that the server part is working. For easy testing I usually use putty in Windows. (I have to use Windows in my laptop for work).

----------

## tekn0mage

Kernel seed worked great!

Linux ark 2.6.31 #1 SMP Mon Sep 21 14:30:27 HST 2009 i686 Intel(R) Xeon(TM) CPU 3.00GHz GenuineIntel GNU/Linux

Excellent assistance  :Smile:  rebooted perfectly! Not to mention being 3,000 miles away added a bit of danger but when I used your kernel seed on the legacy server and it worked, I figured the newer machine would have no problem. Well, I did forget to add RAID support into the kernel, so that was a potentially bad error. Luckily for me though, the root filesystem was not on a RAID volume. So... dodged a few bullets tonight! 

It's bed time before something goes wrong :-p

Thanks again - and as far as the serial console issues, those are still being worked out. We now believe it is something to do with the serial port on the legacy (client) machine. I'm going to send a USB to serial adapter out to the location and use that for the client side. The serial ports are not working at all on the client machine, even when we went from serial A to serial B with a null modem cable, we weren't seeing any communications. A USB to serial connected to the console redirect port on the production machine will ensure the client side is not the issue. If we still run into trouble at that point, I'm not sure how I'm going to resolve it. 

There are some nice RS232 to IP adapters. We'll just have to see...if anyone has some recommendations that would be sweet.

----------

## pappy_mcfae

Cool. That's great to read.

Blessed be!

Pappy

----------

## dj_farid

 *tekn0mage wrote:*   

> There are some nice RS232 to IP adapters. We'll just have to see...if anyone has some recommendations that would be sweet.

 

I only know of the Tibbo devices. They work great. BUT they do only telnet (or some kind of raw connection). The communication goes completely in the clear.

This is not a problem as long as there is some kind of VPN. But I would never connect one of those directly to the Internet.

If anyone knows a similar device (cheap), let me know too.

----------

