# [SOLVED] guest on VMWare ESX 3.5 - problem modifying kernel

## cgmoller

Hi

I am trying to upgrade the kernel of a guest OS on an ESX 3.5 server.  I am running the 2.6.20-r7 kernel and I am aware of the LSI bug and have been trying to get around this with the various fixes proposed

https://forums.gentoo.org/viewtopic-p-5026838-highlight-.html#5026838

http://lkml.org/lkml/2007/3/23/345

At this point I am no longer trying to upgrade the kernel - I want to change the controller driver on the current kernel version.  However, the issue has gotten ... complicated.

I have made some changes to the existing kernel configuration - adding the buslogic driver.  I was going to confirm that the new kernel would still boot on the LSI controller with both drivers built in before I switched to the BusLogic controller.

I have found that despite recompiling the kernel the guest always boots into *THE* previously working version

I first compiled this kernel Apr 27, 2007.  I have subsequently recompiled this same kernel several times, most recently Aug 25 2008 and Sep 17 2008.  These are the only 2 versions of the kernel currently living in /boot/.

However, when I reboot I find 2 disturbing things.

1.  uname -a reports the following "... 2.6.20-gentoo-r7 #1 SMP Fri Apr 27 15:36:54 JAPAN 2007 ..."

2.  zcat /proc/config.gz reports this:  " ... # Fri Apr 27 15:14:45 2007 ... "

It is as if the original kernel is squirrled away somewhere.

What makes this difficult to troubleshoot is that the grub menu does not come up on the console.  I get the VMWare BIOS splash and a line about the grub menu flashes by (the same one that flashes by before the grub spashimage would come up).  Then it stays blank for a while and eventually starts showing the progress of the reboot, but only after it has selected a kernel.

I have commented out the splashimage and added vesafb parameters in the grub menu, but no change.

If anyone has a any thoughts ... well, thanks in advance.Last edited by cgmoller on Tue Oct 28, 2008 1:54 am; edited 1 time in total

----------

## VinzC

Stupid question: have you checked ls -lh /boot/ ?

----------

## cgmoller

not even ls -la reveals anything  - I am baffled.

----------

## VinzC

 *cgmoller wrote:*   

> not even ls -la reveals anything  - I am baffled.

 

I am too... where does your running kernel come from then?  :Shocked: 

Does it exist on a separate /boot partition? (which seems useless to me with virtual disks but I may be wrong.)

BTW post the results of grep -i boot fstab here.

----------

## jcat

If your kernels are supposed to be in /boot/ (or wherever), make sure it's mounted before you copy in the new kernel in.

For example, you might install a kernel in /boot/, but /boot/ may not mounted at boot time (this is generally considered more secure), hence you are really just copying the kernel to a folder in / .  So although you have a shiny new kernel on partition /dev/foo, the old kernel may still be on partition /dev/bar, and that will be used a boot.

Cheers,

jcat

----------

## cgmoller

All (Anyone who may still be following this ... ) Apologies - I had a bad virus that took me out for a week a while ago and am only now getting back to this.  

Per your requests for more information on the current running state:

The running kernel ...

 *Quote:*   

> # uname -a
> 
> Linux iltkyvm99 2.6.20-gentoo-r7 #1 SMP Fri Apr 27 15:36:54 JAPAN 2007 x86_64 Dual-Core AMD Opteron(tm) Processor 2218 AuthenticAMD GNU/Linux

 

The contents of /boot/ ...

 *Quote:*   

> # ls -lh /boot/
> 
> total 10M
> 
> lrwxrwxrwx 1 root root   27 Sep 17 15:40 System.map -> System.map-2.6.25-gentoo-r7
> ...

 

The contents of /proc/config.gz

 *Quote:*   

> # zcat /proc/config.gz | more
> 
> #
> 
> # Automatically generated make config: don't edit
> ...

 

records of boot in fstab

 *Quote:*   

> # grep -i boot /etc/fstab
> 
> # NOTE: If your BOOT partition is ReiserFS, add the notail option to opts.
> 
> /dev/sda1               /boot           ext3            noauto,noatime  1 2
> ...

 

----------

## VinzC

The only reason a different kernel is loaded is because grub.conf contains a line like this:

```
kernel /boot/vmlinuz-2.6.20-gentoo-r7 [...]

...
```

Please check  :Wink:  .

----------

## cgmoller

grub.conf is below - please note the kernel refers to /vmlinuz which is soft linked to the newer kernel.  I have also just cleaned up (reverted back) and re-entered my *.good kernel.

 *Quote:*   

> 
> 
> # cat /boot/grub/grub.conf
> 
> #
> ...

 

fdisk

 *Quote:*   

> /dev/sda1   *           1          17      136521   83  Linux
> 
> /dev/sda2              18         142     1004062+  82  Linux swap / Solaris
> 
> /dev/sda3             143        2088    15631245   83  Linux

 

fstab - boot entry

 *Quote:*   

> /dev/sda1               /boot           ext3            noauto,noatime  1 2

 

----------

## jcat

 *cgmoller wrote:*   

> 
> 
> fstab - boot entry
> 
>  *Quote:*   /dev/sda1               /boot           ext3            noauto,noatime  1 2 

 

You have specified "noauto", so before you you any further, please post the output of 

```
mount /boot
```

 and 

```
ls -la /boot
```

 and 

```
cat /boot/grub/grub.conf
```

Make sure you run those commands in the order listed  :Smile: 

Cheers,

jcat

----------

## VinzC

```
...

drwxr-xr-x 2 root root 4.0K Aug 25 16:18 tmp

...
```

This might be not my business but what's in /boot/tmp?

EDIT: BTW also post the results of cat /proc/cmdline, just in case.

----------

## cgmoller

VinzC & jcat - Thank you for your assistance.  As per your request - I provide the information below.  caveat: it is getting stranger by the moment / reboot.  Read on.

Results of mount /boot were empty so I also included mount

 *Quote:*   

> iltkyvm99 grub # mount /boot
> 
> iltkyvm99 grub # mount
> 
> /dev/sda3 on / type ext3 (rw,noatime)
> ...

 

ls -la /boot  Pleae note the version numbers and dates.  These predate the ones listed from yesterday (!?!?!)

 *Quote:*   

> iltkyvm99 grub # ls -la /boot
> 
> total 6888
> 
> drwxr-xr-x  4 root root    1024 Aug 20 18:16 .
> ...

 

Contents of grub.conf

 *Quote:*   

> iltkyvm99 grub # cat /boot/grub/grub.conf
> 
> #
> 
> # Sample boot menu configuration file
> ...

 

/boot/tmp

 *Quote:*   

> iltkyvm99 grub # ls -la /boot/tmp
> 
> ls: cannot access /boot/tmp: No such file or directory
> 
> iltkyvm99 grub #

 

 *Quote:*   

> iltkyvm99 grub # cat /proc/cmdline
> 
> root=/dev/sda3

 

Something I noticed here.  Yesterday, I did a little housekeeping, removing various file that I know aren't working.  e.g. From my history:

 *Quote:*   

> 
> 
>   430  unlink System.map
> 
>   431  ln -s System.map-2.6.20-gentoo-r7 System.map
> ...

 

I have done nothing else except for rebooting.  The curious thing, compare ls /boot from this morning with that of yesterday (earlier post).  Now it is showing *2.6.24-gentoo-r3 which was not there yesterday and the date on *2.6.20-gentoo-r7 is April 29 2007 whereas yesterday it was September 17 2008.

Another disconcerting thing re grub.conf.  grub.conf from yesterday as compared to grub.conf this morning above.

 *Quote:*   

> iltkyvm99 bin # cat /boot/grub/grub.conf
> 
> #
> 
> timeout 9
> ...

 

These are two totally different files.  I had removed the framebufffer stuff in hopes of getting to the grub menu during boot.  I suspected that the framebuffer was not working correctly and effectively suppressing the grub menu.  So the difference between yesterday's grub.conf and today's was a reboot.  This may also be the same reason why /boot/tmp from yesterday is now missing today.

Just for yucks I am going to delete the *24-gentoo-r3 and reboot to see what happens since it does not work either.

----------

## jcat

Well, it's just as I indicated in my first post   :Wink: 

What you have been doing is modifying the contents of the wrong /boot !  Your /boot directory (and everything under it) is not mounted at boot time, therefore unless you 

```
mount /boot
```

 before you install a new kernel or install/modify grub, your are putting files in the wrong /boot

The following should help to illustrate, all though obviously these paths aren't _real_ paths as such:

You have been putting things in

```
sda3/boot
```

But you should be putting files in

```
sda1/boot
```

So you probably need to make sure you have the latest version of grub in your real /boot by

```
mount /boot
```

```
emerge -1av grub
```

```
grub-install hd0
```

Then install your latest version on the kernel and update grub.conf if you need to  :Smile: 

Cheers,

jcat

----------

## jcat

P.S.

Just in case you require further clarification, this may explain it more succinctly.

The reason why your /boot was different yesterday to today is because /dev/sda3 wasn't mounted as /boot yesterday  :Smile: 

Cheers,

jcat

----------

## cgmoller

Doh!  Oh what a junior admin stunt.  Well that did it.  I had forgotten that I had set /boot to noauto and it did not sink in when you mentioned it.  Subtle as a flying mallet.

Thank you.

----------

## jcat

Hehe, subtlety is not usually something I'm accused of   :Wink: 

Glad you've sussed it  :Smile: 

Cheers,

jcat

----------

