# Forcing hard disk enumeration order [SOLVED]

## dufeu

I have a several generations back low end server motherboard.

I want to upgrade the running kernel from vanilla-sources-2.6.33.6 to vanilla-sources-2.6.34.1.

The problem is that this is one of those mixed IDE/SATA controller issues. What I want to do is know how to force the hard disk enumeration order the same way that the System Rescue CD does it. i.e. The 'Traditional' order - IDE first then SATA.

This board has 2 IDE ports and 8 SATA ports.

caveat - I need to confirm the which IDE slot is 0 and which is 1 - Which means I need to reboot this as I'm entering this post from the PC under discussion.

sysrescuecd enumerates the drives:

```
/dev/sda = IDE 1 0

/dev/sdb = IDE 1 1

/dev/sdc = IDE 0 0  <== DVD/CD device

IDE 1 1 not used

/dev/sdd = SATA 0

/dev/sde = SATA 1

/dev/sdf = SATA 2

/dev/sdg = SATA 3

SATA 4, 5, 6 & 7 not used.
```

vanilla-sources-2.6.33.X enumerates {by default} the drives

```
/dev/sda = SATA 0

/dev/sdb = SATA 1

/dev/sdc = SATA 2

/dev/sdd = SATA 3

SATA 4, 5, 6 & 7 not used.

/dev/sde = IDE 1 0

/dev/sdf = IDE 1 1

/dev/sdg = IDE 0 0

IDE 1 1 not used
```

The root partition resides at /dev/sde3

This was fine, I just adjusted /etc/fstab and /boot/grub/grub.conf and was happy.

I'm using this system as a btrfs based file server testing platform, so it's important to me to upgrade to the later versions of the kernel as they are released. For that reason, I'm trying to upgrade to 2.6.34.1. Note that the only the /public directory is btrfs based. The /boot partition is ext3 and the root partition is ext4. I'm not completely crazy.

vanilla-sources-2.6.34.1 is doing something weird. When I boot 2.6.34.1, I get a kernel panic. The gist is of the panic is telling me that "root=/dev/sde3" isn't valid and presents me with a list of all the partitions/devices it sees. It displays "/dev/sdf3" as part of that list. Which is really weird. When I adjust /boot/grub/grub.conf to "root=/dev/sdf3", it kernel panics again, but this time it tells me "root=/dev/sdf3" isn't valid and displays "/dev/sde3" as part of the list of partitions/devices it sees.

I'm not concerned about the weirdness so much at the moment. My primary concern is to get the kernel to boot with my disk order forced so that I have consistent disk enumeration on this server.

I know it can be done because System Rescue CD does it. But reading the kernel parameter documenation Greg K. wrote, while enlightening, doesn't really make it clear regarding what I'm looking for. i.e. There's a disconnect between the some of the parameter descriptions and their practical application. It's a great read, but I don't know enough to understand everything I'm looking at.

The only thing that looks to me like it might be what I'm looking for is "pci=nosort". Can someone give me some guidance?

Thanks!

For following was generated under kernel 2.6.33.6 and is for informational purposes only:

```
slizard ~ # uname -a

Linux slizard 2.6.33.6 #2 SMP PREEMPT Sat Jul 24 22:27:40 EDT 2010 x86_64 AMD Opteron(tm) Processor 242 AuthenticAMD GNU/Linux

slizard ~ # ls -l /dev/sd*

brw-rw---- 1 root disk 8,  0 Jul 25 04:13 /dev/sda

brw-rw---- 1 root disk 8, 16 Jul 25 04:13 /dev/sdb

brw-rw---- 1 root disk 8, 32 Jul 25 04:13 /dev/sdc

brw-rw---- 1 root disk 8, 48 Jul 25 04:13 /dev/sdd

brw-rw---- 1 root disk 8, 64 Jul 25 04:13 /dev/sde

brw-rw---- 1 root disk 8, 65 Jul 25 08:13 /dev/sde1

brw-rw---- 1 root disk 8, 66 Jul 25 04:13 /dev/sde2

brw-rw---- 1 root disk 8, 67 Jul 25 08:13 /dev/sde3

brw-rw---- 1 root disk 8, 80 Jul 25 04:13 /dev/sdf

brw-rw---- 1 root disk 8, 96 Jul 25 04:13 /dev/sdg

slizard ~ # ls -l /dev/disk/by-path

total 0

lrwxrwxrwx 1 root root  9 Jul 25 04:13 pci-0000:00:02.1-usb-0:7:1.0-scsi-0:0:0:0 -> ../../sdg

lrwxrwxrwx 1 root root  9 Jul 25 04:13 pci-0000:00:06.0-scsi-0:0:0:0 -> ../../sde

lrwxrwxrwx 1 root root 10 Jul 25 08:13 pci-0000:00:06.0-scsi-0:0:0:0-part1 -> ../../sde1

lrwxrwxrwx 1 root root 10 Jul 25 04:13 pci-0000:00:06.0-scsi-0:0:0:0-part2 -> ../../sde2

lrwxrwxrwx 1 root root 10 Jul 25 08:13 pci-0000:00:06.0-scsi-0:0:0:0-part3 -> ../../sde3

lrwxrwxrwx 1 root root  9 Jul 25 04:13 pci-0000:00:06.0-scsi-0:0:1:0 -> ../../sdf

lrwxrwxrwx 1 root root  9 Jul 25 04:13 pci-0000:00:06.0-scsi-1:0:0:0 -> ../../sr0

lrwxrwxrwx 1 root root  9 Jul 25 04:13 pci-0000:01:09.0-scsi-0:0:0:0 -> ../../sda

lrwxrwxrwx 1 root root  9 Jul 25 04:13 pci-0000:01:09.0-scsi-1:0:0:0 -> ../../sdb

lrwxrwxrwx 1 root root  9 Jul 25 04:13 pci-0000:01:09.0-scsi-2:0:0:0 -> ../../sdc

lrwxrwxrwx 1 root root  9 Jul 25 04:13 pci-0000:01:09.0-scsi-3:0:0:0 -> ../../sdd

slizard ~ # btrfs-show

failed to read /dev/sdg

failed to read /dev/sr0

failed to read /dev/pktcdvd/sr0

failed to read /dev/pktcdvd/pktcdvd0

Label: PUBLIC  uuid: b71c7140-e845-4891-a8c9-98599be7d29c

        Total devices 5 FS bytes used 2.33TB

        devid    4 size 931.51GB used 910.63GB path /dev/sdd

        devid    1 size 465.76GB used 465.02GB path /dev/sda

        devid    3 size 931.51GB used 911.62GB path /dev/sdc

        devid    5 size 233.76GB used 233.01GB path /dev/sdf

        devid    2 size 465.76GB used 465.00GB path /dev/sdb

Btrfs v0.19-13-gb72e4c4-dirty
```

----------

## NeddySeagoon

dufeu,

There are two solutions to your problem - maybe.

You could boot by disk label.  The kernel supports that but you also need to use filesystems that support disk labels. extX does, I'm not sure about others.

I've used this years ago in Red Hat but never in Gentoo.

You can boot by UUID. I believe that grub needs a patch for this and you have to use an initrd so you can get the UUIDs of your filesystems before root is mounted

I've not done this ever.

----------

## krinn

or you can just include the pata drivers, so your computer keep booting but use the sata as module so sata drives will be enumerated after the pata ones.

----------

## dufeu

First, thanks for the three suggestions. I'll discuss each suggestion and what I did.Boot by fs label. -- I looked at this first but couldn't figure it out in the time I gave myself. I searched high and low in /dev looking for a place which would give me the fs label of /dev/sde3 and couldn't find it. I now realize that you have to explicitly label the filesystem. i.e. If you haven't applied a label name to a file system, there is no such thing as a default label name. In the case of /dev/sde3, I would have had to do something like:

```
mke2fs -L ROOT /dev/sde3
```

Then I would have had to edit /boot/grub/grub.conf and change "root=/dev/sde3" to "root=/dev/disk/by-label/ROOT". 

I'm certain this is the preferred, more correct way to use the tools provided for the purpose. However, I couldn't figure this out in the time I allowed myself. I will try out something like this in the near future to a) be sure I now understand it correctly and b) that it works.  :Very Happy: 

By UUID -- Tried this and it didn't work. Either I need the afore mentioned patch {which I'm not prepared to deal with assuming I could find it} or I didn't use the correct syntax. Either way, I would need to spend more time researching this to make use of this more effectively. I did try to find a working example of this by peeking into a netbook I have with Linux Mint installed {i.e. Ubuntu based}. The "menu.lst" file used "root=/dev/sda" while /etc/fstab used the UUID to mount the partitions.

By compiling the SATA drivers as modules. These are the relevant lines from .config:

```
[*]   Verbose ATA error reporting

[*]   ATA ACPI Support

[*]   SATA Port Multiplier support

<*>   AHCI SATA support

[*]   ATA SFF support

<M>     NVIDIA SATA support

<M>     Silicon Image SATA support

<*>     AMD/NVidia PATA support
```

To support this, I edited /boot/grub/grub.con and replaced "root=/dev/sde3" with "root=/dev/sda3". I also edited /etc/fstab to reflect the same changes. And ...

It aliiiiiive!

This was my own research regarding kernel boot parameters to use pci=nosort in the kernel boot command. It didn't do anything I could see.

Again. Thanks for the suggestions. All the suggestions were helpful one way or another. I can see some things I'd like to try and investigate further and the server is now running on the version of the kernel I want it on.

 :Very Happy:   :Very Happy: 

The following is for informational purposes only:

```
slizard ~ # lsmod

Module                  Size  Used by

radeon                739747  2 

ttm                    43126  1 radeon

drm_kms_helper         23295  1 radeon

drm                   153391  5 radeon,ttm,drm_kms_helper

i2c_algo_bit            4959  1 radeon

tg3                   122159  0 

k8temp                  3211  0 

sata_sil                7996  6 

libphy                 15490  1 tg3

snd_intel8x0           26162  3 

snd_ac97_codec        107259  1 snd_intel8x0

sata_nv                21567  0 

ac97_bus                1142  1 snd_ac97_codec

i2c_nforce2             5312  0 

slizard ~ # df

Filesystem           1K-blocks      Used Available Use% Mounted on

rootfs               237185020  31154428 193982244  14% /

/dev/root            237185020  31154428 193982244  14% /

rc-svcdir                 1024       136       888  14% /lib64/rc/init.d

udev                     10240       184     10056   2% /dev

shm                    1027680        76   1027604   1% /dev/shm

/dev/sdg             3175415712 2503542288 670145008  79% /public

slizard ~ # btrfs-show

failed to read /dev/sdc

failed to read /dev/sr0

failed to read /dev/pktcdvd/sr0

failed to read /dev/pktcdvd/pktcdvd0

Label: PUBLIC  uuid: b71c7140-e845-4891-a8c9-98599be7d29c

        Total devices 5 FS bytes used 2.33TB

        devid    5 size 233.76GB used 233.01GB path /dev/sdb

        devid    3 size 931.51GB used 911.62GB path /dev/sdf

        devid    1 size 465.76GB used 465.02GB path /dev/sdd

        devid    2 size 465.76GB used 465.00GB path /dev/sde

        devid    4 size 931.51GB used 910.63GB path /dev/sdg

Btrfs v0.19-13-gb72e4c4-dirty

slizard ~ # ls -l /dev/sd*

brw-rw---- 1 root disk 8,  0 Jul 25 08:09 /dev/sda

brw-rw---- 1 root disk 8,  1 Jul 25 12:09 /dev/sda1                                             

brw-rw---- 1 root disk 8,  2 Jul 25 08:09 /dev/sda2                                             

brw-rw---- 1 root disk 8,  3 Jul 25 12:09 /dev/sda3                                             

brw-rw---- 1 root disk 8, 16 Jul 25 08:09 /dev/sdb                                              

brw-rw---- 1 root disk 8, 32 Jul 25 08:09 /dev/sdc                                              

brw-rw---- 1 root disk 8, 48 Jul 25 08:09 /dev/sdd                                              

brw-rw---- 1 root disk 8, 64 Jul 25 08:09 /dev/sde                                              

brw-rw---- 1 root disk 8, 80 Jul 25 08:09 /dev/sdf                                              

brw-rw---- 1 root disk 8, 96 Jul 25 08:09 /dev/sdg                                              

slizard ~ # ls -l /dev/disk/by-label/

total 0                                                                                         

lrwxrwxrwx 1 root root 9 Jul 25 08:09 PUBLIC -> ../../sdg                                       

slizard ~ # ls -l /dev/disk/by-uuid

total 0                                                                                         

lrwxrwxrwx 1 root root 10 Jul 25 12:09 17e9be7e-1737-4956-9ce8-d83400e3d8bc -> ../../sda1     

lrwxrwxrwx 1 root root 10 Jul 25 12:09 cd388d93-ba7c-4a65-8be3-9b8e904a61f2 -> ../../sda3       

lrwxrwxrwx 1 root root 10 Jul 25 08:09 eec01208-1216-48fa-baa0-b371f17a0ce8 -> ../../sda2

slizard ~ # uname -a

Linux slizard 2.6.34.1 #5 SMP Sun Jul 25 14:27:13 EDT 2010 x86_64 AMD Opteron(tm) Processor 242 AuthenticAMD GNU/Linux
```

----------

## devsk

Do you use initrd? Is it generated by genkernel?

Booting with LABELs is the best and most future proof method and with genkernel you do something like "real_root=LABEL=myroot real_rootflags=noatime,commit=30". No /dev/blah path involved.

You can assign labels without having to recreate the FS. Use e2label, dosfslabel, ntfslabel and assign distinct labels today.... :Smile: 

Another good approach is to write custom udev rules to label your drives with good names e.g. I have /dev/seagateA1TB for my first 1TB drive, and it can point to anything: sda or sde, depending on how I cable my drives. Then, put these rules inside your genkernel initrd. Then, u can say root=/dev/seagateA1TB3. This way, it doesn't matter whether your drive is sda or sde, root=/dev/seagateA1TB3 is still valid. I can provide u the udev rule and the custom script I wrote for creating the well-known-name symlinks if you want.

----------

## dufeu

 *devsk wrote:*   

> Do you use initrd? Is it generated by genkernel?

 

Nope and nope. I've never liked genkernel because it caused too many problems for me. I suppose I should explain that. 

It has always offended my sense of what's right when I saw older, but otherwise still perfectly good computer equipment being thrown out during a hardware refresh. So I would intercept the best pieces and refurbish {using linux of course} them. I'd then either donate to non-profits or to friends. The problem was I used to run across a lot of strange combinations of off brand motherboards and mystery brand add-in cards. genkernel is great if the hardware you're working with is common and well behaved. I just had too many problems where the kernel would load the wrong drivers or conflicting drivers.

In fact, the server motherboard currently under discussion still doesn't get the correct hwmon drivers assigned if I let sensors-detect or the kernel do it. They always chooses w83627htf. The hardware monitor chip which is installed is actually the W83792AD/D which means it needs the w83792d driver. If I have both compiled as modules, the wrong driver is always the one loaded into the kernel.

I ended up just keeping tools like System Rescue or Gentoo Installation CDs on hand. Actually, I keep multiple versions of each CD on hand due to the borked BIOS issues I run into. Then I run tools like lspci and lsmod and isapnptools to figure out what was actually present. Finally, it becomes a matter of real simple hand tweaking of the needed kernel settings to only have those drivers the system requires. There have been a lot of improvements made in auto-detection. But I still run into problems.

The problem with genkernel for me is that it need(ed)s to work on all the systems I work(ed) on and it doesn't(didn't).

As for initrd, I finally actually have a reason to learn how to use it for the first time. I'm planning on building a PC with a btrfs based root. So initrd is going to be a requirement. But ... I still need to learn/practice how.

 *devsk wrote:*   

> Booting with LABELs is the best and most future proof method and with genkernel you do something like "real_root=LABEL=myroot real_rootflags=noatime,commit=30". No /dev/blah path involved.
> 
> You can assign labels without having to recreate the FS. Use e2label, dosfslabel, ntfslabel and assign distinct labels today....

 

Here's another one of those: "I finally need to learn to use it" things. Up till now, I've never had a reason to move beyond the default assigned paths for devices. But ... to work with mulit-device btrfs file systems, it's really easiest to label them. 

Here's a free tip: Don't make the btrfs file system and then label it. You'll lose the one you made and a new one will be created with the label specified. I still need to post an email to the mailing btrfs mail list about this with some suggestions.

You are correct that people should start doing this, but there are issues regarding coordination not only with the kernel but also with grub/lilo capabilities as well as potential Label issues - depending on file system of course.

 *devsk wrote:*   

> Another good approach is to write custom udev rules to label your drives with good names e.g. I have /dev/seagateA1TB for my first 1TB drive, and it can point to anything: sda or sde, depending on how I cable my drives. Then, put these rules inside your genkernel initrd. Then, u can say root=/dev/seagateA1TB3. This way, it doesn't matter whether your drive is sda or sde, root=/dev/seagateA1TB3 is still valid. I can provide u the udev rule and the custom script I wrote for creating the well-known-name symlinks if you want.

 

Not ready for this yet. But I'll keep it in the back of my head.

Thanks for the input!

----------

## devsk

Never use genkernel for creating kernel image. NEVER! Its dumb in that department!

Use it for what it is good for. For creating an initramfs which you can customize to your heart's content. It is awesome for that! This helps you setup your devices for encryption, LVM, DMRAID, MD, software suspend, TOI, BTRFS volume auto-detection, make boot look pretty etc...you name it! Its really good for that.

What I do to create a multi-system livecd is to create a custom livecd using i686 gentoo stage3, an i686 SMP kernel made for generic architecture with no optimizations, and everything but most common things, compiled as modules and squashed to fit on a CD with squashfs. Same i686 custom livecd (or rather liveUSB: its just a squashfs image, which I can even boot from a USB drive or CD or a hard disk) boots all my systems, even the one from 2002.

One more practice I follow for easy recovery is my first partition on every drive I have is a 4GB FAT32 drive which is bootable grub partition. It has a squashfs image to boot Gentoo livecd style, BartPE to boot windows XP, Win98 (for many DOS things like flashing firmware on my SSDs or flashing BIOS) and OpenSolaris image to boot OSOL.

----------

## krinn

i personally use the label trick too, but not from grub (i don't want to play with initrd & such).

but, you may like to use it too.

udev auto-create /dev/disk/by-label/ and /dev/disk/by-UUID/ entries when it find one (for label of course, UUID are always there).

so without doing any other tricks you can just assign your fstab with the /dev/disk/by-label(UUID) entries to easier your management (you will, as i said, still need to handle big disk change in grub, but as grub is so flexible, i just edit the line and change it before booting in the grub menu "the e key") and my fstab will always handle the change.

have a look

```
/dev/disk/by-label/belegBOOT      /boot      ext2      noauto,noatime   1 2

/dev/disk/by-label/belegROOT      /      ext3      noatime0 1

/dev/disk/by-label/belegSWAP      none      swap      sw   0 0

/dev/disk/by-label/belegHOME            /home           ext3

ls /dev/disk/by-label/

belegBOOT  belegHOME  belegROOT  belegSWAP

```

another easy thing i use is the generic grub entry

```
title Gentoo

root (hd0,1)

kernel /vmlinuz root=/dev/sdb1

title Gentoo Previous kernel

root (hd0,1)

kernel /vmlinuz.old root=/dev/sdb1

```

because when doing make install from a kernel, if kernel find the /boot/vmlinuz & grub, it will backup that file as vmlinuz.old and put the newer kernel there as vmlinuz.

in fact it will do that for many files, look at my current /boot, except for files "failsafe" & the fbsplash ones, i never create myself all the others files, yeah i'm sooooo lazzy

it's also useful for the kernel config file, named by kernel version...

```
ls /boot

boot                  fbsplash-livecd-2006.1-1024x768  System.map-2.6.33-gentoo

config                fbsplash-livecd-2007.0-1024x768  System.map.old

config-2.6.31-gentoo  grub                             vmlinuz

config-2.6.33-gentoo  lost+found                       vmlinuz-2.6.31-gentoo

config.old            System.map                       vmlinuz-2.6.33-gentoo

failsafe              System.map-2.6.31-gentoo         vmlinuz.old

```

----------

## devsk

I see people are really scared of initrd as if it was some sort of plague... :Very Happy: 

Its as easy to create as a single command:

```
genkernel --splash=livecd-2007.0 --disklabel --mdadm --dmraid --splash-res=1600x1200 --no-initrdmodules --initramfs-overlay=/tmp/initramfs-overlay initrd
```

This creates a /boot/initramfs-x86_64-<KV>, which you need to put inside the grub.conf.

Ok, may be there is a bit of genkernel setup in /etc/genkernel.conf as well but defaults are typically good enough.

----------

