# framebuffer causing slow IO (kicks many drives from RAID6)

## dinominant

I spent a few hours yesterday systematically removing and exchanging hardware our main storage server trying to figure out why so many drives were getting kicked from the raid array (all the time) when the drives passed smart and badblocks tests. I tried removing everything but the 1 controller card in use, moving it to different PCIe slots and even upgraded the kernel from 3.6.30.9 to 2.6.31.12.

I found out that when I boot with append="video=vesafb" and vga=0x3E6 or any other vga set I get incredibly poor IO performance on my SAS controller card. It's so slow that many drives get kicked from the RAID6 array taking the server offline.:

```
masterbackup ~ # cat /proc/mdstat

Personalities : [raid0] [raid1] [raid6] [raid5] [raid4]

md1 : active raid6 sdk2[10] sdj2[5] sdi2[3] sdh2[7] sdf2[6] sdg2[4] sde2[8] sdl2[11] sda2[9] sdb2[1] sdd2[0] sdc2[2]

      4979200 blocks level 6, 4k chunk, algorithm 2 [12/12] [UUUUUUUUUUUU]

md2 : active raid6 sdk3[10] sdj3[9] sdi3[3] sdh3[7] sdf3[6] sdg3[4] sde3[8] sda3[11] sdb3[1] sdd3[0] sdc3[2]

      4873075840 blocks level 6, 4k chunk, algorithm 2 [12/11] [UUUUU_UUUUUU]

md0 : active raid1 sdk1[10] sdj1[5] sdi1[3] sdh1[7] sdf1[6] sdg1[4] sde1[8] sdl1[11] sda1[9] sdb1[1] sdd1[0] sdc1[2]

      128384 blocks [12/12] [UUUUUUUUUUUU]

unused devices: <none>

masterbackup ~ # hdparm -tT /dev/sd[a-z] /dev/md2

/dev/sda:

 Timing cached reads:   7902 MB in  2.00 seconds = 3952.76 MB/sec

 Timing buffered disk reads:  326 MB in  3.02 seconds = 108.07 MB/sec

/dev/sdb:

 Timing cached reads:   7924 MB in  2.00 seconds = 3963.73 MB/sec

 Timing buffered disk reads:  304 MB in  3.01 seconds = 100.95 MB/sec

/dev/sdc:

 Timing cached reads:   7868 MB in  2.00 seconds = 3936.27 MB/sec

 Timing buffered disk reads:  330 MB in  3.00 seconds = 109.85 MB/sec

/dev/sdd:

 Timing cached reads:   7856 MB in  2.00 seconds = 3929.92 MB/sec

 Timing buffered disk reads:  296 MB in  3.00 seconds =  98.65 MB/sec

/dev/sde:

 Timing cached reads:   7786 MB in  2.00 seconds = 3894.57 MB/sec

 Timing buffered disk reads:  310 MB in  3.01 seconds = 102.82 MB/sec

/dev/sdf:

 Timing cached reads:   7842 MB in  2.00 seconds = 3923.13 MB/sec

 Timing buffered disk reads:  314 MB in  3.02 seconds = 104.05 MB/sec

/dev/sdg:

 Timing cached reads:   7804 MB in  2.00 seconds = 3903.63 MB/sec

 Timing buffered disk reads:  324 MB in  3.01 seconds = 107.68 MB/sec

/dev/sdh:

 Timing cached reads:   7856 MB in  2.00 seconds = 3930.13 MB/sec

 Timing buffered disk reads:  322 MB in  3.01 seconds = 106.98 MB/sec

/dev/sdi:

 Timing cached reads:   7864 MB in  2.00 seconds = 3933.84 MB/sec

 Timing buffered disk reads:  322 MB in  3.00 seconds = 107.32 MB/sec

/dev/sdj:

 Timing cached reads:   7794 MB in  2.00 seconds = 3899.08 MB/sec

 Timing buffered disk reads:  324 MB in  3.00 seconds = 107.83 MB/sec

/dev/sdk:

 Timing cached reads:   7862 MB in  2.00 seconds = 3932.73 MB/sec

 Timing buffered disk reads:  314 MB in  3.01 seconds = 104.35 MB/sec

/dev/sdl:

 Timing cached reads:   7818 MB in  2.00 seconds = 3910.93 MB/sec

 Timing buffered disk reads:  232 MB in  3.01 seconds =  77.12 MB/sec

/dev/md2:

 Timing cached reads:   7912 MB in  2.00 seconds = 3958.34 MB/sec

 Timing buffered disk reads:   56 MB in  3.10 seconds =  18.06 MB/sec

masterbackup ~ #
```

Notice that /dev/md2 is transfering at only 18MB/s. Yesterday every device on that SAS controller were reading at only 2MB/s and /dev/md2 was reading at only 1MB/s! Today the individual devices all perform as expected on the SAS controller but the on-board sata controller has some slower devices (/dev/sdl) at 77MB/s when it should be roughly be 100MB/s. And /dev/sdl is the device that was kicked from the array (today, yesterday it was whole groups of devices on the SAS controller).

However, when I boot without append="video=vesafb" and vga=0x3E6 the performance is what it should be:

```
masterbackup ~ # cat /proc/mdstat

Personalities : [raid0] [raid1] [raid6] [raid5] [raid4]

md1 : active raid6 sdk2[10] sdj2[5] sdi2[3] sdh2[7] sdg2[4] sdf2[6] sde2[8] sdl2[11] sda2[9] sdb2[1] sdd2[0] sdc2[2]

      4979200 blocks level 6, 4k chunk, algorithm 2 [12/12] [UUUUUUUUUUUU]

md2 : active raid6 sdk3[10] sdj3[9] sdi3[3] sdh3[7] sdg3[4] sdf3[6] sde3[8] sdl3[5] sda3[11] sdb3[1] sdd3[0] sdc3[2]

      4873075840 blocks level 6, 4k chunk, algorithm 2 [12/12] [UUUUUUUUUUUU]

md0 : active raid1 sdk1[10] sdj1[5] sdi1[3] sdh1[7] sdg1[4] sdf1[6] sde1[8] sdl1[11] sda1[9] sdb1[1] sdd1[0] sdc1[2]

      128384 blocks [12/12] [UUUUUUUUUUUU]

unused devices: <none>

masterbackup ~ # hdparm -tT /dev/sd[a-z] /dev/md2

/dev/sda:

 Timing cached reads:   7816 MB in  2.00 seconds = 3909.58 MB/sec

 Timing buffered disk reads:  326 MB in  3.02 seconds = 108.12 MB/sec

/dev/sdb:

 Timing cached reads:   7844 MB in  2.00 seconds = 3924.34 MB/sec

 Timing buffered disk reads:  304 MB in  3.00 seconds = 101.19 MB/sec

/dev/sdc:

 Timing cached reads:   7790 MB in  2.00 seconds = 3897.15 MB/sec

 Timing buffered disk reads:  330 MB in  3.00 seconds = 109.96 MB/sec

/dev/sdd:

 Timing cached reads:   7746 MB in  2.00 seconds = 3874.48 MB/sec

 Timing buffered disk reads:  298 MB in  3.01 seconds =  98.88 MB/sec

/dev/sde:

 Timing cached reads:   7764 MB in  2.00 seconds = 3883.71 MB/sec

 Timing buffered disk reads:  320 MB in  3.07 seconds = 104.29 MB/sec

/dev/sdf:

 Timing cached reads:   7778 MB in  2.00 seconds = 3891.09 MB/sec

 Timing buffered disk reads:  314 MB in  3.01 seconds = 104.26 MB/sec

/dev/sdg:

 Timing cached reads:   7834 MB in  2.00 seconds = 3919.38 MB/sec

 Timing buffered disk reads:  324 MB in  3.00 seconds = 107.98 MB/sec

/dev/sdh:

 Timing cached reads:   7782 MB in  2.00 seconds = 3892.45 MB/sec

 Timing buffered disk reads:  322 MB in  3.01 seconds = 107.00 MB/sec

/dev/sdi:

 Timing cached reads:   7792 MB in  2.00 seconds = 3897.77 MB/sec

 Timing buffered disk reads:  322 MB in  3.00 seconds = 107.30 MB/sec

/dev/sdj:

 Timing cached reads:   7774 MB in  2.00 seconds = 3888.96 MB/sec

 Timing buffered disk reads:  326 MB in  3.00 seconds = 108.52 MB/sec

/dev/sdk:

 Timing cached reads:   7764 MB in  2.00 seconds = 3884.06 MB/sec

 Timing buffered disk reads:  318 MB in  3.02 seconds = 105.42 MB/sec

/dev/sdl:

 Timing cached reads:   7754 MB in  2.00 seconds = 3878.81 MB/sec

 Timing buffered disk reads:  228 MB in  3.01 seconds =  75.77 MB/sec

/dev/md2:

 Timing cached reads:   8212 MB in  2.00 seconds = 4108.13 MB/sec

 Timing buffered disk reads:  588 MB in  3.01 seconds = 195.41 MB/sec
```

What concerns me is the root cause. While I'm currently booting without CLI framebuffer to fix the slowness I'm concerned that the root cause will re-surface at a later date, likely when we need this system most and have lots of data to lose.

Why could booting with a framebuffer and higher-than-default resolution cause very poor IO?

----------

## Ant P.

Is this an onboard graphics chip? In that case it'll be eating main memory bandwidth constantly just to display the screen - a 1024x768 fb will be using over 1GB/s.

----------

## dinominant

No, its a dedicated video card:

```
01:05.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY [Radeon 7000/VE]
```

```
masterbackup ~ # lspci

00:00.0 Host bridge: ATI Technologies Inc RD790 Northbridge only dual slot PCI-e_GFX and HT3 K8 part

00:02.0 PCI bridge: ATI Technologies Inc RD790 PCI to PCI bridge (external gfx0 port A)

00:06.0 PCI bridge: ATI Technologies Inc RD790 PCI to PCI bridge (PCI express gpp port C)

00:07.0 PCI bridge: ATI Technologies Inc RD790 PCI to PCI bridge (PCI express gpp port D)

00:0c.0 PCI bridge: ATI Technologies Inc RD790 PCI to PCI bridge (external gfx1 port B)

00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode]

00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 Controller

00:12.1 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller

00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller

00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 Controller

00:13.1 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller

00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller

00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 3a)

00:14.1 IDE interface: ATI Technologies Inc SB700/SB800 IDE Controller

00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA)

00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host controller

00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge

00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2 Controller

00:18.0 Host bridge: Advanced Micro Devices [AMD] K10 [Opteron, Athlon64, Sempron] HyperTransport Configuration

00:18.1 Host bridge: Advanced Micro Devices [AMD] K10 [Opteron, Athlon64, Sempron] Address Map

00:18.2 Host bridge: Advanced Micro Devices [AMD] K10 [Opteron, Athlon64, Sempron] DRAM Controller

00:18.3 Host bridge: Advanced Micro Devices [AMD] K10 [Opteron, Athlon64, Sempron] Miscellaneous Control

00:18.4 Host bridge: Advanced Micro Devices [AMD] K10 [Opteron, Athlon64, Sempron] Link Control

01:05.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY [Radeon 7000/VE]

01:06.0 PCI bridge: Hint Corp HB6 Universal PCI-PCI bridge (non-transparent mode) (rev 11)

02:08.0 USB Controller: NEC Corporation USB (rev 41)

02:08.1 USB Controller: NEC Corporation USB (rev 41)

02:08.2 USB Controller: NEC Corporation USB 2.0 (rev 02)

02:09.0 FireWire (IEEE 1394): NEC Corporation uPD72874 IEEE1394 OHCI 1.1 3-port PHY-Link Ctrlr (rev 01)

02:0a.0 Mass storage controller: Silicon Image, Inc. SiI 3112 [SATALink/SATARaid] Serial ATA Controller (rev 02)

03:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08)

04:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. Device 3403

05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)

06:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
```

----------

## roarinelk

Build a kernel with "radeonfb" instead of vesafb; it should

work with your 1st-gen radeon.

----------

