# Bad performance 3ware Escalade RAID-controller

## Sorcerer

Hi!

I recently bought a new serverputer and put a RAID-controller (Escalade with 2 ports - kernel recognizes it as a scsi-controller, which is a good/normal thing) in it for RAID1.

Well, the machine's performance is below-average. Writing stuff causes high load! Transferring stuff with 10 MB/s causes a 0.25 load.

hdparm gives me this output:

hdparm -Tt /dev/sda

/dev/sda:

 Timing buffer-cache reads:   1052 MB in  2.00 seconds = 524.69 MB/sec

 Timing buffered disk reads:  116 MB in  3.04 seconds =  38.10 MB/sec

But what bothers me most is that sometimes my load shoots up from 0.00 to 0.30 within 5 seconds! That means for a second or so there must be a HUGE load on the system. When running vmstat simultaneously, looking for anomalies at the same time the load shoots up, I see nothing abnormal, except for cpu-usage going up 15% or so for 1 second.

I use a gentoo-sources kernel 2.4.20-r6 with Low-Latency Patch enabled, and don't use any abnormal options. Would it be a good idea to enable Preemptible kernel as well btw?

I run a few gameservers (UT) on the box, but what's really weird, is that if people are playing, the cpu-load at a certain point is about 40%, while the load average shoots up to above 1.00!!! How is this possible? The game uses UDP (of course) to transmit and receive its data, and doesn't do a lot of disk io.

I've got 1.5 GB of RAM, and tried disabling swap before (1.5 Gig is for swap - bit of overkill  :Razz: ) but that didn't have any positive effect at all.

I hope someone recognizes this problem and knows how to solve it  :Smile: 

Thanks in advance!

----------

## Heretic

What motherboard?  What's the model on your controller card?  Using PCI-X?

----------

## Sorcerer

My motherboard is an Asus A7V8X-X. All cards (video, network, raid) are on a PCI risercard.

The RAID-controller is a 3ware Escalade 7000-2. The harddrives are 2x 120 GB Western Digital JB1200 w/ 8 MB cache.

The 3ware site says this about it:

- PCI 2.2 compliant 64-bit/66MHz bus master

(33 MHz compliant)

Anyone got tips/ideas?

----------

## Sorcerer

Also I'd like to add that the rest of the performance of the machine is 'weird'... I often see that if a gameserver which runs on the device, uses 20% CPU-capacity, the load average shoots up to 0.50 or similar... 40% = peaks of 1.00 and so on I guess...

Again: I have preempted kernel disabled and low latency patch enabled. Might it not have something to do with this?

----------

## Heretic

 *Sorcerer wrote:*   

> Also I'd like to add that the rest of the performance of the machine is 'weird'... I often see that if a gameserver which runs on the device, uses 20% CPU-capacity, the load average shoots up to 0.50 or similar... 40% = peaks of 1.00 and so on I guess...
> 
> Again: I have preempted kernel disabled and low latency patch enabled. Might it not have something to do with this?

 

I also run a server that hosts several gameservers.  There are regularly 40 people connected to my box across 3 gaming servers.

Load is equal to the average number of processes in the run queue (ie running at once).  A load average higher than your CPU usage means that you are I/O bound.  What's happening is you have processes waiting on I/O to happen.  There's definitely some sort of problem and/or bottleneck there I suspect, your drives should get closer to 100MB/sec sustained read performance.  I have a WD1200JB in my home computer, and I get 40-50MB/sec sustained on it.  Do you know if anything else is using a lot of I/O on your computer?

When you ran the hdparm test, was there anything running on your server?  Just downloading a large file @ 300KByte/s from my server to my house lowers my hdparm scores about 5MB/sec.  My server runs a single 18GB 10K RPM Ultra2 SCSI drive that sustains around 25MB/sec, but has 4.3ms access time.  

Oh, yes.  Definitely enable BOTH the preemptible and low-latency kernel patches.  They're very nice for real-time services, ie a gaming server.  I still run the 2.4.20-gentoo-r2 kernel since my server has a 178 day uptime I haven't had the oppurtunity to try any new kernels on it.

Oh, it's also best to leave on some swap, even if less than your RAM.

----------

## Heretic

 *Sorcerer wrote:*   

> My motherboard is an Asus A7V8X-X. All cards (video, network, raid) are on a PCI risercard.
> 
> The RAID-controller is a 3ware Escalade 7000-2. The harddrives are 2x 120 GB Western Digital JB1200 w/ 8 MB cache.
> 
> The 3ware site says this about it:
> ...

 

Hrm, I was thinking about this some more.  There is the possibility that your PCI bus is saturated, or there are driver problems.  I would try, say, either the latest ck or ac kernel to see if you get better performance with either of those kernels.  There have been a lot of drivers changes/improvements between 2.4.20 and 2.4.22-whatever.

I'm assuming this box hasn't been deployeed to a datacenter yet or whatever.  Another thing you could try is to run software RAID1 using VIA's IDE controllers.  They aren't on the PCI bus, so if you're saturating the PCI bus, this will avoid said bottleneck--the southbridge has it's on seperate 533 MB/s link to the north bridge.

http://unthought.net/Software-RAID.HOWTO/

The only trick with this is that you have to manually setup the MBR for both harddrives if you want to use GRUB.  That just means you have to run setup() from grub on both harddrives before putting them in RAID1.

----------

## Sorcerer

The server is in a datacenter, so trying dangerous things is not an option. This is the first server in which I've used hardware RAID, and I did not expect these kinds of problems.

I'm going to the center next week, and I'll prepare a few kernels before I go, so I can try different kernels there quickly. I don't think it's saturation, because also when there's 0.00 load and no players, load sometimes tends to shoot up to 0.30 without any cpu-load showing up in vmstat or where-ever...

Maybe the server doesn't like the videocard or something like that btw? I could always replace it of course  :Smile: 

As I said I run gentoo-sources now, but I think a pretty much bare vanilla-sources may be better in my case.

Btw, the install docs say this:

 *Quote:*   

> There is also a gaming-sources kernel optimized for game-playing responsiveness that works wonderfully for this purpose when the "Preemptible kernel" option is enabled. 

 What do you think about that?  :Wink: 

----------

## Heretic

Check out the ck-sources.  It's the vanilla sources plus about 8 patches--preempt, low-latency, O(1) scheduler, etc.  Incredibly useful patches.  Definitely enable preemptible kernel support.  Nothing is using your video card right?

I don't know about the gaming sources, sounds like it's for client computers.  You should get more performance than you're getting...  Something is killing your I/O...

----------

## Sorcerer

Hmm, I'll try to get as much info as possible about the different types of kernels and then compile a set to try a few different options when I'm at the datacenter.  :Smile: 

I'll also take along another videocard in case the current one (old Riva 128) is the cause of the problems...

Is there a good overview with descriptions of the different types of kernels somewhere?

----------

## Heretic

 *Sorcerer wrote:*   

> Hmm, I'll try to get as much info as possible about the different types of kernels and then compile a set to try a few different options when I'm at the datacenter. 
> 
> I'll also take along another videocard in case the current one (old Riva 128) is the cause of the problems...
> 
> Is there a good overview with descriptions of the different types of kernels somewhere?

 

This is for the ck-sources: http://members.optusnet.com.au/ckolivas/kernel/

Let me know how swapping out the video card helps.   A cheap AGP graphics card might help.  I actually run a PCI Matrox Millenium LOL.  I'm looking at a 3Ware 8605-4 RAID 1+0 setup so I'm keen on seeing you get higher performance.  I'm waiting to run it on a motherboard with PCI-X however.

----------

## Heretic

http://www.gentoo.org/proj/en/metastructure/kernel.xml

There's a list of sources I just stumbled onto.

----------

## Sorcerer

Can't put an AGP card in there... Plus I just found out I don't have any spare cards, but I also found out that the load is always caused by writing or reading to or from the disk...

Which kernel do you recommend and how do I know if all fixes I need (critical updates for raidcontroller perhaps?) are patched in that kernel?

----------

## Sorcerer

Btw what do I need to do with the Hz setting?  :Very Happy: 

And which kernel-sources do you - or other people - recommend then? ck or ac?

Latency/preemptibility/efficiency are the only criteria. The machine won't be used for gaming, but for serving games.

----------

## Heretic

 *Sorcerer wrote:*   

> Which kernel do you recommend and how do I know if all fixes I need (critical updates for raidcontroller perhaps?) are patched in that kernel?

 

Look at the changelog for the kernel since 2.4.20 and then test to make sure.  Trial and error.

 *Sorcerer wrote:*   

> Btw what do I need to do with the Hz setting?  
> 
> And which kernel-sources do you - or other people - recommend then? ck or ac? 

 

1000 Hz for me.

I just found a source tree in that link I posted I never new about: Gentoo Stable (gs-sources).  It seems to be the most complete 2.4.22 kernel.  I plan to upgrade to this on next reboot.

----------

## Sorcerer

Ok, I've got an ac-kernel, a ck-kernel, plus a gentoo-sources r7-kernel with exactly the same options as my current kernel with added preemptible kernel-support. I used 200 Hz for the gentoo-sources kernel btw. ck didn't offer an option to set Hz and I set the ac-kernel to 1000 Hz.

I'll go and fetch gs-sources now as well and build that too... Monday I will go to the datacenter and benchmark all built kernels.

You or anyone else got an idea on how to test them as extensive as possible?

----------

## Heretic

Well, you're looking for improved disk performance.  Use bonnie++ and hdparm -tT

You should get at least 80MB/sec read throughput.  Hey, are you using ReiserFS?

----------

## Sorcerer

I only get 45 MB throughput, but I use RAID1 (non-striped) and 45 seems to be okay for this Western Digital...

Anyway, I upgraded to ac-kernel. Haven't seen a lot of players in 1 time yet, so I hope to report on performance soon  :Smile: 

It seemed to be a little faster though now... very curious what happens under higher player counts  :Smile: 

----------

## Heretic

 *Sorcerer wrote:*   

> I only get 45 MB throughput, but I use RAID1 (non-striped) and 45 seems to be okay for this Western Digital...
> 
> Anyway, I upgraded to ac-kernel. Haven't seen a lot of players in 1 time yet, so I hope to report on performance soon 
> 
> It seemed to be a little faster though now... very curious what happens under higher player counts 

 

RAID1 still uses both harddrives in parallel for reading.  Reading should be about twice as fast as writing.  When writing, it writes the same data to both harddrives, but when reading it reads different parts from each harddrive to increase performance over a single HD configuration.

----------

## Sorcerer

Hmm, I guess something's wrong with either the controller or the disks, or perhaps even the cable?

----------

## Heretic

 *Sorcerer wrote:*   

> Hmm, I guess something's wrong with either the controller or the disks, or perhaps even the cable?

 

You use two cables right?  One per channel?

----------

## Sorcerer

 *Heretic wrote:*   

>  *Sorcerer wrote:*   Hmm, I guess something's wrong with either the controller or the disks, or perhaps even the cable? 
> 
> You use two cables right?  One per channel?

 Yes, of course! How else could I connect two drives lol?

----------

## Heretic

 *Sorcerer wrote:*   

>  *Heretic wrote:*    *Sorcerer wrote:*   Hmm, I guess something's wrong with either the controller or the disks, or perhaps even the cable? 
> 
> You use two cables right?  One per channel? Yes, of course! How else could I connect two drives lol?

 

One cable with two drives...  Oh, but the 3Ware only does one drive per channel.  Yup...  No other way.  Hah.

gs-sources-2.4.23_pre6-r1

----------

