# Making a Gentoo Opteron Box

## darrylbleau

Seems there are a few people interested in building an Opteron box to run Gentoo on, and though there are threads that relate to the software side of things there don't seem to be any that relate to the hardware side of things. So, I thought I'd start one here.

There has been some discussion of the topic here.

To recap, I'm personally looking at building a Gentoo box around dual Opterons.

For motherboard, the MSI K8D Master-F looks fairly decent. Up to 333MHz DDR, 4 dimms for one CPU and 2 for the other, 3 64-Bit PCI slots and 2 32-bit. Dual on-board broadcom gigabit ethernet and onboard video.

My personal choice of CPUs are the 240s, mostly because of price. 242s (or 244s!) would be nice, though.

For power supplies, I haven't found anything better than Sparkle, and I will be putting in their 460W as it is the only 24-Pin I can seem to find. (The MSI motherboard takes a 24 pin power supply).

2 Items I'm still researching / thinking about:

RAM. How much, speed, and configuration. The Opterons are dual-channel per CPU, from what I can find out, so I would need two sticks per CPU. Also, there doesn't seem to be a 'Front Side Bus', but from what I can find out, one should use PC2700? (DDR333). Maybe someone else will be able to shed some light on this.

Hard Drives. I'm going to be putting in a RAID, and 1/0 is my preference. I'm trying to decide between a 4 drive SATA RAID with Western Digital Raptor 10k RPM 36GB, or a 4 drive IDE RAID with Western Digital Caviar SE 7200RPM 40GB. Of course the Raptors are faster but are they worth it over the Caviars? Also, I'm looking into controllers for the drives. I only want true-hardware controllers, so either the Escalade (4-port SATA or 4-PORT IDE) or the Adaptec (4-port SATA or 4-Port IDE). Any suggestions/comments on the controllers?

Please post any comments/suggestions, and if you are thinking about or in the process of putting together a Gentoo Opteron box, what sort of hardware decisions you made and why.

----------

## Moled

from the amd site:

 *Quote:*   

>  	With the memory controller integrated onto the AMD Opteron processor, the front side bus (interface to memory) runs at the speed of the processor

 

4 raptors would give you 220mb/sec which is just insane currently heh :\

----------

## taskara

hello hello  :Smile: 

ok... if each cpu is dual channel, then yeah you will need 2X ram for each cpu, but I don't know how to confirm this.

what does the MSI mainboard manual say ?

----------

## markkuk

 *darrylbleau wrote:*   

> 
> 
> RAM. How much, speed, and configuration. The Opterons are dual-channel per CPU, from what I can find out, so I would need two sticks per CPU. Also, there doesn't seem to be a 'Front Side Bus', but from what I can find out, one should use PC2700? (DDR333). Maybe someone else will be able to shed some light on this.

  You must use registered ECC RAM, the maximum speed is DDR333.

----------

## darrylbleau

I hadn't mentioned this specifically but yes, I was planning on ECC Registered RAM. I don't think you can use any other kind with this motherboard/cpu combo? It's a good idea to run ECC REG on any server, anyway.

Each CPU runs it's own dual-channel RAM, correct? (Each CPU has a built in memory controller?).

----------

## darrylbleau

Does anyone know if the Escalade SATA RAID controllers support command queueing?

----------

## taskara

does IDE itself even support command queueing ?

I didn't think it did... but maybe that's a new thing SATA does.. ?

----------

## taskara

ahh yes.. as I thought, command queueing is now (edit - sorry I mistyped as "not" lol) a technology INCORPORATED into SATA

so parallel ide will NOT support command queueing

sata does support command queueing

whether the controller includes it or not requires further research  :Very Happy: 

----------

## taskara

http://www.pcextreme.net/si3112.php

review here claims to support command queueing on the silicon image 3112 controller..

if this thing can, I'm sure a US$500 hardware controller will.

but I have submitted a question to 3ware to ask them.

I doubt they will reply, but I thought it was worth a try.

will let you know if I get a reply.

Cheers!

----------

## taskara

WELL I am quite impressed.

3Ware got back to me within 24 hours.

Here is the guy's email

 *Quote:*   

> Chris, we support command queueing on both parallel and serial 
> 
> controllers. Regards,
> 
> David Graas
> ...

 

So there you are.

They got my vote  :Smile: 

----------

## darrylbleau

Wow that's pretty sweet. I ordered a 8506-4LP kit yesterday, should have it tommorow, and the rest of the gear Thursday. I'll have benchmarks for you soon :)

----------

## taskara

groovy  :Very Happy: 

so did u go raptors or caviar in the end?

----------

## darrylbleau

Heh.

Went with 4 WD Raptors.

Power hungry, here, we are. Going to RAID 10 them. (Or 1/0 or 1-0 or however you'd like to say it).

----------

## taskara

raid 10  :Wink: 

cool.. so you should have the speed of 2 raptors in raid 0

let me know when you have some benchies  :Wink: 

----------

## firaX

lol oh well i still wonder what you re gonna use that for  :Smile: 

but i guess its all about having it eh  :Smile: 

----------

## taskara

the fact that you get a 5 yr warranty with raptor hdd's as opposed to only 1 yr with the caviar makes them better value for money I think  :Very Happy: 

----------

## darrylbleau

firaX: It's about having it first, finding uses for it second :)

Really, this server is going to handle a lot of different things, each service uses DB access. Small reads, so the transfer rate really isn't important, but the latency should be decent.

The server this is replacing has a raid 0 SCSI with two 10k maxtor atlases and it's noticeable during peak times that access to services is slower. Most of what's going on are small disk reads.

Not that it really matters, mind you, do you really care if it takes .1 second or .8 seconds to open your webmail? But we're hardware junkies and the 10k raptors look like fun to play with. :)

As for speed, we should have the read speed of 4 raptors and the write speed of 2. That's what RAID 10 is about, high performance and decent redundancy. Can't wait to see what this escalade controller does. Command queueing is just gravy :)

----------

## taskara

got a quote yeserday for the controller, US$310 so that should be good  :Smile: 

won't need it for a month or so yet tho... maybe I'll buy it myself  :Wink: 

----------

## darrylbleau

Well, we got our stuff today, and are in the process of struggling through a Gentoo install. I've started a thread on an opteron install process, hopefully we'll be able to get at least sort of 64-bit running soon :)

----------

## taskara

COOL!  :Exclamation: 

----------

## Swiss.Mage

I'm looking to buy a similar server in the next weeks ...

What about the Promise Fasttrak SX4 controller ?

And just a question about the disks ... I plan to use 4 WD Raptors but in a Raid-5 configuration ... Why use Raid-1 or Raid-10 ?

----------

## xUltra

I'm also looking into buying a Opteron server within the next few weeks... here's what I came up with:

Tyan S2880UGNR (Built in U320 SCSI)

Dual Opteron 240's

2GB PC2100 ECC Reg. Micron RAM (512x4)

Fujitsu 36GB 15K RPM Ultra160 SCSI Drive

I'm trying to stick in a 1U Rackmount... Datacenter rackspace is expensive... I've found some barbones for the Dual Opteron in a 1U solution- could cooling be a problem for it?

I'm thinking a 350W power supply will do as well....?

Also... can the servers take advantage of 64bit technology anymore? 

When will 1.4 release Opteron optimized binaries be available?

----------

## JATMAN

I am also setting up a dual Opteron server.

Hardware:

Tyan S2880GNR (no SCSI)

Dual 240s

1GB PC2700 ECC Reg. Samsung RAM (512x2)

1 - Maxtor 80GB ATA133 drive

2 - Seagate 80GB SATA drives (using on-board Promise PDC20378 controller)

Made a couple of attempts at installing the experimental AMD-64 Live CD.  Have abandoned this it is currently too incomplete and fragile for my needs and skill level.  Am now setting up using 32 bit x86 Live CD.  Purpose of the machine is as an Oracle 9.2 server for development work.  Was hoping to use this machine to play with Oracle's 64 bit developer release...

I too am very interested in the status of the AMD-64 Gentoo development.  Anyone know if an "improved" experimental release is planned anytime soon?

----------

## get sirius

Has this thread died?  I notice no new posts since October.

So how are people's plans coming along?  I ask because in the last two weeks I've managed to acquire a Tyan S2885 m/b (Thunder K8W), two Opteron 248's, two gigs of Corsair TwinX low-latency registered RAM (4x512MB), a PC Power & Cooling Turbo 510AG power supply, and a 3Ware 8506-4LP SATA-Raid card.  I'm still waiting on the case I ordered (spl case to accommodate the ext ATX sys board).  I'm going to wait on the SATA hard drives and use Maxtor 20GB ata-133 IDE drives with SATA adaptors for awhile.  When I can afford them, I'll get the WD 73GB 10K Raptors because they do support command queuing and have even faster access times than the 36gig Raptors.

So I have a vested interest in learning of your trials, tribulations, and successes.

----------

## scottsadlo

i too am making trying to put gentoo on a Tyan s2885 k8w.  I am using a single Western Digital Raptor 10k RPM 36GB drive.  Booted the AMD64 basic LiveCD via USB cd drive. Got to  the section 

"Detecting USB" (which i assume it finished doing) and then the screen refused to change.

Not sure what the issue is.  First time with Gentoo.

I tried a bunch of different boot options (nousb, doataraid, etc).

ANy thoughts

----------

## JATMAN

As noted in an earlier post, I have a Tyan S2880GNR with dual 240s.  I recently upgraded the system to 3GB RAM.   Several weeks ago I made a leap of faith and wiped out a working 32bit Gentoo system and proceded to install an early November version of the "experimental" AMD64 release.  My experience has been very positive.

Everything I have tried to install on the machine has been successful.  This includes X, KDE, Samba, Apache, tightvnc, Java (just recently installed Blackdown's 64bit JDK), and Oracle's "developer" AMD64 release of their 9i Release 2 database software.

I'm running the 2.6.0 gentoo-dev-sources kernel which is working very well.  I'm a happy guy.

----------

## Superduck

3ware is very cool indeed I'm that you decided to go with them but one thing I don't understand is why you would run raid 1+0 instead of just buying one more disc and run raid5.

----------

## taskara

 *scottsadlo wrote:*   

> i too am making trying to put gentoo on a Tyan s2885 k8w. I am using a single Western Digital Raptor 10k RPM 36GB drive. Booted the AMD64 basic LiveCD via USB cd drive. Got to the section
> 
> "Detecting USB" (which i assume it finished doing) and then the screen refused to change.
> 
> Not sure what the issue is. First time with Gentoo.
> ...

 

perhaps try the latest experimental 2.6 kernel amd64 cd?

----------

## mark

 *scottsadlo wrote:*   

> i too am making trying to put gentoo on a Tyan s2885 k8w.  I am using a single Western Digital Raptor 10k RPM 36GB drive.  Booted the AMD64 basic LiveCD via USB cd drive. Got to  the section 
> 
> "Detecting USB" (which i assume it finished doing) and then the screen refused to change.
> 
> Not sure what the issue is.  First time with Gentoo.
> ...

 

Two thoughts, although I dont speak from first hand experience.

1  I think the tiger k8w comes with an sillicon image 3114 SATA controller, support for these is highly experimental at best even in 2.6 I believe.  You may want to connect a PATA drive for now.  

2  You could join the gentoo-amd64 mailing list see

http://www.gentoo.org/main/en/lists.xml

Mark

----------

## get sirius

Mark: the s2885 is the Thunder line, not the Tiger.  Tyan has a K8W in each line (don't know why they did that!), but they're different animals.  Be that as it may, the Thunder version also has the Silicon Image SATA chipset - that's why I purchased the 3ware card.

Superduck: granted, raid 1+0 requires 4 drives vs the 3 drives needed for raid 5 but I think that raid 5 is significantly slower than 1+0 (please correct me if I'm wrong!)

----------

## malloc

Little tip, it will cost you a bit more but you'll thank me later  :Wink: 

Trade in those SATA for 2 SCSI drives 15k rpm and just buy another 2 IDE big ones for storaging purposes. Put the OS in the SCSI disks and say goodbye to bottlenecking  :Smile: 

----------

## doerrfleischfee

I've build a system containing tyan k8w (S2885) and 3ware serial ata raid controller (Escalade 8506- :Cool: ... Now I have trouble using it:

with kernel 2.6.0 (attached to pci-x bus 1 (133 MHz max)): decreased system stability, kernel module 3w-xxxx reporting fatal error "nothing to flush?", data loss

with kernel 2.6.1 (attached to pci-x bus 1):

system locks at boot when loading 3w-xxxx driver (compiled into kernel)

with kernel 2.6.0 (attached to pci-x bus 2):

3w-xxxx still producing "nothing to flush?"-error, but no data loss

with kernel 2.6.1 (attached to pci-x bus 2):

no big problems, but still bad performance:

```

# bonnie++ -u root -s 16g -n 100:50k -d /some-directory

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-

                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

machina         16G 24959  65 24993  10 17894   5 21846  71 86172  11 211.8   0

                    ------Sequential Create------ --------Random Create--------

                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

files:max            /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP

           100:50:0   327  98 200883 100 41699  99   327  98 188339 100   390  49

machina,16G,24959,65,24993,10,17894,5,21846,71,86172,11,211.8,0,100:50:0,327,98,200883,100,41699,99,327,98,188339,100,390,49

```

----------

## taskara

you may need to get the source driver from 3ware

7.7.0 is out.

compile and see how you go

----------

## Rhysem

We're also seeing poor performance from a tyan 2885 and ware raid 8506/8006 controllers. It's pulling only about 10 Mb/sec per disk. I'm not sure if it's firmware on the controllers that's the issue, the bios, linux drivers, or what.

----------

## Rhysem

3ware's claim is that you need to tune some kernel parameters with various commands. Search for linux performance on their knowledge base.

Unfortunatly, my observations with RHEL 3.0 WS show that while indeed that tuning does make the drives faster, bringing them up as a group to 4x the individual drive performance, unfortunatly that's still only 4x crap which is still crap.

My desktop with 1 drive outperforms allmost all but the best tuned stuff on the tank box with 4 drives. At best tuneing, the 4 drives win by 25% and that's frankly sad.

----------

## taskara

hmm that doesn't seem right!

did you take the kernel driver OUT and replace it with their source driver?

Or maybe you could try a "genkernel" kernel and use hotplug just for the sake of testing incase you're missing some vital component.

----------

