# Building a cluster!

## nephros

Okay I have a friend who is into 3D modelling and rendering, and who has too much money but not enough FLOPS  :Smile: 

So he wants to buy about ten machines and turn them into a renderfarm, and has asked me to set those up for him.

Hardware in question will be about 10x 1GHz CPUs (the cheapest we can get, AMD most likely), and 512MB SDRAM (memory speed is not an issue) each. We will also need a *large* storage solution.

Software will be the Alias|Wavefront Maya renderer, running on Linux obvioulsy, and possibly Gentoo. (And yes, I plan to have distccd running there as well  :Twisted Evil:  )

I am currently googling about this, but I also wanted to ask you guys for recommendations on:

Which clustering technology to use? (Mosix vs. OpenMosix vs. Beowulf(?) vs. other?)

Should we get consumer-grade mobos in bulk, or are there cheap rack-mountable solutions? 

How about building a rack DIY-style in an old filing cabinet? Cooling?

Which motherboards are there without all the unecessary stuff like onboard VGA/sound, USB etc.

Are 100MB/s + switch needed or will 10Mbit/s NICs suffice? (EDIT: scratch that, openMosix sais 100MBit or firewire.)

Is it a good idea to have the nodes diskless, and mount the OS images from a master machine via NFS? (Network overhead?)

Is it probably smarter for a render farm to not cluster at all, but simply runing the machines n parallel (probably still diskless)?

General caveats, what are the problems I am likely to run into?

I am grateful for any suggestions.

Sorry if this is the wrong Forum, but it _is_ about Gentoo-compatible hardware in part.

----------

## lbrtuk

Strangely, I was thinking about this recently. Only for use with Blender/Yafray.

I figured out that I would be able to make a 32 node cluster of Athlon XP 2000+s for £6158.

Which isn't bad if you think about it.

 *nephros wrote:*   

> 
> 
> Which clustering technology to use? (Mosix vs. OpenMosix vs. Beowulf(?) vs. other?)
> 
> 

 

I'm an openmosix fanboy, as there will be some things you want to run which aren't network enabled. ie- many renderers nowadays have this network renderfarm stuff built in. However, if you ever want to write your own code and run it or run different code, anything which is multithreaded will take advantage of the cluster.

 *nephros wrote:*   

> 
> 
> Should we get consumer-grade mobos in bulk, or are there cheap rack-mountable solutions? 
> 
> 

 

I personally would go for the consumer grade mobos. Because really, consumer grade hardware nowadays is getting stupidly cheap.

 *nephros wrote:*   

> 
> 
> How about building a rack DIY-style in an old filing cabinet? Cooling?
> 
> 

 

You can save about £18 per node if you don't bother with cases. Might be fun to rig up your own storage system, but keep in mind that you'll want all the mobos accessible so if one fails, you can just remove one without having to dismantle the whole system. Plus if you go with cases you dont have to worry about cooling. But it will take up more space.

 *nephros wrote:*   

> 
> 
> Which motherboards are there without all the unecessary stuff like onboard VGA/sound, USB etc.
> 
> 

 

In my experience, ironically, the cheapest mobos are the ones with things on them simply because they are sold in bulk. So it may be best to settle for the fact that you'll have 32 onboard vga chips that aren't doing anything. If you aren't going to bother with cases, definitely get ones with onboard lan (they all do nowadays i think): you'll be able to pack the mobos closer together if they dont all have ethernet cards coming up from them.

 *nephros wrote:*   

> 
> 
> Are 100MB/s + switch needed or will 10Mbit/s NICs suffice? (EDIT: scratch that, openMosix sais 100MBit or firewire.)
> 
> 

 

You try buying something that isn't 100Mbps nowadays  :Wink: 

Seriously, It's almost as though they've stopped making hubs, switches are everywhere. And very cheap. Consider using a gigabit backbone if you use more than one switch.

 *nephros wrote:*   

> 
> 
> Is it a good idea to have the nodes diskless, and mount the OS images from a master machine via NFS? (Network overhead?)
> 
> 

 

Booting.

The million dollar question.

The real way of doing things is netboot. EEPROMs on nics and nfs. And yes, that is how you should do it.

But IMHO that can be a real pain in the arse. So consider cheating. At the end of the day, all you need is something very cheap (& reliable: floppies are out) to just boot from.

USB memory sticks? (if all your bioses can boot from usb)

Or a real hack: cdrom drives can be had for about £12 today. Burn 32 bootable cds. After all, it's just for the booting, after that it's all done over nfs. Or even just leave the machine running in the livecd environment and just write the output to nfs.

 *nephros wrote:*   

> 
> 
> General caveats, what are the problems I am likely to run into?
> 
> 

 

Hire monkeys to do all the repetitive work. I hear bonobos do a good job.

----------

## nephros

Thanks a lot.

Browsing my local computer dealer homepages I would come to ~200 Euros per (AMD Duron based)node, adding up to about 3000 Euros for the whole 10-way thing, rack-building cost not included.

What's the deal with all clusters being 2^n nodes? Any special reason for that?

My next concern is power. Assuming each node eats ~100Watts under full load, will the normal housing power lines have problems with this?

----------

## lbrtuk

 *nephros wrote:*   

> 
> 
> What's the deal with all clusters being 2^n nodes? Any special reason for that?
> 
> 

 

Erm. Efficiency. Everywhere in computer science you will find instances where the most efficient number of something is a power of two. It really depends on the situation as to why it turns out like this. In this case it is probably primarily to do with the threading model.

 *nephros wrote:*   

> 
> 
> My next concern is power. Assuming each node eats ~100Watts under full load, will the normal housing power lines have problems with this?

 

My house has a max load of 100A. Here in the UK we have around 230V RMS, so we can supply 23kW. ish.

An assumption of 100W would be a good approximation for just a motherboard. Remember to add overhead - switches and servers. (raid array?)

32 * 100W = 3.2kW

Round up to 4kW to account for overhead.

And if power costs are an issue, just run it in winter, and use it to heat your house. Because after all, that's where all the energy is going in the end.

It should be fine. Of course it'll be even easier for you, as I'm accounting for 32 nodes.

----------

## timfreeman

Check out this setup

https://forums.gentoo.org/viewtopic.php?t=54913

(There are a lot of openmosix threads here of course, I'm enjoying this one:

https://forums.gentoo.org/viewtopic.php?t=41171 )

(I've got no advice, I've never done it)

----------

## waverider202

Check out what clustering solutions Maya supports, cause you are stuck any other way.  I've played around with mosix and openmosix.  Maya will probably not work with these.  mosix/openmosix do process migration, so a process on one computer will migrate to another, but Maya is one process.  Even if Maya is multithreaded, threads don't migrate, only whole processes.  Beuwolf clusters will only work for software that is programmed specifically for a beuwolf cluster.  Maya may be programmed for this.  Maya may also have it's own clustering solution built in.  I'ave played around with cinalerra, which has its own built in clustering solution.  You definitly want to go with 100base+.  If you can afford hdd,s definitly get them.  At that point, its up to you how you want to do the os (local or net boot), but you are gonna want local swap space for each machine.  Esp with only 512mb of ram.  Unfortunatly, I've never played with Maya, so I don't know what to say past this.

----------

## lbrtuk

 *waverider202 wrote:*   

> ...threads don't migrate, only whole processes.

 

oh. didn't realise that.

----------

## nephros

 *waverider202 wrote:*   

> Check out what clustering solutions Maya supports, cause you are stuck any other way.  I've played around with mosix and openmosix.  Maya will probably not work with these.  mosix/openmosix do process migration, so a process on one computer will migrate to another, but Maya is one process.  Even if Maya is multithreaded, threads don't migrate, only whole processes.  Beuwolf clusters will only work for software that is programmed specifically for a beuwolf cluster.  Maya may be programmed for this.  Maya may also have it's own clustering solution built in.  I'ave played around with cinalerra, which has its own built in clustering solution.  You definitly want to go with 100base+.  If you can afford hdd,s definitly get them.  At that point, its up to you how you want to do the os (local or net boot), but you are gonna want local swap space for each machine.  Esp with only 512mb of ram.  Unfortunatly, I've never played with Maya, so I don't know what to say past this.

 

Good point. Still, if all else fails, I could do some scripting and let the cluster run one Render process per node and let those round-robin about the animation frames.

Maya lets you specifiy how many cpus to use, I will have to look into this.

Also thanks for the hint with swap, NFS won't work for swap space directly, correct?

So if I want to have it networked I will have to use swap files. After all, file I/O isn't the bottleneck here, CPU cycles are.

Any estimations about network bandwidth use for the clustering alone, without NFS?

----------

