# High performance redundant mail servers

## RageX^NZ

I am currently planning the implementation of a high volume mail server for a local ISP.  

This post is to ask the Gentoo community for their experiences and reccomendations on high performance/reliability mail servers.

Currently they are using Postfix and Courier-IMAP for their roughly 6000 accounts with no virus or spam filtering.

I am planning on implementing a system with the following requirements/features:

-	High reliability.

-	Scalability, e.g. must be able to grow substantially without downtime

-	Ease of management.

-	Virus detection and removal.

-	Spam detection and marking.

-	Must support Secure Password Authentication to allow users on other ISPs to authenticate with the MTA and relay mail.

-	Operates on commodity hardware.

-	Fully redundant operation! 

the software I am planning on using is:

Postfix

Courier-Imap

Amavisd with integral Spam-Assasin

NOD32 for virus scanning from Amavis

ClamAV as a secondary virus scanner

Replicated MySQL or LDAP for configuration settings

I was planning in having a cluster of servers, initially starting with two initial SMTP/POP servers and a single virus scanning machine.

As load increased I would increase the virus scanning machines, I was planning on using round robin DNS to do primitive load balancing on the virus scanning machines.

I was also planning on doing this to load balance the pop3/smtp data and having low TTL's on the record so that if a server failed the NS record could quite quickly be changed to only use the working server.

As always your reccomendations are really appreciated.

----------

## RageX^NZ

I forgot to mention that I will be documenting this completely for the benefit of the community.

I also forgot to mention that I was planning on having the maildirs shared using CodaFS so all machines of course could access the same info.

----------

## SpinDizzy

Not sure I have any answers for you, just a few thoughts.

It's best to consider pop and smtp to be two completely seperate services and treat them as such.

I've found the highest load we get on our email servers is from disk access on the pop servers. Our smtp servers run virus and spam scanning locally because their load is quite small generally.

Lots of users checking their mailboxes can cause plenty of disk thrashing so your solution will have to include a method of spreading this load (ie: their mailboxes) across several machines as required.

I'm still in the middle of the above, so I'm not much help   :Sad:  .

----------

## Kope

Definitely stripe your storage disks. Better yet, get a seperate RAID box just for storage. 

Mail is all about the disk access.

----------

## tuxmin

For HA take a look at this http://drbd.cubit.at/. I've been using this with apache with great success. For performance optimization you would need two dedicated Gigabit NICs.

Alex!!

----------

## kashani

You can think of your machines as one of 4 clusters and then build them as appropiate. In your case this might not work so well since you probably don't want more than 4-5 machines. Here are the groups, but I'll do a general config for the time being.

smtp - sends mail for your users

pop3 - provides pop access for your users

mx - recieves mail from other providers

filter - does your filtering

Two boxes sounds too few and three too many for the number of users. As long as you know a single box can handle the load two total servers should be fine. I'd build something along the lines of:

P4

1GB RAM

128MB RAID card, 18GB SCSI drive 15K RPM

or something similar

Of the above the RAID card and the fast drives are the most important. Mail is ALL about I/O. SMALL drives are also going to be best. We aren't going to keep any mail locally and smaller/faster drives on the servers will increase performance.

The RAID card is there to even out I/O by turning those short reads and writes into long ones. Mail is all about moving 10k files around and the raid card will help us with that. BTW you don't want to use anything other than .maildir format. 

You can run Mysql locally on each box, if so don't change the my.cnf to take advantage of more RAM. Your tables and database will be less than 50MB as well as your queries being tiny so you don't need Mysql easting any more RAM than it needs. I'd think about going to a dedicated Mysql server in the future especially if you've already go one on the network for billing or whatever. 

Forget about shared filesystems. This is mail, not a research project. A central NFS server is your best bet and proven technology.

spam/virus filtering can be very CPU intensive. It's possible with your work load that adding a second CPU to your main POP/SMTP/MX machines might make them fast enough to run everything. I would probably run mail through a single filter machine to begin with and then roll it into the main cluster if things look good. Maybe push the whole thing up to 4-5 machines. Makes you much less vunerable in an outage as you're losing 20% of your capacity instead of 50%.

You can also do fun tricks like adding a second or third drive onto your machines and rebuilding postfix to run independently on each drive to get more I/O from each machine. You can also use it to sand box the filtering part of the equation on the same machine. You might need a second CPU to pull this off... I'd run the numbers and see if a second machine is cheaper or if this mackes sense. Also you'll need to up the concurrency setting in Postfix as well. 

DNS roundrobin should be okay, but I'd start looking about for some cheap hardware load balancing or doing the Linux load balancing stuff. It'll make your life that much easier.

kashani

----------

## RageX^NZ

Thanks for the advice guys, it's all being taken on board.

I will be using NFS with a central file server.

For the file server I will use 4x 36GB scsi drives in RAID5 to store the maildir's and will boot it from a Western Digital Raptor 36GB drive.

I am planning on using one of the Gigabyte server boards for this machine and will utilise the onboard intel gigabit lan.

For the POP3/SMTP machines in the cluster I will use:

Pentium IV 2.8Ghz HT

1024MB DDR Ram

36GB Western Digital Raptor

2x Intel Pro1000 NIC's

For the virus scanning machine I will have to check the performance/scalability of running NOD32 on OpenMOSIX, as if this is possible I could just use another one of the above boxes and if load increases it could split the threads accross the other machines.

Otherwise it will most likely be a Pentium IV 3.4 with 2GB ram.

Of course, I have to make sure the budget is big enough to afford this but it should be.

I am currently researching the linux HA and loadbalancing stuff and plan on playing with this sometime in the next week or so.

----------

## Kope

HA is really kind of overkill for mail -- DNS MX records are really a far easier solution to server outage issues for mail.

Loadbalancing the systems would not be a bad idea, but again, DNS round robin probably is all you really need. 

Unless you have some major systems, both of those options for mail servers is overkill.

----------

## kashani

On the load balancing front:

It's the diiference between getting a page and knowing 20-33% of your users can't pop or smtp mail and knowing it can wait till morning. Or that the user experience hasn't been affected in the 30 minutes it took to tigger the alert system, you to get the page, login and fix things, and then wait for DNS to update. For an ISP of 6k users I'd seriously look into it.

on the RAID front:

Watch out using RAID 5. It's write performance isn't so good, but may work well enough for your work load. I'm not saying don't do it, but keep an eye on performance stats.

Also you can go with a smaller raid card then I had mentioned, 32MB of RAM should be fine which will make the card cheaper.

the big caveat. I did the mail system at Netzero years ago so I tend to think bigger than you probably need. Keep that mind though I've tried to tone things down a bit.  :Smile: 

kashani

----------

## georwell

If you plan on scaling to more users(10s of thousands)  you might want to consider looking at cyrus-imap it is very scaleable and very useful for very large organizations. 

Mailscanner is also very nice for scanning large amounts of mail for spam and virii.  And it supports a ton of different options as well.

sounds like fun!

----------

## Kope

kashani,

for a 6k user base, I seriously doubt that load balancing is going to effect 33% of the user-base. I have no idea what the max load numbers are on his specfiic system, but I really really doubt it's so bad that a reasonable dns round-robin system would crack under the pressure.

HA and Load Balancing have their place, but a 6,000 user environment probably really can get by with out them just fine.

Of course, if he looks at his system statistics and sees an unacceptable number of refused or failed connections, then by all means include it. But I'd not start out expecting to need it in that environment.

----------

## kashani

Let's run the numbers.

For an average ISP the modem to user ratio is 1:10 or possibly 1:8 or 1:12 depending. So worst case scenario is 6k/8 or 750 users. Assuming we have 3 servers and one breaks we've affected 250 users. Let's further assume that their email clients caches or does not cache the IP of the server which some do and some don't. Either 250 people are affected or 750 people affected intermittently. Probably some combination of the two.

for 1:10 it's 200 people of 600.

250 users at $20 represents a monthly reoccurring revenue of $5000. This does not take into account DSL or T1 office users if any.

Looking at that I think buying a pair of cheap machines and loading them up with the Linux HA scripts is pretty cost effective. Not to mention that  there are probably other services that could be load balanced.

I'm not saying spend 40k on BigIP gear, but it's not hard or overly expensive to stop outages from effecting your users.

kashani

----------

## Kope

Actually, looking at that, and  considering labor rates, I don't see cost-effectiveness. What are the most common outages on a system like that? Drive failures. But those are a none-issue given that the disks aren't local to the mail servers anyway and are raided, hot-swappable, etc., etc., etc.

Ok, what else? configuration errors 'cause lots of outages .. but those won't happen when teh system is live, so we can ignore them.

What's left? component failure -- power supplies and motherboards. But mean time to failure on those is pretty high adn we're buying all new equipment for the system up front. 

What's a realistic annual probability for failure on this system that your HA solution will cover? 10% 15%? Certainly no higher than 20%.

So $5000 * .20 = $1000. 

Are you telling me that you can buy, install, and maintain a couple of HA linux boxes doing load-balancing for less than $1000 a year? 

In my market, a UNIX sysadmin with 10 years of experience runs about $75 an hour.  Let's assume a really amazing admin who can do all the maintenance and install on this system for only 10 hours a year for the life of the box. That leaves me a whole $250 a year for 5 years for the hardware. 

I don't care what amortization schedule you're using, you aren't going to buy the equipment and rack space for this project for that. I also don't buy that it's only going to take 10 hours a year in human resource costs, but I'll grant that for argument's sake.

Sorry, it's not cost effective.

----------

## kashani

Your arguements here is seriously flawed. A local drive failure will take down a server. That's where your MTA will write the email before it gets delievered. Only the maildirs are on the NFS. Additionally RAM can go bad, the OS can lockup, your MTA can crash, a fan can break causing the machien to overhead, someone can trip over the power cord, etc. 

Second it's an ISP, they have admins. Linux HA ain't hard. The only admin overhead would be to remove a machine during an extended outage or add a new one. I'd expect them to spend more time handling abuse mail than messing with HA.

Lastly it's the price of adminssion in my book. You're an ISP and it's your job to keep my mail up. If you can't I'll go down the street to the 900 other ISP's who can.

kashani

----------

## Kope

changing from a cost effectiveness argument to an argument over customer expectations is not exactly addressing the cost effectiveness point.

As to the rest of the items you mention, I'd still contend that your outage rate is going to be no where close to 20% annually including all of those.

Labor is labor, it's not free and is part of calculating cost effectiveness of a solution.

It may well be that your point about customer expectations is correct. However, it is not the case that you the solution you're advocating is cost effective. In other words  -- if it's true that a mail outage will result in customer loss reducing the revenue stream inexcess of the cost of overages of the solution you propose, then it may be the case that it's a necessary, though not cost effective solution. 

However, the reality of that claim is going to be highly dependent on the local market, which does in fact vary widely across various regions.

That is to say, customer expecations/potential customer loss is not something we can know with the data we have. But we can compute that the solution is not going to be cost effective for the user base presented even assuming a very high annaul failure rate.

----------

## tuxmin

What's all that noise about?

In RageX^NZ specs it says: - Fully redundant operation! . So how do you obtain this without an appropriate HA setup? And I don't mean RAID, redundant PSUs and such things. Your OS may crash with roughly the same probability as any other vital hardware component.

Of course it's a question of costs. But in serious business nothing is more expensive than downtime, isn't it!?

And what I find a big advantage of HA setups is, that you can do maintanence cycles and security updates without downtime!

Regards,

Alex!!!

----------

## kashani

Kope,

You have yet to make a decent point and back it up. Your single point of "there shouldn't be too many outages because the only things I can think of that might fail are the power supply and the mother board" is hardly a balanced and well reasoned argument.

You completely missed the fact that mail systems rely heavily on their local drives making your knowledge of mail systems suspect. Machines have other moving parts that can break, and or course switches never lockup, circuits never trip, UPS batteries don't go bad... I've got a list of 40 more things that I've personally seen. 

As to admin costs, the admins cost of adding HA to the overhead of running a mail system is a drop in the bucket which was my point. Not even counting the benefits of not having to have your staff scramble in an outage, increased tech support calls, which you conveniently left out of the cost equation. And yes it'll take less than 10 hours a year because I've used almost every load balancer on the market including the Linux HA stuff. 

I added customer expectations as yet another factor to which I'm also adding reputation, word of mouth, and other intangibles. Not changing my position, adding to it.

Additionally HA doesn't have to be for the mail system only, you keep avoiding that little point as well.

It may well be that the budget for this year is going to blown on 3 machines and a raid array and yes they'll probably do okay without it. On the other hand $1500 buys a lot of insurance and if they can afford it adds a lot of value.

kashani

----------

## Kope

Kashani,

No, my point is, toss in all of that and a 20% annual failure rate is still high, unless you're buying the cheapest components you can find, and even then it'd be rather amazing.

I'm not saying there shouldn't be too many outages becuase I can only think of a few components to fail. I'm saying that looking at the mean time to failure on the entire list of components that make up a system and you're still hard pressed to get to 20% probability of an annual failure.

And the admin costs aren't a drop in the bucket. HR costs are almost always the biggest component of any system. 

I grant that reputation and customer expectations may be an issue. But I also contend that the degree to which those are a consideration is a market issue that we don't have enough information to deal with. If this ISP is the only local ISP provider in a 200 mile radius (which in the central and western USA is not uncommon) then it may not be as necessary to cover those bases because you simply aren't going to lose that many customers. If you're one of a hundred ISP's in NYC, then it's a critical aspect of the system design because customer service is your only differentiator.

----------

## RageX^NZ

Wow, this has turned into somewhat of a debate.

The ISP is one of 4 Wireless ISP's locally, they also provide dialup services for a very limited number of customers.

So if their mailservers were unreliable it is possible that the customers could change to another local ISP.

Either that or they could change to one of the many ADSL plans that Telecom/TelstraClear offer.

So yes, the reputation of the company is important and in the past all of the email has been processed by a single postfix server leaving a single point of failure.  Even now this machine is not that loaded.  

As the ISP is growing very rapidly a new mail system is required to scale with the growth of the ISP and also to handle the increased loads of having virus scanning.

One thing with this is that unlike most ISP's where users are getting their email at 56kbits users on this ISP are generally getting it at 20 - 60mbits which will load the pop/imap server up more than the average ISP.

I have also researched MailScanner and it looks like this is the path to take, unfortunately it does not use TCP/UDP to transport from Postfix but I dont think this will be too much of a performance hit and from what I have seen it is faster than Amavis.

----------

## Kope

It's actually a good debate to have . . . far too many people do the engineering without looking at business realities, and the other way around -- building a system based on a business plan with no thought to the engineering considerations.

I don't think that kashani and I really disagree that much -- it seems we both agree that at some point HA and load balancing are optimal but we don't agree on how you should properly cost those items for determining when it makes the most business sense to go with them.

From what you've just stated, however, I'd tend to back off my position because if you are in a competitive market and have a history of service issues surrounding mail, then the money spent is probably worth it in terms of customer retention. 

It could be worth even more to you if you turn that around and spend some advertising dolars on top of the project budget to let people know you're doing this.

Of course, that's more, and different, analysis  :Wink: 

----------

## kashani

Completely agree with Kope on that, if you do HA, plug the hell out of it. Most of your competitors if they are the same size probably haven't done it yet.

On thing that hasn't been mentioned is webmail. If that's in the future it'll affect how you want to progress. imap can be pretty heavy weight depending on how you do things and it may affect how you distribute services. The apache side isnt' too bad, but you might have to dedicate two machine to imap/pop3 earlier than expected. 

kashani

----------

## georwell

 *Quote:*   

> The apache side isnt' too bad, but you might have to dedicate two machine to imap/pop3 earlier than expected.

 

Hence the cyrus-imap recommendation.   :Smile: 

----------

## groovin

i really enjoyed reading this thread, myself being a sys admin who majored in management/econ in college.

at one company i worked at, we outsourced our pop3 email and smtp to our ISP. in one year time, their pop3 went down around 4-5 times and smtp went down around the same amount. we actually lost some mail here and there... strange thing is months later we started getting mail from months past... i guess something got flushed. 

anyways, not being happy at all about this, i just walked over to another building whos sys admin i knew and asked her about their mail. she told me about another local ISP that in 2 years time had no downtime. needless to say, when our contract was up, we switched. 

now we werent a big company so im sure the ISP wasnt reeling in pain, but im sure there were plenty of other admins out there that were rubbed the wrong way as well. one thing about admins is, we talk. i recommended against that ISP to a few other people. it adds up.

----------

## Janne Pikkarainen

Wow, this really is an interesting thread to read.

As an admin for 70k user system I can give you one thought: should something be possible to fail, eventually it WILL fail. Power supplies and UPSes WILL break, hard disks WILL break, operating systems and applications WILL crash even with Linux, someone WILL make a configuration error, some update WILL break things, break-in attempts WILL occur... the best you can do is that you prepare to all of these in advance. In 6k user scale do not expect that everything "just works" and some admin can configure your systems with only "10 hours of work per year". That's simply not realistic. Test your backups, implement decent monitoring system (e.g. Nagios+snmp), try to break your own system (before taking it online, of course  :Smile: ),  benchmark & torture the hell out of it... and after taking your new system online, be prepared for the worst from the beginning!

If it's possible to implement some fail-over mechanism to whatever part of your system, do it if it's not completely out of your price range. It's better to experience the pain in advance by taking the bit harder but more robust route (e.g. Linux-HA/BigIP vs. round-robin DNS) than it is to just lazily throw in something without decent planning. Every piece of laziness will hit you back sooner or later. If you're designing something from ground up, it really, really, really is better to make it really scaleable and failsafe from day zero.

I would definitely take Cyrus+OpenLDAP+Postfix route. It is a little pain to setup at first, but relatively easy to scale afterwards and doesn't require too much babysitting on day-to-day operations.

----------

## groovin

"Power supplies and UPSes WILL break"

i found out the hard way that redundand PSU's arent fully redundant.

----------

## Kope

 *Janne Pikkarainen wrote:*   

>  should something be possible to fail, eventually it WILL fail. Power supplies and UPSes WILL break, hard disks WILL break, operating systems and applications WILL crash even with Linux, someone WILL make a configuration error, some update WILL break things, break-in attempts WILL occur... the best you can do is that you prepare to all of these in advance. 

 

Absolutely. But when planning on what is a good solution to those eventual breakages you have to balance costs against actual risks. Given enough time, every component in a system will fail. That is a given. But one of the reasons we look at MTTF numbers is to gain an insight into what level of breakage we can expect for a given budget period.

Sometimes the best business solution to redundancy is to not impliment full redundancy up-front. With all new equipement the chance of first year failures is often acceptalbe (not always and of course other business issues do come into play -- real time, critical systems that can't go down for example should be fully redundant up front). If you can accept some level of minimal risk -- the you can do some pretty cool accounting tricks with the project -- like using first year accelerated depreciation from the project to fund a secondary project the next year for redundancy. 

No redundacy solution should be automatic. Good business practice should dictate that the costs, risks and benefits are fully examined before resources are committed. And costs should always include opportunity loss costs when talking about small business with very limited dollars.

----------

## groovin

so we can all basically agree that each business is unique with different needs. know your business, know your needs, form your plan carefully.

[/url]

----------

## Robelix

I'm also looking for such a thing. In my case it's only about 3K users, but a lot of very active ones - a lot of them are checking for mail every minute (most using POP, only a few with IMAP).

Currently it's a single server with postfix, postfixadmin, mysql, amavis, clamav, spamassassin, courier. Sooner or later it's going to be overloaded due to increasing number of users - to be expected in about a year.

At the moment I'm thinking about 2 Variants:

a) 2+ identical boxes, DNS-round-robin and mail-storage on a NAS-System. (a dedicated 1GB-Net for the storage traffic is no problem)

b) Instead of the NAS a box with a good Raid controller - and run courier there. I know this is a single point of failure, but my experience says that courier is not the thing that makes problems. And it would take away a lot of IO from the MXes.

Which way would you prefer? Why?

----------

