One of our customers is running our ECMSquared Email server solution and recently decided they had outgrown the platform it was installed on. Mailbox access was slow, webmail was slow and it felt constantly overloaded.
When planning for an upgrade like this you have to allot for not only the hardware, but the expert’s time and this customer was on a tight budget, so they decided that spending money on our services making sure the transition was a higher priority than getting the biggest fanciest hardware rig. After all this is email, a service that may not seem critical, but it’s the first thing that people notice is not functioning correctly. So we put together a proposal for the migration.
Old system: Apple Xserve G5 – 2x 2.0Ghz G5 – 6GB RAM – 3 x 250GB SATA H/W RAID 5 running Tiger Server.
Upgrading the OS on the system from Tiger to Leopard Server should have yielded some performance gains, especially with the finer grained kernel locking that was introduced in Leopard, but with the main issue being slow mailbox access, we felt that the file system was going to continue to be the biggest bottleneck. HFS+ doesn’t handle 1000s of files in a single directory very efficiently and having the enumerate a directory like that on every delivery and every POP3/IMAP access was taking it’s toll. Also with Apple discontinuing PPC support along with the demise of the Xserve, the longevity of this hardware was assessed as low.
The decision was made to go to a Linux based system running ext3 as the file system. Obviously this opened up the hardware choices quite a bit.
A mail server is very much like a database server in that the biggest bottleneck is almost always disk throughput, not CPU or network. Based on the customers budget concerns we wanted to get them the biggest fastest drive array in the eventual system for the budget allowed. There aren’t a lot of choices when it comes to bigger/faster hard drives within a reasonable budget, so we ended up choosing a 3 x 146GB 10k rpm SCSI drives in a RAID 5 array.
New System: Dell PowerEdge 1750 – 2x 3.2Ghz Xeon – 8GB RAM – 3x 146GB NEW SCSI drives in HW RAID 5
Obviously this is relatively old hardware, but we were able to get everything procured along with some spare drives for ~$600
We installed Debian Lenny and custom-compiled version of Exim onto the system and ran several days of testing.
Then we migrated their system over late one night and everything went smoothly.
The change in that hardware/OS/file system stack produced the following graphic for the Load Average for the system:
You can see how dramatic the difference in how loaded the server was from before. The customer is very happy in the snappiness of the system now.
Even though the server hardware is a bit older, it’s applying the right resources in the right spot that makes thing run very smoothly.
We expect many more years of usage from this system.
Posted by Brian Blood as Hardware, Linux, Mail Server, Servers at 10:48 PM UTC
1 Comment »
There are times when managing ECMSquared installs where a customer will want to migrate to a new piece of hardware.
One the biggest chores is migrating all of the stored mailbox data. When we migrate a customer from EIMS (Eudora Internet Mail Server) or another product to ECM, the migration must go through a IMAP to IMAP process using imapsync. With a hardware only change, there is no format change, so it’s just a matter of copying data. Well, how do you efficiently copy tens of gigabytes of mail data with tens of thousands of folders and hundreds of thousands of files?
but not just any old rsync. Here is the command I ended up using after culling through a bunch of web sites looking for the most efficient way of making the connection:
rsync -av -e “ssh -c blowfish” –delete root@newserverIP:/var/mail/data/1* /var/mail/data/ &
rsync -av -e “ssh -c blowfish” –delete root@newserverIP:/var/mail/data/2* /var/mail/data/ &
rsync -av -e “ssh -c blowfish” –delete root@newserverIP:/var/mail/data/3* /var/mail/data/ &
Several items of note here:
- the -a switch is an “archive” option meaning that rsync will maintain all file permissions and dates as best as possbile.
- the -e switch and -c blowfish tell rsync that when using ssh to communicate with the other system to have ssh use the blowfish cipher. Blowfish is a very fast block level cipher that is well suited to bulk data transfers.
- the –delete switch tells rsync to delete anything in the target that is no longer in the source. This will be become important on the second pass (see below)
- The multiple forked rsync calls can all run simultaneously as each set of directories picked up by numerical wildcards (ECM Maildir data is stored primarily by site id) will have some different folder contents/data sets to work on. The system will settle down and balance itself out between network and disk activity amongst the various processes. This makes sure all parts of the system are utilized to their fullest.
- We make these calls based on having a passwordless ssh pre-setup between the servers.
- We make two passes with this set of commands. Once to pre-load the data to the new server, and then once again at the cutover point. The second time there should be much less data to actually move over the network.
Posted by Brian Blood as Mail Server, Servers at 9:08 PM UTC
No Comments »
Here is a list of the basics that every system administrator should implement:
- Set your Reverse DNS. Don’t leave it empty.
- Have geographically separated DNS servers
- MTAs should have properly formed HELO names
- rDNS should match the HELO on your MTA
- HELO should resolve to your IP address
- MX records must point to A records
- Filter Bogons at the first opportunity in your network architecture and at appropriate routing points.
- More to come
Posted by Brian Blood as General, Mail Server, Routers and Firewalls, Servers, Soap Box at 3:53 PM UTC
No Comments »
If you are unfamiliar with the basics of how email messages are sent and what happens with bounces, please read the previous article.
This article explains ONE way of dealing with bounces. It has its pros and its cons and does not account for ALL instances of bounces, but it does deal with the basics of implementing bounce handling and can be used as the foundation of a more sophisticated system.
The primary logic in this system is the ability to control the SMTP Envelope FROM address. We want to construct this so that when a message is returned, it will have encoded in it information that will tell us exactly who the message was originally sent to. The drawback to this method is that if we had originally a single message that was being blasted to 1000 recipients, we now have to create 1000 messages each with it’s own customized/encoded FROM address. (there can be only one FROM)
Once we can control the Envelope FROM, we need a domain for the bounces to be returned to. Whereas the messages may have been originally FROM firstname.lastname@example.org, we want bounces to be returned to a special host name. This way we can segregate bounce handling to a different system (if so desired.) With that in mind the FROM address will be constructed like so: email@example.com. Don’t forget to setup an MX record for this host name (which must point to an A record, CNAMEs or bare IPs are not allowed as MX data)
So, how do we encode the local part of the FROM address? It’s really up to you, but pick one way and stick with it. Our solution uses the following:
You could get fancy like so:
where 2e64665495eab1fa4c276f73a610e054 is an MD5 hash of the original email address.
Whatever method chosen, it’s necessary to track that particular encoding somehow as we will see it as the recipient on any possible bounces.
Here is the SQL table we used:
CREATE TABLE EmailAddressTracker (
EmailAddress varchar(255) NOT NULL,
EncodedFROM varchar(255) NOT NULL,
IgnoreBounces tinyint(1) unsigned NOT NULL default '0',
MsgCount int(10) unsigned NOT NULL default '0',
FirstEmail datetime NOT NULL,
LastEmail datetime NOT NULL,
FirstBounce datetime NOT NULL,
LastBounce datetime NOT NULL,
LastBounce2 datetime NOT NULL,
LastBounce3 datetime NOT NULL,
BounceCount smallint(5) unsigned NOT NULL default '0',
PRIMARY KEY (EmailAddress),
UNIQUE KEY EncodedFROM (EncodedFROM)
EncodedFROM holds the ENTIRE local part.
In the function that is ultimately responsible for sending the email out, we lookup/maintain entries in this table. This would be the place to apply policy and either let the message actually be sent or disable the email address somehow or ignore any policy if the IgnoreBounces flag were enabled for this particular email address.
If a message ultimately is rejected, we have to have some way of accessing this table. This is where Postfix and PHP come into play.We could simply have all messages for that domain fall into a mailbox which is accessed and read and parse the payload for undeliverable recipients, but we want direct access to the Envelope information. We could create a two tiered system that does do parsing as a fallback, but for now let’s keep it simple.
We are using Postfix on the server that is responsible for handling bounces. Two main additions to the postfix configuration are necessary.
- add a transport to the master.cf file:
mybh unix – n n – 10 pipe
user=mailadmin argv=/usr/local/bouncehandler/mybh.php $sender $recipient
This defines for postfix a transport that is of the pipe variety. Postfix will pipe any bounces we tell it to, to the executable script in question with the given parameters.
- add a domain entry to the transport map so that messages that come in for our bounces.mycompany.com domain are sent to the newly defined transport:
this can be a file called transport.map in /etc/postfix.
don’t forget to the call postmap on the file so that it becomes a map hash file for fast access by postfix.
Once postfix is ready, the script defined can then do pretty much anything we want it to do.
Here is the relevant section of the PHP shell script that does the decoding and updating of the table.
#! /usr/bin/php -q
$sender = trim($argv); // should be EMPTY
$recipient = trim($argv);
$bounceProcd = FALSE;
$conn = ConnectToDB();
if (FALSE !== $conn)
list($encodedFrom, $bhDomain) = explode(‘@’, $recipient, 2);
$encodedFromSQL = mysql_real_escape_string($encodedFrom, $conn);
$query = “UPDATE EmailAddressTracker “.
“SET FirstBounce = IF(FirstBounce=0, NOW(), FirstBounce), BounceCount=BounceCount+1, “.
“LastBounce3=LastBounce2, LastBounce2=LastBounce, LastBounce=NOW() “.
“WHERE EncodedFROM = ‘$encodedFromSQL’”;
// We keep track of the datetime of the last three bounces to allow time based policy
// to be applied
$qResult = mysql_query($query, $conn);
$bounceProcd = mysql_affected_rows($conn) > 0;
// We have to read the data that postfix is sending to us in stdin
// we don’t have to necessarily do anything with the data, but we could store it into a table for later
// processing if we couldn’t determine the original recipient or wanted to double check our results
$dataLen = IgnoreMessageData();
// if we couldn’t connect to the db or there was not a record in the table that matched
// our clause for the specific encoded FROM, then exit back to postfix with a
// Temporary Failure. This will cause postfix to queue up the bounce
// message for later processing
$exitStatus = (TRUE == $bounceProcd) ? 0 : 75;
// 75 = EX_TEMPFAIL per sysexits
$msgLen = 0;
$fd = fopen(‘php://stdin’, ‘r’);
while (FALSE === feof($fd))
$dunsel = fread($fd, 1024);
$msgLen += strlen($dunsel);
Notice that this script does NOT apply policy. It merely is there for statistical tracking and that is all it should do.
Policy of whether to allow any future messages to be sent to the user are applied in the Sending function, since that is closer to where the emails are actually generated. The bounce handling system has it’s one job and can do it well without complications.
So there it is. A simple and effective way of catching bounces for your web application.
My favorite part of this solution is the extremely minimal configuration required inside postfix.
Happy bounce tracking.
Posted by Brian Blood as Mail Server, Servers, Web App Development at 9:07 PM UTC
1 Comment »
As much as we depend on it, it was never designed to be a 100% reliable communications medium. And with the rise of spam over the past 10 years, it has become a blessing and a curse to System Admins the world over. We continually are balancing on that line of “How come I never got that email?” vs “Why do I get so much spam?”
So, when your client asks you to build and manage a mailing list system, you cringe as you KNOW that sometime, somewhere down the line that these thousands of messages being sent out WILL cause you a headache.
Companies and service providers tend to use a mix of internal and external rules, blacklists, whitelists and other automated policy to achieve a reduction in hearing either of those two questions above. (We love you users, but not hearing from you is the best praise)
These systems tend to clash with each other when thrown into the real world. We don’t want YOUR spam, but you had better accept messages from MY customers mailing list.
In terms of infrastructure, the best way to stay off the radar of other systems as a possible source of spam is to make sure you have good reverse DNS, proper and resolvable HELO responses and that you adhere to RFCs in the way your MTA behaves.
In terms of the actual email you send and the entire life-cycle of those messages, the best way to build and maintain a good reputation is not to send email to recipients that don’t exist. Yahoo is one of those providers that will ding you hard and shut down incoming mail from you if you send too many messages to unknown or disabled recipients. Two of our clients learned the hard way when their older systems for sending email did not include a method for handling bounces.
What they did have in their favor was a bottleneck function for sending email. Any email sent by the system was sent by this function and not directly using the built-in PHP mail() function. This gave us a place in the code to alter how the email is delivered. We could then make sure that bounces would come back to us in a way that we could easily detect the original recipient. Mailing list software that does automated bounce handling (like mailman) does this kind of thing all the time.
Quick overview of how email is sent.
You have what is known (in SMTP parlance) as the envelope and the payload. This is very analogous to sending a letter through the postal mail. You have a letter (the payload) which could be your letterhead, it may have a date and a To: and From: and Regarding:, etc… You pop that into an envelope (the envelope) and address it with who the letter is to be delivered to and what the return address is for if the letter cannot be delivered. Your message could be delivered to someone who is not actually the person listen in the headers of your actual letter and the return address could also be different from the From: portion of your letter as well. You could send a Blind Carbon Copy of the letter to a third party who is unnamed in the letter by sending another copy with their delivery address on the envelope.
So, we’ve established that the information on the outside of the envelope doesn’t necessarily have to have any relation to those named on the letter inside the envelope. Since it is your email program that reads the information in the payload and never sees the information on the “envelope”, this gives us tremendous flexibility in how we send email messages with customized envelopes that aid in our bounce detection.
What happens when an email bounces? A new message is created with special parameters.
Who gets the bounce message? The envelope recipient of a bounce is exactly what was defined as the envelope sender of the original message.
What gets bounced? That depends. There are no strictly adhered to standards as to what a bounce message looks like. What is in the payload could take a hundred different forms as mail server software vary as to what they place into the bounce message payload. Usually a Subject header with “Undeliverable” something or other.
What is the sender of a bounce message: BLANK. This prevents bounces from eternally being rebounced as there is no one to return it to (this is known as a double bounce)
Efficient delivery of email can send a message that was originally addressed to multiple people by only delivering a single digital copy of the message to a server (assuming the recipients are all hosted on the same end server.) However, with bounce messages, the address(es) the message did NOT make it to are NOT part of the only strictly adhered to portion of the Delivery cycle, the SMTP envelope. Parsing of the bounce payload is required with standard SMTP envelope usage. This is not exact and can fail as a bounce detection method.
What we need is to have some way of detecting the EXACT recipient that bounced. See the next article for one method of solving this problem.
Posted by Brian Blood as Mail Server, Servers, Web App Development at 9:06 PM UTC
No Comments »
Today I added a new datum for the Users table for our mail server: Last Message Received
What prompted me to add this was I was trying to prune down the over 150 accounts we have in the macserve.net domain and I had no idea which email addresses were actually in use or when they last received an email. I needed a quick reference that would not necessitate a trip to the Recent Mail table.
So I added two new columns to the Users table: last msg recvd and last msg sent. For now I’m only dealing with the former as I haven’t implemented tracking of sent mail yet.
The big thing was figuring out a quick and easy way of getting the most recent datetime for a user from the Recent Mail table and updating that in the Users table.
I started writing a separate script for this, but realized it would be just fine to drop this process into our daily database maintenance script. I had started to write some PHP code that would loop through the Recent Mail table for entries at most a week old and figure out the most recent message and then make a call to update that record in the Users table.
In the course of creating a temporary table to go back to the archive of recent mail we keep, I realized I could simply use an SQL temporary table to hold that data (duh) and then simply run a joined update from that temp table into the Users table. It turned out to be a simple 3 statement SQL process like so:
CREATE TEMPORARY TABLE most_recent_email
SELECT recipient_id,MAX(recent_msgs.msg_when) as last_msg_when
WHERE (msg_when >= ‘$oneWeekAgo’) AND (recent_msgs.recipient_id > 0)
GROUP BY recent_msgs.recipient_id;
SET last_msg_rcvd = last_msg_when
WHERE site_users.user_id = most_recent_email.recipient_id;
DROP TABLE most_recent_email;
Nice and simple and I let the database do all the work for me. I like it.
Posted by Brian Blood as Database, Mail Server at 11:47 PM UTC
No Comments »
An recent email inquiry I received:
> I saw you had posted a reply to my inquiry about large installs running ECM2.
This mail server is my baby, so if I gush a bit, please forgive me.
> At this point, we’ve totally outgrown EIMS (as you can understand),
> and ECM2 is defintely the front-runner as far as replacements go. I
> have looked a lot at AtMail, which is basically a commercial ECM2,
SquirrelMail is what we use now for customer webmail, but we are seriously considering using something different and @mail’s webmail system might do the trick. We have some pretty sophisticated customers and a better webmail system is definitely needed.
> but decided that I really think I’d rather have a firm foundation and
> be able to modify it myself, instead of relying yet again on a
> commercial developer’s whims.
I hear you. EIMS lasted us a very long time, but we had to bite the bullet and make a change. I built a whole bunch of applescripts and php import scripts to migrate accounts over and that data then fed into a program that did syncing of mail over from EIMS mailboxes to the new ECM based system that mostly kept “read/unread” flags on the email. THAT was a big deal.
It’s been about a year now running on our server and it’s totally kick butt. We see almost no spam now, and the manageability is orders of magnitude more than it was with EIMS.
Most emails we get to make changes for email accounts we simply reply back to the admin of that account with the admin pwd and a URL to the mailadmin site. It has saved us mountains of support time.
The great thing about exim is that I can actually PROGRAM each phase of the SMTP conversation and the delivery phase to however complicated or personalized for each of our domains/users it needs to be.
Our exim config is definitely one of the more complex I’ve seen on the net.
And exim runs it without any issue.
> That said, it sounds like your ECM2 installation is handling your
> traffic well. May I ask what architecture you’re running your server
> on, and what types of loads you see?
OK, you asked…..
We run it on a Dual 1.0 Ghz Xserve G4 (10.4.8) with 2GB of ram, 2x60GB for boot and 2 x300 for data. Here is the Daily Load Average graph for that box:
Those spikes are when I’m running a mysqldump.
Here is the monthly graph:
1835 users defined
1069 of those defined as a mailbox.
Now, I took the package that George built and highly modified it. ECMs database schema and exim config is actually based on vexim, so there were some things I found on that site that I did pull into the system.
The primary difference on our system is that I abstracted domains out of the schema. I’ve always like the way EIMS implemented domain aliases. It so easily and transparently overlaid onto a “site”. So that’s what I did:
so, the primary unit is the site, then all the users and then you can have any number of domains on that site. Those graphics represent a state of the system over a year ago, there are quite a bit more fields in the sites table now, but this gives you the basic outline. the user can use any domain and can even use the % hack in their login id as well for all three services: SMTP, POP3 and IMAP. There is still a PRIMARY domain that you define in the site preferences.
> I have heard conjecture that
> Exim isn’t all that great under load,
We haven’t seen that and we run it on “old” hardware and it runs like a champ for us. I’ve implemented it for two other customers (one on Mac, one on Debian Linux) and we have two more lined up who want it.
The one thing I didn’t want exim to do was to handle outbound mail. The reason is that every message for delivery would have incurred another database lookup which would have caused unnecessary load and slow performance. So, we use an instance of Postfix with a very light config to handle all that delivery. This could be setup on the same box by having Postfix listen on a different port, but we already had in place an existing system on a different server, so we just used that.
> but I’ve also heard version 4
> took care of a lot of that. I don’t think we’re huge load here, but
> we do do at least a couple thousand messages per hour.
ok, I looked at our graphs of connections:
and taking the hour of 14:00 Dallas time, which seemed the busiest…..
received mail: 587
received mail: 1172
Now, I was able to pull those numbers very quickly because we LOG all blocks and all accepted messages into mysql:
SELECT count(*) FROM `recent_mail` WHERE recvd BETWEEN ’2007-03-19 19:00:00′ AND ’2007-03-19 19:59:59′
SELECT count(*) FROM `block_log` WHERE `when_blocked` BETWEEN ’2007-03-19 19:00:00′ AND ’2007-03-19 19:59:59′
(We store ALL times in UTC and then adjust for the user when displaying through the web admin pages)
The admin of an site or even an individual account holder can now see with their own eyes what emails were blocked and why.
The block log:
The Recent mail log:
I also built into it:
- greylisting exceptions by:
- site (stable through whatever domains are on that site)
- source IP or IP range
- sender domain or sender email
- recipient domain or recipient email
- auto blacklist of ips:
When an incoming server tries to helo with one of my ips or one of my names, that IP address is automatically added to the blacklisted hosts table with an expiration for a month.
- spam assassin:
- sql based bayesing scoring
- sql based auto white listing (the more mail you get from a sender, the lower their email is rated for spam)
- global, site and user level based prefs
- connection logging/profiling
I keep track all every single IP address that connects to our server.
As they progress through a SMTP connection, I update certain values on that record:
ip cnxn_count, cnxn_first, cnxn_last, reverse_ok_count,
helo_ok_count, quit_count, bad_from_count, bad_rcpt_count,
ok_rcpt_count, dnsbl_block_count, last_dnsbl_time, last_dnsbl
- whitelisting (globally or per site):
- by sender
- by recipient
- by ip or ip range
I also implemented catchalls similar to the way that EIMS has them:
Through the use of a preferences table, I can also selectively turn on/off certain features of the mail server in real-time without touching the config:
allow_cnxns allow_trusted allow_authd allow_other greylist_on
Also I liked the options for users that EIMS had: mailbox, forward and both, but ours has expanded features:
we also now have TLS based SMTP, POP3 and IMAP using a self-signed certificate.
> Thanks for any input you may be able to provide!
Here is a screen shot of the admin interface. ( I completely rebuilt the ECM web admin interface to handle more features and deal with the changes in architecture)
The Blocks column is the number of blocks in the past 24 hours and the last hour.
The Recent column is the number of accepted msgs in the past 24 hours and the last hour.I also wrote a bunch of support scripts that tail through some of the logs and update the login times for those users in the database.
Things I haven’t implemented but are mostly already built, just need testing:
- logging of email sent by authenticated users.
- automatically feeding email sent to spam traps into Spam Assassin for bayesian scoring
- Archiving of email per site or per user for corporate entities required to do so.
So, there it is.
Hope that answers all your questions and more!
Posted by Brian Blood as Database, Mail Server at 12:03 AM UTC
No Comments »