Press "Enter" to skip to content

Month: November 2007

SSL Cert with Subject Alternate Name

One of our customers runs a system that has a shopping cart system and they like it when a customer upgrades their site to include full store functionality. This of course means an SSL certificate. And traditionally, this would mean an additional IP address that would need to added to the server to support the new certificate.

Entering wider spread usage is what is becoming known as the UCC (Unified Communications Certificate), which is just a fancy name for a regular X509 v3 certificate that utilizes the Subject Alternate Name extension. This extension allows the certificate creator to embed multiple alternate names that are cryptographically tied to the primary key that defines the certificate.

What this means in practical terms is that you could purchase a certificate whose primary name is and with the same certificate and IP address, support multiple variations of that domain name, such as or,, etc… This is different than a wildcard certificate that is tied to a specific domain name and can be used on any third level host name as desired: *

Since many entities that have a web presence tend to pick up the .net/.biz/.org variations on their .com domain name, this type of certificate means that those additional names can be secured with the same certificate/IP address. This means simpler configuration and not having to burn an IP address for every single name variation.

I’ve tested GoDaddy’s version of this certificate that gives you a 5 name cert for $60/year on plain jane Apache under 10.4 client and it works without issue. I have two different vhosts with different ServerName directives and the SSL config under both simply use the same SSL

Example. Assume I have a cert that I have based on as the primary name and has the following names embedded in as Subject Alternate Name extensions:


Also assume that I have two different facets of my web application. The public facing side that is served by the .com and the admin/extranet that is served under the .net variation. My Apache config would look like so:

Listen 443
NameVirtualHost *:443

<VirtualHost *:443>
DocumentRoot “/www/mydomain-com/”
<IfModule mod_ssl.c>
SSLEngine On
SSLCertificateFile “/etc/httpd/ssl/mydomain-com.crt”
SSLCertificateKeyFile “/etc/httpd/ssl/mydomain-com.key”
SSLCertificateChainFile “/etc/httpd/ssl/gd_intermediate_bundle.crt”

<VirtualHost *:443>
DocumentRoot “/www/mydomain-net/”
<IfModule mod_ssl.c>
SSLEngine On
SSLCertificateFile “/etc/httpd/ssl/mydomain-com.crt”
SSLCertificateKeyFile “/etc/httpd/ssl/mydomain-com.key”
SSLCertificateChainFile “/etc/httpd/ssl/gd_intermediate_bundle.crt”
notice the certificate files are the same for both vhosts and there is no distinguishing between IP addresses on the VirtualHost directive. Nice, clean, simple and it Just Works.

I’ve tested the certificate in the following browsers and none of them complain at all:

IE 6, FF2, Camino 1.5, Safari 2/3, Opera 8.5

X.509 v3 has been around at least since 2002 so it should be well supported.

Now, OS X Server should work exactly the same way as the underlying Apache system pieces are the same. The only issue comes into play with the way that ServerMgr handles storing certificates, their keys and passphrases and how Apache integrates all of these items.

Using the Server Admin app, you will not be able to use one of these certificates to secure two different Hosts that you enter as the host name in Apache is used by the “getsslpassphrase” binary to locate the certificate, private key, and password for the host in question to start SSL.

Since the primary name of the certificate is NOT the name of this secondary vhost, the loading process will fail. Any usage of this type of certificate will require you to manually create whatever non-primary VirtualHosts that would be setup to take advantage of these additional names secure by the “UCC” cert.

In order to get these certificates to load you will need to remove the passphrase from the key file so that Apache doesn’t call the getpassphrase and then fail.

the basic command for doing this is:
openssl rsa -in mydomain.key -out

For those who are use to working with OS X Server and doing something a bit out of the ordinary, this should not be surprising.

More SSL Certificate vendors are starting to offer this type of certificate, but I like the extra feature that the folks at DigiCert include with ALL their certificates: unlimited usage on any number of servers.

The DigiCert products are more expensive than the ones at GoDaddy, but theirs are single root certificates and they also go through additional layers of validation more than the simple “can you get an email at the domain” verification that GoDaddy utilizes.

A Gigabyte of Power

Was reading this article today on Google’s initiative to expand the economics of alternative energy:

Google expands into alternative energy

I love this paragraph here:

Toward that end, Google aims to produce one gigabyte of power from renewable energy at prices below the rates of electricity generated at coal-burning plants. One gigabyte power would be enough to supply the needs of a city the size of San Francisco.

I’m not exactly sure what one “gigabyte” of power is, but if Google is using them, I’m sure it’s a lot. 😉

I wonder if it’s comparable to the power you can generate by tapping into a clock tower that’s about to be struck by lightning.

Semi-related is the speculation that Google is building their own 10Gbit optical switches.

Guess they’ll combine their gigabytes of power with the gigabits of light. See: I knew lightning would be in there somewhere!


Basic Guidelines for Internet Connected Systems

Here is a list of the basics that every system administrator should implement:

  1. Set your Reverse DNS. Don’t leave it empty.
  2. Have geographically separated DNS servers
  3. MTAs should have properly formed HELO names
  4. rDNS should match the HELO on your MTA
  5. HELO should resolve to your IP address
  6. MX records must point to A records
  7. Filter Bogons at the first opportunity in your network architecture and at appropriate routing points.
  8. More to come

Managing Bulk Email Delivery – Part 2 – Bounce Handling with Postfix and PHP

If you are unfamiliar with the basics of how email messages are sent and what happens with bounces, please read the previous article.

This article explains ONE way of dealing with bounces. It has its pros and its cons and does not account for ALL instances of bounces, but it does deal with the basics of implementing bounce handling and can be used as the foundation of a more sophisticated system.

The primary logic in this system is the ability to control the SMTP Envelope FROM address. We want to construct this so that when a message is returned, it will have encoded in it information that will tell us exactly who the message was originally sent to. The drawback to this method is that if we had originally a single message that was being blasted to 1000 recipients, we now have to create 1000 messages each with it’s own customized/encoded FROM address. (there can be only one FROM)

Once we can control the Envelope FROM, we need a domain for the bounces to be returned to. Whereas the messages may have been originally FROM, we want bounces to be returned to a special host name. This way we can segregate bounce handling to a different system (if so desired.) With that in mind the FROM address will be constructed like so: Don’t forget to setup an MX record for this host name (which must point to an A record, CNAMEs or bare IPs are not allowed as MX data)

So, how do we encode the local part of the FROM address? It’s really up to you, but pick one way and stick with it. Our solution uses the following:

You could get fancy like so:

where 2e64665495eab1fa4c276f73a610e054 is an MD5 hash of the original email address.

Whatever method chosen, it’s necessary to track that particular encoding somehow as we will see it as the recipient on any possible bounces.
Here is the SQL table we used:

CREATE TABLE EmailAddressTracker (
EmailAddress varchar(255) NOT NULL,
EncodedFROM varchar(255) NOT NULL,
IgnoreBounces tinyint(1) unsigned NOT NULL default '0',
MsgCount int(10) unsigned NOT NULL default '0',
FirstEmail datetime NOT NULL,
LastEmail datetime NOT NULL,
FirstBounce datetime NOT NULL,
LastBounce datetime NOT NULL,
LastBounce2 datetime NOT NULL,
LastBounce3 datetime NOT NULL,
BounceCount smallint(5) unsigned NOT NULL default '0',
PRIMARY KEY (EmailAddress),

EncodedFROM holds the ENTIRE local part.

In the function that is ultimately responsible for sending the email out, we lookup/maintain entries in this table. This would be the place to apply policy and either let the message actually be sent or disable the email address somehow or ignore any policy if the IgnoreBounces flag were enabled for this particular email address.

If a message ultimately is rejected, we have to have some way of accessing this table. This is where Postfix and PHP come into play.We could simply have all messages for that domain fall into a mailbox which is accessed and read and parse the payload for undeliverable recipients, but we want direct access to the Envelope information. We could create a two tiered system that does do parsing as a fallback, but for now let’s keep it simple.

We are using Postfix on the server that is responsible for handling bounces. Two main additions to the postfix configuration are necessary.

  1. add a transport to the file:
    mybh unix – n n – 10 pipe
    user=mailadmin argv=/usr/local/bouncehandler/mybh.php $sender $recipient
    This defines for postfix a transport that is of the pipe variety. Postfix will pipe any bounces we tell it to, to the executable script in question with the given parameters.
  2. add a domain entry to the transport map so that messages that come in for our domain are sent to the newly defined transport: mybh:
    this can be a file called in /etc/postfix.
    don’t forget to the call postmap on the file so that it becomes a map hash file for fast access by postfix.

Once postfix is ready, the script defined can then do pretty much anything we want it to do.

Here is the relevant section of the PHP shell script that does the decoding and updating of the table.

#! /usr/bin/php -q
$sender = trim($argv[1]); // should be EMPTY
$recipient = trim($argv[2]);

$bounceProcd = FALSE;

$conn = ConnectToDB();
if (FALSE !== $conn)
list($encodedFrom, $bhDomain) = explode(‘@’, $recipient, 2);
$encodedFromSQL = mysql_real_escape_string($encodedFrom, $conn);
$query = “UPDATE EmailAddressTracker “.
“SET FirstBounce = IF(FirstBounce=0, NOW(), FirstBounce), BounceCount=BounceCount+1, “.
“LastBounce3=LastBounce2, LastBounce2=LastBounce, LastBounce=NOW() “.
“WHERE EncodedFROM = ‘$encodedFromSQL'”;
// We keep track of the datetime of the last three bounces to allow time based policy
// to be applied

$qResult = mysql_query($query, $conn);
$bounceProcd = mysql_affected_rows($conn) > 0;

// We have to read the data that postfix is sending to us in stdin
// we don’t have to necessarily do anything with the data, but we could store it into a table for later
// processing if we couldn’t determine the original recipient or wanted to double check our results

$dataLen = IgnoreMessageData();

// if we couldn’t connect to the db or there was not a record in the table that matched
// our clause for the specific encoded FROM, then exit back to postfix with a
// Temporary Failure. This will cause postfix to queue up the bounce
// message for later processing
$exitStatus = (TRUE == $bounceProcd) ? 0 : 75;
// 75 = EX_TEMPFAIL per sysexits


function IgnoreMessageData()
$msgLen = 0;
$fd = fopen(‘php://stdin’, ‘r’);
while (FALSE === feof($fd))
$dunsel = fread($fd, 1024);
$msgLen += strlen($dunsel);
return $msgLen;

Notice that this script does NOT apply policy. It merely is there for statistical tracking and that is all it should do.

Policy of whether to allow any future messages to be sent to the user are applied in the Sending function, since that is closer to where the emails are actually generated. The bounce handling system has it’s one job and can do it well without complications.

So there it is. A simple and effective way of catching bounces for your web application.
My favorite part of this solution is the extremely minimal configuration required inside postfix.

Happy bounce tracking.

Managing Bulk Email Delivery – Part 1 – Basics of SMTP and Bounces


As much as we depend on it, it was never designed to be a 100% reliable communications medium. And with the rise of spam over the past 10 years, it has become a blessing and a curse to System Admins the world over. We continually are balancing on that line of “How come I never got that email?” vs “Why do I get so much spam?”

So, when your client asks you to build and manage a mailing list system, you cringe as you KNOW that sometime, somewhere down the line that these thousands of messages being sent out WILL cause you a headache.

Companies and service providers tend to use a mix of internal and external rules, blacklists, whitelists and other automated policy to achieve a reduction in hearing either of those two questions above. (We love you users, but not hearing from you is the best praise)

These systems tend to clash with each other when thrown into the real world. We don’t want YOUR spam, but you had better accept messages from MY customers mailing list.

In terms of infrastructure, the best way to stay off the radar of other systems as a possible source of spam is to make sure you have good reverse DNS, proper and resolvable HELO responses and that you adhere to RFCs in the way your MTA behaves.

In terms of the actual email you send and the entire life-cycle of those messages, the best way to build and maintain a good reputation is not to send email to recipients that don’t exist. Yahoo is one of those providers that will ding you hard and shut down incoming mail from you if you send too many messages to unknown or disabled recipients. Two of our clients learned the hard way when their older systems for sending email did not include a method for handling bounces.

What they did have in their favor was a bottleneck function for sending email. Any email sent by the system was sent by this function and not directly using the built-in PHP mail() function. This gave us a place in the code to alter how the email is delivered. We could then make sure that bounces would come back to us in a way that we could easily detect the original recipient. Mailing list software that does automated bounce handling (like mailman) does this kind of thing all the time.

Quick overview of how email is sent.
You have what is known (in SMTP parlance) as the envelope and the payload. This is very analogous to sending a letter through the postal mail. You have a letter (the payload) which could be your letterhead, it may have a date and a To: and From: and Regarding:, etc… You pop that into an envelope (the envelope) and address it with who the letter is to be delivered to and what the return address is for if the letter cannot be delivered. Your message could be delivered to someone who is not actually the person listen in the headers of your actual letter and the return address could also be different from the From: portion of your letter as well. You could send a Blind Carbon Copy of the letter to a third party who is unnamed in the letter by sending another copy with their delivery address on the envelope.

So, we’ve established that the information on the outside of the envelope doesn’t necessarily have to have any relation to those named on the letter inside the envelope. Since it is your email program that reads the information in the payload and never sees the information on the “envelope”, this gives us tremendous flexibility in how we send email messages with customized envelopes that aid in our bounce detection.

What happens when an email bounces? A new message is created with special parameters.
Who gets the bounce message? The envelope recipient of a bounce is exactly what was defined as the envelope sender of the original message.
What gets bounced? That depends. There are no strictly adhered to standards as to what a bounce message looks like. What is in the payload could take a hundred different forms as mail server software vary as to what they place into the bounce message payload. Usually a Subject header with “Undeliverable” something or other.
What is the sender of a bounce message: BLANK. This prevents bounces from eternally being rebounced as there is no one to return it to (this is known as a double bounce)

Efficient delivery of email can send a message that was originally addressed to multiple people by only delivering a single digital copy of the message to a server (assuming the recipients are all hosted on the same end server.) However, with bounce messages, the address(es) the message did NOT make it to are NOT part of the only strictly adhered to portion of the Delivery cycle, the SMTP envelope. Parsing of the bounce payload is required with standard SMTP envelope usage. This is not exact and can fail as a bounce detection method.

What we need is to have some way of detecting the EXACT recipient that bounced. See the next article for one method of solving this problem.

Leopard Installer discs – no CDs, DVDs only

Question to my Apple Authorized dealer that I just ordered Leopard
Server from to install on an Xserve G4:

> can you find out for me how I can get CD versions of the Leopard
> Server installer?
> we have some xserves that do not have DVD capable optical drives.

His Reply:
> From what I understand it doesn’t exist, one of the requirement of
> Leopard is a DVD drive. I’m guessing you’ll have to
> boot from another computer that has a DVD drive in Firewire Target
> mode.

My retort:
> From:
> System Requirements
> Mac server or desktop computer with an Intel, PowerPC G5, or
> PowerPC G4 (867MHz or faster) processor; 1GB of physical RAM; 20GB
> of available disk space.

His Reply (looks like text pulled from somewhere)
> This is the first release of Mac OS X that’s not available in any
> form on CD, as all consumer-class computers that are capable of
> running Leopard also have at least a Combo Drive (DVD reading plus
> CD writing). Some Xserve models can run Mac OS X Server 10.5 but
> have only a CD-ROM drive; for such machines, you can perform a
> network installation using another computer running Leopard Server,
> or put the computer into Target Disk Mode and install Leopard
> Server from another computer that has a DVD reader.

Asking the Google, I find that text here:

So, installing Leopard Server on an Xserve G4 requires some sort of external piece of hardware. (Which of course, won’t be supplied by Apple (-; )

Target Disk Mode is not really an option as TGM on an Xserve G4 only exposes the device in Bay 1. Since we always build our Xserve G4s out with Mirrored arrays, we couldn’t start with a fresh pair of disks to install onto as a RAID pair. Perhaps a single drive RAID mirror might work and then add in second disk once booted up.

Ugh. I hate having to perform convoluted workarounds just to install an OS.

I guess all machines are equally supported for Leopard, just some are more equal than others.

UPDATE:  A really good suggestion was made to me on the OS X Server mailing list of making an image of the Leopard disc, then restoring onto a partition of a external firewire drive. This will boot the machine and run the installer just as if from an disc.This can even be used to put multiple install partitions of any number of install discs.

IDE and SATA Hardware RAID PCI cards

Every time we setup a server for a client, we always specify, at the minimum, mirrored drive pairs for boot and for data.

How those pairs are created has varied depending on the hardware involved.

PowerMac G5s (having only 2 native internal bays) get either a software RAID pair or a hardware RAID card. Or they get a drive bracket installed to increase internal capacity.

Xserve G4s are always built with 2 software RAID mirrors as the four bays in that system and not ability to use a hardware RAID card make it the most natural setup. Usually 2 x 60GB drives for the Boot and whatever size for the Data is appropriate for the application. A system where there will lost of media files will tend to get 2 x 500GB, otherwise, 2 x 250GB is now the sweet spot for longevity and reliability.

Xserve G5s I really don’t see any use in deploying if you aren’t using the Apple Hardware RAID card, but that also has some issues as well.

PowerMac G4s (yes we still use these) are a different story. With their 4 internal bays (5 if you want to squeeze one into where the Zip drive was spec’d to go, gives you a lot of options. We use PowerMac G4s almost exclusively at our office for things like development servers, log processing, internal project management, etc… Most of these have been with us for quite some time and have gone from one version of the OS to another as time progresses.

For some reason, I guess penuriousness is the most likely culprit, the boot drives on these systems were never more than whatever IDE drive we threw in at the point of setup. No mirroring or redundancy at all. Our data drives tended to be pairs of Ultra 2/3 SCSI disks software mirrored together.

Well I was determined that we should follow our own advice, so I’ve been on a crusade of upgrading/migrating disks/partitions/sharepoints on our internal servers so that all systems enjoy mirrored goodness.

I like the RAID cards over software RAID for 2 reasons: 1. I can easily partition the “disk” presented by the RAID card to the OS; 2. No hit on the CPU (which on a faster machine would be negligible) for performing the mirroring.
In the end, we’ve used the following devices/setups:

  • ACARD AEC-6880M in mirror mode with 2 x 400GB IDE – PowerMac G4/500DP
  • ACARD AEC-6890M in mirror mode with 2 x 320GB SATA – PowerMac G4/500DP

We still use 10K and 15K rpm SCSI disks in these systems to hold speed critical data like MySQL tables.

For some clients we’ve used:

  • ACARD AEC-6880M in mirror mode with 2 x 250GB IDE – PowerMac G4/500DP
  • ACARD AEC-6890M in mirror mode with 2 x 400GB SATA – PowerMac G5 DP
  • HighPoint Tech RocketRAID 2224 with X4 external enclosure connected by Infiniband (This was FAST)

We have seen some issues with the ACARD devices, but we’re leaning towards power supply problems more than anything directly wrong with the cards. Definitely flash the cards to the latest firmware before deploying is a good tip.

As for drives, I tend to spec Seagate drives almost exclusively these days.

A mishmash of ideas and technologies, but with the inexpensive options available today, there really isn’t any reason to deploy anything (even user desktops) without some redundancy. An extra $150 at the initial setup (drive, card, whatever) can save you and your employee a lot of time (meaning money if you bill for your time) down the road when a drive WILL fail.