Email.
As much as we depend on it, it was never designed to be a 100% reliable communications medium. And with the rise of spam over the past 10 years, it has become a blessing and a curse to System Admins the world over. We continually are balancing on that line of “How come I never got that email?” vs “Why do I get so much spam?”
So, when your client asks you to build and manage a mailing list system, you cringe as you KNOW that sometime, somewhere down the line that these thousands of messages being sent out WILL cause you a headache.
Companies and service providers tend to use a mix of internal and external rules, blacklists, whitelists and other automated policy to achieve a reduction in hearing either of those two questions above. (We love you users, but not hearing from you is the best praise)
These systems tend to clash with each other when thrown into the real world. We don’t want YOUR spam, but you had better accept messages from MY customers mailing list.
In terms of infrastructure, the best way to stay off the radar of other systems as a possible source of spam is to make sure you have good reverse DNS, proper and resolvable HELO responses and that you adhere to RFCs in the way your MTA behaves.
In terms of the actual email you send and the entire life-cycle of those messages, the best way to build and maintain a good reputation is not to send email to recipients that don’t exist. Yahoo is one of those providers that will ding you hard and shut down incoming mail from you if you send too many messages to unknown or disabled recipients. Two of our clients learned the hard way when their older systems for sending email did not include a method for handling bounces.
What they did have in their favor was a bottleneck function for sending email. Any email sent by the system was sent by this function and not directly using the built-in PHP mail() function. This gave us a place in the code to alter how the email is delivered. We could then make sure that bounces would come back to us in a way that we could easily detect the original recipient. Mailing list software that does automated bounce handling (like mailman) does this kind of thing all the time.
Quick overview of how email is sent.
You have what is known (in SMTP parlance) as the envelope and the payload. This is very analogous to sending a letter through the postal mail. You have a letter (the payload) which could be your letterhead, it may have a date and a To: and From: and Regarding:, etc… You pop that into an envelope (the envelope) and address it with who the letter is to be delivered to and what the return address is for if the letter cannot be delivered. Your message could be delivered to someone who is not actually the person listen in the headers of your actual letter and the return address could also be different from the From: portion of your letter as well. You could send a Blind Carbon Copy of the letter to a third party who is unnamed in the letter by sending another copy with their delivery address on the envelope.
So, we’ve established that the information on the outside of the envelope doesn’t necessarily have to have any relation to those named on the letter inside the envelope. Since it is your email program that reads the information in the payload and never sees the information on the “envelope”, this gives us tremendous flexibility in how we send email messages with customized envelopes that aid in our bounce detection.
What happens when an email bounces? A new message is created with special parameters.
Who gets the bounce message? The envelope recipient of a bounce is exactly what was defined as the envelope sender of the original message.
What gets bounced? That depends. There are no strictly adhered to standards as to what a bounce message looks like. What is in the payload could take a hundred different forms as mail server software vary as to what they place into the bounce message payload. Usually a Subject header with “Undeliverable” something or other.
What is the sender of a bounce message: BLANK. This prevents bounces from eternally being rebounced as there is no one to return it to (this is known as a double bounce)
Efficient delivery of email can send a message that was originally addressed to multiple people by only delivering a single digital copy of the message to a server (assuming the recipients are all hosted on the same end server.) However, with bounce messages, the address(es) the message did NOT make it to are NOT part of the only strictly adhered to portion of the Delivery cycle, the SMTP envelope. Parsing of the bounce payload is required with standard SMTP envelope usage. This is not exact and can fail as a bounce detection method.
What we need is to have some way of detecting the EXACT recipient that bounced. See the next article for one method of solving this problem.