With all the recent attention on the DNS exploit and the delay in response by Apple for providing a response, I thought I would undertake a deeper review some of our dns systems. We were already protected from the exploit as we do not provide recursive service with any of our unpatched dns servers.
Turns out that not only has Apple not patched the BIND install as of July 29, they haven’t really kept up with the installed config files, specifically the root servers hints file (/var/named/named.ca)
DNS is a distributed system, but a DNS resolver has to be given a place to start in it’s search for your name resolution request. That’s what the 13* root servers are about. They are the top of the tree so to speak. They are strategically placed around the world and load balanced and are THE critical part of the Internet infrastructure.
The hints file is populated with names/IP addresses of the 13 root servers preset like so:
A.ROOT-SERVERS.NET. 3600000 A 198.41.0.4
this goes on down from A to M
Occasionally, an IP address of one of the root server will change. There have been two updates in the past 4 years: B and L
Old
B.ROOT-SERVERS.NET. 3600000 A 128.9.0.107
L.ROOT-SERVERS.NET. 3600000 A 198.32.64.12
New:
; updated Jan 29, 2004
B.ROOT-SERVERS.NET. 3600000 A 192.228.79.201
; updated Nov, 1, 2007
L.ROOT-SERVERS.NET. 3600000 A 199.7.83.4
In Tiger Server, both B and L are out of date even with all updates and security patches applied
Leopard Server (up to 10.5.4) still shows the L root server out of date.
Update your /var/named/named.ca with the new entries preferably with this file:
ftp://rs.internic.net/domain/named.root
then stop/start the named process
The hints file from internic will also include IPv6 information.
The guys at Renesys wrote a good article about the potential dangers of not querying the correct root servers.
Links:
http://www.root-servers.org/
* - The count is 13 and 13 being the count because that’s the maximum amount of dns records that can be crammed into a single IP packet response.
Posted by Brian Blood as OS X Server, Servers at 12:47 PM CDT
No Comments »
A customer of ours recently asked us to help them troubleshoot some performance problems they have been having with their ASP/MySQL based solution. They are running the latest version of MySQL 5 under Windows 2003 Server 64-bit edition. Their IIS/ASP based application is using an ODBC connection to connect to MySQL.
They have recently added a partner that is sending them sales leads through an API call. There are occasional peaks in this traffic where the incoming connections into IIS far exceeds what they have been able to handle in the concurrent open connections into MySQL. Tracking this down through the application stack we looked at several low-level file/network system items:
- Ephemeral port exhaustion
- Stale socket disposal
- Max Open Files
#1 and #2 has caused some issues for another customer in the past, but this time it turned out to be #3. Even though they had set an appropriate value for max-connections in their my.ini file:
max-connections = 4000
table-open-cache=512
MySQL was artificially reducing these settings at runtime to much lower than was needed to support this flash traffic. Upon startup, MySQL would emit this into the Application Event Log:
Changed limits: max_open_files: 2048 max_connections: 1910 table_cache: 64
According to this bug report on the MySQL site, the approximate formula is used to determine the maximum open files for MySQL:
table_cache*2+max_connections
This condition has been reported as far back as 2006. A fix that switched MySQL from using the C Runtime Library in the Windows binary for opening files to using the native Win32 API calls was completed in mid June 2008. However, this will only be rolled into the MySQL 6 line of development.
We had looked at seeing about increasing this limit to something higher than 2048, but according to this site, that value is hardcoded. This particular customer has at least one thing going for them, their exclusive use of the InnoDB engine. This engine uses native calls so the table cache value does not need to be anything very high and they can maximize the number of open descriptors for use in incoming connections.
Realistic Options:
- Artificially limit the maximum number of incoming connections to IIS so as to limit the passthrough the MySQL. This leaves serious amounts of processing power of this particular server untapped.
- Move to MySQL running on a non-Windows OS
- Move their backend to MS SQL Server.
Posted by Brian Blood as Database, MySQL, Servers, Web App Development at 3:42 PM CDT
No Comments »
We’ve been managing a mail server for a customer of ours for about 5 years now. We started them out, as their needs were very modest, on a Windows 2000 Server running ArgoSoft Mail Server and then early 2005, we migrated them to a newer system running Windows 2003 Server Web Edition running MDaemon from AltN.
We chose Web Edition as it was the least expensive of the Server product line and since we were adding third-party software, this system didn’t require any specific built-in Microsoft server components that are available with Standard, Enterprise or Small Business Server.
This system ran very nicely for quite some time as usage continued to grow.
Late 2006, the system had over 1500 active email addresses and over 500 active POP3 users. Over one particular weekend, many users started to see intermittent connection failures. After investigating the issue we came up with two particular issues that needed tweaking.
First, a bit background material. When an application based on the TCP/IP protocol wants to communicate with another system two pieces of information are required: the destination IP address and the port. Systems that provide a service listen on the Well Known Ports since a convention is needed for the particular protocol. For example, Simple Mail Transfer Protocol (SMTP), the language that mail servers use to transfer mail, uses the well known port 25.
A system that is initiating a connection to another that is listening on a well known port (like an email program on a client machine to a mail server) must also use a particular port at the source so that packets that are sent from the receiving end can properly reach back to the initiating program. These ports are usually picked at random and are only used for a very short duration and as such called the Ephemeral Ports. On a mail server, such ports will come into use when the mail server software wants to process certain tests for doing such things as spam processing, dns blacklist checking, anti-virus updates, etc… so a system can quickly eat a large number of these ports as necessary for mail processing. Also in delivery, ephemeral ports will be required for the outgoing SMTP connection.
Second, sockets that are closed (communication between the two ends is closed gracefully) do not immediately go back into the pool of available. These are in what is called the TIME_WAIT state This is so that reopening the connection to the client and server costs less than establishing a new connection. IBM: Windows Tuning
In the end, we decreased the TcpTimedWaitDelay value, so sockets could be recycled faster and we increased the MaxUserPort value, so we could have a larger pool of available sockets.
What was curious was that these values are essentially the same when the base OS is a Windows client version such as Windows XP. The base OS on this server was the “Web Edition”, so having to find and then tune the values for a server OS was a bit strange.
As they say, if you can’t measure it, you can’t manage it, so we setup two data sources and graphs for this system to monitor the active and stale socket connections that feed into cacti:
Since making these tweaks, the server has been humming along smoothly.
Posted by Brian Blood as Servers at 3:30 PM CDT
No Comments »