This is a continuation of the saga of helping a customer of ours with their MySQL on Windows issues.

The basic premise is that MySQL 5 running under Windows has problems with large numbers of connections/open files.

We initially presented our client with 2 choices for solving their problem:

1. Setup MySQL on a different server running Linux
2. Move their database backend from MySQL to Microsoft SQL Server

Option 1 would require a non-trivial capital outlay and time to setup and migrate the data over

Option 2 held more interest for them as a longer term solution as they were interested in some of the powerful reporting and analysis tools. However, the server that all of this was running on turned out to present a problem in the following way:

The system is a Dell PowerEdge 2950 with dual dual-core Xeons (with HyperThreading enabled, resulting in 8 logical cpus), 8GB RAM and running Windows 2003 Server x64. For a general ballpark estimate of pricing for MS SQL Server, I went to the Dell website and configured a similar machine and went to select a SQL Server as an add-on. Due to the way that Microsoft prices SQL Server for a web environment, you must license it for each processor you have installed in your server. A dual-processor licensed SQL Server 2005 running under Windows x64 turned out to be in excess of $14,000 on that page.

Needless to say, the customer had a bit of sticker shock. Not only would they have to plop down FOURTEEN grand, they would still need man-hours to work out kinks in transitioning the backend of the app from MySQL. Getting a fully loaded separate server for half that cost running Linux/MySQL was looking more attractive.

I was chatting with a friend about the customers dilemma and he had a brilliant suggestion: Run a Linux instance under VMWare on the existing hardware.

I broached the idea with the client and they were game to give it a shot. The server was at this point extremely underutilized and was only limited by the specific software implementations. They were already running MySQL here anyway, so putting a different wrapper around it in order to escape those limits was worth investigating.

The server needed some more disk space so they ordered 6 more of the 146GB 2.5″ SAS drive modules like the 2 that were already installed in a mirrored pair. These are great little devices for getting a decent number of spindles into a database server.

While they did that I installed the latest Release Candidate of VMWare Server 2.0 and went about setting up a Debian based Linux install with the latest MySQL 5.0 binary available. During testing we had some interesting processor affinity issues for the VM environment and after a very good experience and excellent responses in the VMWare community forums, I tweaked the config to have that VirtualMachine attach itself to processors 5-8 which correspond to all of the logical cpus on the second socket. This left the first socket’s worth of cpus for OS and IIS/ASP usage. Doing this would help avoid the cache issues that occur with multi-cpu systems and multi-threaded processes.

I ended up configuring the VM for 3.5GB RAM, 2 processors and tweaking the MySQL config to make good use of those resources for their INNOdb based tables.

The system has been running very reliably for over 2 months and is making very good use of the resources on that system. Load Average in the VM stays well under 1.0 and the cpus in the Windows system are easily handling all the different loads.

Overall I spent probably 8-10 hours installing and tweaking the config and helping them migrate their data. This was much less expensive to the other options and they got to make use of their existing hardware and software configurations.

A very positive experience and outcome.

Posted by Brian Blood as Database, Hardware, Linux, MySQL, Servers at 2:41 PM UTC

No Comments »

In our continuing adventures of putting the highly versatile Apple MacMini to work in all sorts of applications:

A customer of ours has a specialized application that is extremely low bandwidth, but needs to be able to be accessed through a VPN connection to a protected network resource. All the usual suspects for providing this kind of service (Netscreen/SSG, Netopia) all have artificial limits on the number of concurrent VPN connections. In order to service the number of concurrent users (anywhere from 1 to 200), the customer would be looking at least $5,000 for a capable device. Again, this system is very low bandwidth, so these kinds of devices would be serious overkill.

In comes the MacMini running Tiger Server with the built-in VPN service.

However… the MacMini has always had a single weak spot: the hard drive. Even with a very low disk based usage that a dedicated VPN Server would entail that vulnerability to a disk failure has always bothered me and I wanted this VPN server to be on par in terms of reliability as much as possible compared to a hardware device. Even the MacMinis that we build out for customers in a load balanced environment, we always replace with a extended use rated drive, as the extra $100 is a nominal expenditure compared to the time that would be involved in dealing with a failed system.

In comes the Solid State Disk. A 32GB SSD with a IDE interface runs just under $100. For a system running primarily as a VPN Server, this is PLENTY of space as the only real ongoing writes will be log files. I chose a MLC (Multi-Level Cell) device over a SLC (Single Level Cell) based device as the differences in speed and longevity between the two variations of SSD just were not a factor in this application. A MLC SSD already has a useful lifetime/MTBF an order of magnitude greater than a standard Winchester based disk with it’s moving parts.

In the end I used a 1.33Ghz G4 (PowerPC, not Intel) MacMini with 1GB of RAM, as that CPU should be able to easily handle the tasks of handling a couple hundred concurrent VPN connections using 1-2Kbits/sec each. Again, VERY low bandwidth application.

I took the MacMini with it’s original 40GB Seagate drive and installed Tiger Server so I could get it updated and setup while I waited for the SSD to arrive from the online vendor. When I was ready to install the SSD, I used Disk Utility to save the installed system to a disk image. Installing the SSD was straightforward and not any different that replacing the internal drive on a MacMini. I then booted from CD, formatted the new SSD device and then performed a Restore operation of the installed Tiger Server onto the new SSD.

One thing that was very interesting to watch… SSDs perform their best with you throw a large amount of sequential reads/writes at them. When doing a Restore with Disk Utility it’s best to use the “Erase Destination” option as that will enable the imaging system the best opportunity to use a Block level Restore, instead of a File based Restore. A Block level Restore streams data at the disk and the SSD just ate this up. I Restored the 3.2GB installed System image in about 3 minutes. This is about 3 times faster than restoring back to the 40GB Seagate (I went back and tried it for comparison)

As always, when deploying a new type of setup we test the power utilization and found that 1) the G4 MacMini uses a ridiculously low amount of Amps; 2) the SSD didn’t really save much power usage.

Some anecdotes from my ammeter (we use 120V here in the States):

Adding a USB Ethernet Adapter increased all values by about 0.06A
This is necessary for the backside network resource routing.

Some pictures

Transcend 32GB SSD

Transcend 32GB SSD upright

Transcend 32GB SSD in Apple System Profiler

And of course the obligatory speed of boot/login videos:

G4MiniSSD-Boot

G4MiniSSD-Login

We have several more G4 MacMinis and are looking for appropriate applications to utilize them especially with an ultra-reliable drive installed.

Posted by Brian Blood as Colocation, Hardware, OS X Server, Routers and Firewalls, Servers at 1:41 PM UTC

No Comments »

There are times when managing ECMSquared installs where a customer will want to migrate to a new piece of hardware.

One the biggest chores is migrating all of the stored mailbox data. When we migrate a customer from EIMS (Eudora Internet Mail Server) or another product to ECM, the migration must go through a IMAP to IMAP process using imapsync. With a hardware only change, there is no format change, so it’s just a matter of copying data. Well, how do you efficiently copy tens of gigabytes of mail data with tens of thousands of folders and hundreds of thousands of files?

rsync

but not just any old rsync. Here is the command I ended up using after culling through a bunch of web sites looking for the most efficient way of making the connection:

rsync -av -e “ssh -c blowfish” –delete root@newserverIP:/var/mail/data/1* /var/mail/data/ &
rsync -av -e “ssh -c blowfish” –delete root@newserverIP:/var/mail/data/2* /var/mail/data/ &
rsync -av -e “ssh -c blowfish” –delete root@newserverIP:/var/mail/data/3* /var/mail/data/ &
Several items of note here:

1. the -a switch is an “archive” option meaning that rsync will maintain all file permissions and dates as best as possbile.
2. the -e switch and -c blowfish tell rsync that when using ssh to communicate with the other system to have ssh use the blowfish cipher. Blowfish is a very fast block level cipher that is well suited to bulk data transfers.
3. the –delete switch tells rsync to delete anything in the target that is no longer in the source. This will be become important on the second pass (see below)
4. The multiple forked rsync calls can all run simultaneously as each set of directories picked up by numerical wildcards (ECM Maildir data is stored primarily by site id) will have some different folder contents/data sets to work on. The system will settle down and balance itself out between network and disk activity amongst the various processes. This makes sure all parts of the system are utilized to their fullest.
5. We make these calls based on having a passwordless ssh pre-setup between the servers.
6. We make two passes with this set of commands. Once to pre-load the data to the new server, and then once again at the cutover point. The second time there should be much less data to actually move over the network.

Posted by Brian Blood as Mail Server, Servers at 9:08 PM UTC

No Comments »

We have customers that we provide support for both unix and windows based systems. We like put metrics for these systems into our cacti monitoring system, especially performance based values. Here is an explanation I provided to a customer as we recently deployed a Linux based system for their MySQL database alongside their ASP.NET based web app:

–

On unix based systems, the metric of Load Average is based on the number of processes that have asked the kernel for cpu time and are currently waiting for that to be made available to them.

Ideally, you want your hardware to be powerful enough so that your Load Average is always below 1.0, meaning that when a process asks for cpu time it gets it without having to wait. It’s called an average because the cpu scheduler in the kernel reports these as an average over the past 1 minute, past 5 minutes and past 15 minutes and that is what you are seeing in that graph.

This is a completely different perspective than the graph that shows specific percent CPU utilization that we have setup for the Windows system.

It is possible to have 100% cpu utilization of a system, but a load average of <1.0. In this scenario, there is only One process that is asking for time and it is using all the cpu it can. The Load Avg is 1.0 or less because there aren’t other threads/processes that need the cpu as well during that period.

The more cpus, the more processes can concurrently request/have access to cpu time before the Load Average starts to reach 1.0, even if those processes are maxing out their individual cpus.

With more powerful cpus, the quicker a process will complete a task before the next task gets its time, so the “run queue” is emptied quicker, thereby keeping the Load Avg lower

Posted by Brian Blood as General, Hardware, Servers at 12:00 AM UTC

No Comments »

This is from an exchange on the OS X Server mailing list.

The OP was trying to get mcrypt compiled into his PHP5 install on a 64-bit machine running 10.5.4

On Jul 24, 2008, at 4:53 PM, Someone wrote:
> I’m trying to compile my own PHP (which I’ve done plenty of times
> in the past on other systems), but
> I need libmcrypt support. Because 10.5 OS X Server is all 64 bit, I
> need a 64bit php binary, which
> means all my linked libraries need to be 64 bit as well.
>
> I tried to compile libmcrypt by hand and with the SVN version of
> macports, but both versions give me
> the following error when I do `make test` after compiling php with
> them:
>
> dyld: Symbol not found: _arcfour_LTX__is_block_algorithm
> Referenced from: /usr/local/lib/libmcrypt.4.dylib
> Expected in: flat namespace
>
> Googling for this error just brings up a forum post on entropy.org,
> which isn’t much help. How can I get my 64bit php to play nice?
>
> I’m using CFLAGS=’-arch x86_64′ before all my ./configure
> statements to get a x86_64 build.
>

Jul 25 14:20:59 gerry org.apache.httpd[625]: httpd: Syntax error on line 1462 of /private/etc/apache2/httpd.conf: Syntax error on line 8 of /etc/apache2/sites/entropy-php.conf: Cannot load /usr/local/php5/libphp5.so into server: dlopen(/usr/local/php5/libphp5.so, 10): no suitable image found.
Did find:
/usr/local/php5/libphp5.so: no matching architecture in universal wrapper

—–

My response for solving the problem in a different way that I found on the Entropy.ch forums

—–

How about going the other way…..

I ran into a similar situation setting up an Intel Xserve with a new install of Leopard Server.
I was trying to get the entropy.ch PHP module (which contains mcrypt FYI) running and it would not load due to “no appropriate image found” errors.

Apache on Leopard is a fat binary and was trying to load as a 64-bit app, which of course means that all plug-ins, loadable libraries need to be 64-bit as well and the Entropy PHP doesn’t currently have a 64-bit x86 image.

Since this is for an Apache 2 web server and I would be using the pre-forked model and any single one of my requests that would be running PHP code would be very unlikely to need anywhere near more than even 1GB of RAM, I forced Apache to run as a 32-bit process so that the PHP module would load.

I did this by using the lipo command to thin out Apple’s apache2 binary (after making a backup of course) to just contain a 32-bit x86 image.

Here is a small shell script that will make a backup and run the lipo command:

#!/bin/sh -
#
#

cd /usr/sbin;
mv httpd httpd.ub;
lipo -info httpd.ub;
lipo -thin i386 httpd.ub -output httpd.i386;
lipo -info httpd.i386;
ln -s httpd.i386 httpd;

Posted by Brian Blood as OS X Server, Servers, Web App Development at 9:02 PM UTC

No Comments »