The new G4 MacMini with the SSD is running beautifully. However, there is one little detail I’d like to take care of to help prolong the life of the SSD: disable the atime updating in the file system.
When we build out Linux servers, one of the configuration changes we always make is to add a noatime flag to the mount options for the file systems. atime is the Last Access Timestamp and really is useless in a server environment.
After some empirical testing…..
Under Tiger:
# mount -uvw -o noatime /
/dev/disk0s3 on / (local, journaled)
no effect. Even produced this entry in the system.log:
Jan 8 14:19:27 vpn KernelEventAgent[34]: tid 00000000 received
unknown event (256)
Leopard:
# mount -vuw -o noatime /
/dev/disk4 on / (hfs, local, journaled, noatime)
where it looks to be supported…
The test is to check the last access time with a ls -lu , then simply cat the file, then ls -lu again.
—
I guess I’ll need to upgrade the Mini to Leopard Server!
Posted by Brian Blood as Hardware, OS X Server at 3:27 PM UTC
No Comments »
This is a continuation of the saga of helping a customer of ours with their MySQL on Windows issues.
The basic premise is that MySQL 5 running under Windows has problems with large numbers of connections/open files.
We initially presented our client with 2 choices for solving their problem:
- Setup MySQL on a different server running Linux
- Move their database backend from MySQL to Microsoft SQL Server
Option 1 would require a non-trivial capital outlay and time to setup and migrate the data over
Option 2 held more interest for them as a longer term solution as they were interested in some of the powerful reporting and analysis tools. However, the server that all of this was running on turned out to present a problem in the following way:
The system is a Dell PowerEdge 2950 with dual dual-core Xeons (with HyperThreading enabled, resulting in 8 logical cpus), 8GB RAM and running Windows 2003 Server x64. For a general ballpark estimate of pricing for MS SQL Server, I went to the Dell website and configured a similar machine and went to select a SQL Server as an add-on. Due to the way that Microsoft prices SQL Server for a web environment, you must license it for each processor you have installed in your server. A dual-processor licensed SQL Server 2005 running under Windows x64 turned out to be in excess of $14,000 on that page.
Needless to say, the customer had a bit of sticker shock. Not only would they have to plop down FOURTEEN grand, they would still need man-hours to work out kinks in transitioning the backend of the app from MySQL. Getting a fully loaded separate server for half that cost running Linux/MySQL was looking more attractive.
I was chatting with a friend about the customers dilemma and he had a brilliant suggestion: Run a Linux instance under VMWare on the existing hardware.
I broached the idea with the client and they were game to give it a shot. The server was at this point extremely underutilized and was only limited by the specific software implementations. They were already running MySQL here anyway, so putting a different wrapper around it in order to escape those limits was worth investigating.
The server needed some more disk space so they ordered 6 more of the 146GB 2.5″ SAS drive modules like the 2 that were already installed in a mirrored pair. These are great little devices for getting a decent number of spindles into a database server.
While they did that I installed the latest Release Candidate of VMWare Server 2.0 and went about setting up a Debian based Linux install with the latest MySQL 5.0 binary available. During testing we had some interesting processor affinity issues for the VM environment and after a very good experience and excellent responses in the VMWare community forums, I tweaked the config to have that VirtualMachine attach itself to processors 5-8 which correspond to all of the logical cpus on the second socket. This left the first socket’s worth of cpus for OS and IIS/ASP usage. Doing this would help avoid the cache issues that occur with multi-cpu systems and multi-threaded processes.
I ended up configuring the VM for 3.5GB RAM, 2 processors and tweaking the MySQL config to make good use of those resources for their INNOdb based tables.
The system has been running very reliably for over 2 months and is making very good use of the resources on that system. Load Average in the VM stays well under 1.0 and the cpus in the Windows system are easily handling all the different loads.
Overall I spent probably 8-10 hours installing and tweaking the config and helping them migrate their data. This was much less expensive to the other options and they got to make use of their existing hardware and software configurations.
A very positive experience and outcome.
Posted by Brian Blood as Database, Hardware, Linux, MySQL, Servers at 2:41 PM UTC
No Comments »
In our continuing adventures of putting the highly versatile Apple MacMini to work in all sorts of applications:
A customer of ours has a specialized application that is extremely low bandwidth, but needs to be able to be accessed through a VPN connection to a protected network resource. All the usual suspects for providing this kind of service (Netscreen/SSG, Netopia) all have artificial limits on the number of concurrent VPN connections. In order to service the number of concurrent users (anywhere from 1 to 200), the customer would be looking at least $5,000 for a capable device. Again, this system is very low bandwidth, so these kinds of devices would be serious overkill.
In comes the MacMini running Tiger Server with the built-in VPN service.
However… the MacMini has always had a single weak spot: the hard drive. Even with a very low disk based usage that a dedicated VPN Server would entail that vulnerability to a disk failure has always bothered me and I wanted this VPN server to be on par in terms of reliability as much as possible compared to a hardware device. Even the MacMinis that we build out for customers in a load balanced environment, we always replace with a extended use rated drive, as the extra $100 is a nominal expenditure compared to the time that would be involved in dealing with a failed system.
In comes the Solid State Disk. A 32GB SSD with a IDE interface runs just under $100. For a system running primarily as a VPN Server, this is PLENTY of space as the only real ongoing writes will be log files. I chose a MLC (Multi-Level Cell) device over a SLC (Single Level Cell) based device as the differences in speed and longevity between the two variations of SSD just were not a factor in this application. A MLC SSD already has a useful lifetime/MTBF an order of magnitude greater than a standard Winchester based disk with it’s moving parts.
In the end I used a 1.33Ghz G4 (PowerPC, not Intel) MacMini with 1GB of RAM, as that CPU should be able to easily handle the tasks of handling a couple hundred concurrent VPN connections using 1-2Kbits/sec each. Again, VERY low bandwidth application.
I took the MacMini with it’s original 40GB Seagate drive and installed Tiger Server so I could get it updated and setup while I waited for the SSD to arrive from the online vendor. When I was ready to install the SSD, I used Disk Utility to save the installed system to a disk image. Installing the SSD was straightforward and not any different that replacing the internal drive on a MacMini. I then booted from CD, formatted the new SSD device and then performed a Restore operation of the installed Tiger Server onto the new SSD.
One thing that was very interesting to watch… SSDs perform their best with you throw a large amount of sequential reads/writes at them. When doing a Restore with Disk Utility it’s best to use the “Erase Destination” option as that will enable the imaging system the best opportunity to use a Block level Restore, instead of a File based Restore. A Block level Restore streams data at the disk and the SSD just ate this up. I Restored the 3.2GB installed System image in about 3 minutes. This is about 3 times faster than restoring back to the 40GB Seagate (I went back and tried it for comparison)
As always, when deploying a new type of setup we test the power utilization and found that 1) the G4 MacMini uses a ridiculously low amount of Amps; 2) the SSD didn’t really save much power usage.
Some anecdotes from my ammeter (we use 120V here in the States):
|
40GB
Seagate |
32GB SSD |
| Max CPU/Disk |
0.25A |
0.24A |
| Cold Start spike |
0.26A |
0.25A |
| Idle |
0.13A |
0.12A |
Adding a USB Ethernet Adapter increased all values by about 0.06A
This is necessary for the backside network resource routing.
Some pictures
 Transcend 32GB SSD |
 Transcend 32GB SSD upright |
 Transcend 32GB SSD in Apple System Profiler |
And of course the obligatory speed of boot/login videos:
G4MiniSSD-Boot
G4MiniSSD-Login
We have several more G4 MacMinis and are looking for appropriate applications to utilize them especially with a ultra-reliable drive installed.
Posted by Brian Blood as Colocation, Hardware, OS X Server, Routers and Firewalls, Servers at 1:41 PM UTC
No Comments »
We have customers that we provide support for both unix and windows based systems. We like put metrics for these systems into our cacti monitoring system, especially performance based values. Here is an explanation I provided to a customer as we recently deployed a Linux based system for their MySQL database alongside their ASP.NET based web app:
–
On unix based systems, the metric of Load Average is based on the number of processes that have asked the kernel for cpu time and are currently waiting for that to be made available to them.
Ideally, you want your hardware to be powerful enough so that your Load Average is always below 1.0, meaning that when a process asks for cpu time it gets it without having to wait. It’s called an average because the cpu scheduler in the kernel reports these as an average over the past 1 minute, past 5 minutes and past 15 minutes and that is what you are seeing in that graph.
This is a completely different perspective than the graph that shows specific percent CPU utilization that we have setup for the Windows system.
It is possible to have 100% cpu utilization of a system, but a load average of <1.0. In this scenario, there is only One process that is asking for time and it is using all the cpu it can. The Load Avg is 1.0 or less because there aren’t other threads/processes that need the cpu as well during that period.
The more cpus, the more processes can concurrently request/have access to cpu time before the Load Average starts to reach 1.0, even if those processes are maxing out their individual cpus.
With more powerful cpus, the quicker a process will complete a task before the next task gets its time, so the “run queue” is emptied quicker, thereby keeping the Load Avg lower
Posted by Brian Blood as General, Hardware, Servers at 12:00 AM UTC
No Comments »
We have some devices in our server room in our office that due to the AC not running full blast on nights and Sundays, can get a little warm, even with spot cooling. To ensure better air flow, I’ve added a hi-tech piece of equipment to a PowerMac G4.
I normally would have added a PCI slot fan, but this server doesn’t have any available.
Posted by Brian Blood as Hardware, Servers at 10:52 AM UTC
No Comments »
Many years ago, before there were Xserves, in an attempt to save rack space in our cabinets, we experimented with ripping the guts out of a PowerMac G4 and stuffing them into a 2U server case. Here is a photo of one of those attempts:
It worked out fairly well and we had a couple of these running for several years.
The main piece to the puzzle was finding the right L riser card to give us access to the AGP slot and 2 of the PCI slots, one for a secondary Ethernet card and another for an ATTO SCSI card.
Posted by Brian Blood as Colocation, Hardware, Servers at 10:15 AM UTC
No Comments »
In our previous adventures with Mac Minis as “blade” servers, I thought we might try installing Ubuntu/Debian on an Intel MacMini and seeing how the system performed against an OS X client based system.
Well, we did that and about a week later we wiped the machine and imaged off one of the other Minis and set it back up under OS X.
We had one of our techs scour all he could find on the net about installing Linux on an Intel MacMini and the biggest hurdle was getting something working in the EFI realm.
We ended up using rEFIt, a project on sourceforge, to allow us to dual boot into either Debian or Tiger. This had some issues, but in the end it worked out ok.
The USB Ethernet adapter also worked rather well right out of the box.
No, the real kicker was the on-board gigabit ethernet which is used on the backside primarily for database access. The Mini uses the Yukon based chipset for it’s GigE port and and this combination with the default ethernet driver installed by Debian induces a flow-control hang under certain loads.
Marcus Bointon hinted as much in comment #6 to my original article and so when the Debian Mini developed problems communication over that interface, I was pretty sure where to look.
Debian by default picks the “sky2″ driver for that PHY and it wasn’t cutting the mustard. Apparently this bug has been around for a couple of years (the chipset is also used on some other system boards) and the “workaround” is to recompile a different ethernet driver into the kernel and it solves the issue. Since running Debian on this system was merely a trial, we decided to punt instead of sinking more time in tinkering with it.
Under Debian, the Mini did actually perform about 10% better than when it was running Tiger. Ultimately, the OS turned out to not be the biggest factor in getting more performance out of the load balanced system as a whole. Tuning Apache and making some other improvements to the web application proved to be far more useful.
Posted by Brian Blood as Hardware, Linux, Servers at 11:57 PM UTC
No Comments »
Server Monitor is an application that allows you to monitor the health of several Xserves over the network:
Sometimes the application gets a bit cranky about the connections it makes to the servers and reports that it can’t communicate or as you see here in the picture “reply not understood”. So we don’t really use it for serious monitoring other than as a cursory glance usually to check some items.
However, Apple really takes the cake with this knowledge-base article:
Xserve: Server Monitor does not authenticate with server over subnet
in which they claim that the way to fix the problems with their SOFTWARE, is to:
- Make the necessary changes to the username or password using Server Monitor.
- Quit Server Monitor.
- Shut down the Xserve that is the target of these changes.
- Remove the power cord from the back of the Xserve.
- Wait 30 seconds and plug the power cord back in.
- Power the server back on.
This sounds suspiciously similar to something an old tech friend of mine once told me:
There are sound scientifically proven reasons why one must sometimes sacrifice a chicken in order to get a SCSI chain to work.
Ugh.
Posted by Brian Blood as Colocation, Hardware, OS X Server, Servers, Soap Box at 11:33 PM UTC
No Comments »
I forgot that I had my 2GB MicroCenter USB flash drive in the pocket of my sweat pants and they got put into the wash by my wife.
I was searching and searching the whole house trying to find it.
Finally, my wife pulled it out of the dryer, I plugged it into our Philips DVP5982 1080p Upscaling DVD Player and worked perfectly.
Amazing.
Posted by Brian Blood as Hardware at 10:38 PM UTC
No Comments »
In the past couple of months, we’ve started to see the IBM/Hitachi 180 GB drives that came with the Apple Hard drive modules for the Xserve G4 and ATA Xraids fail.
We always have these in a software mirror RAID, so we don’t have any data loss, but we are proactively starting to replace these drives with new Seagate 250GB drives. These drives not only have better caches, but run cooler and with less power usage.
Posted by Brian Blood as Hardware, Servers at 9:15 AM UTC
3 Comments »
