Many years ago, before there were Xserves, in an attempt to save rack space in our cabinets, we experimented with ripping the guts out of a PowerMac G4 and stuffing them into a 2U server case. Here is a photo of one of those attempts:
It worked out fairly well and we had a couple of these running for several years.
The main piece to the puzzle was finding the right L riser card to give us access to the AGP slot and 2 of the PCI slots, one for a secondary Ethernet card and another for an ATTO SCSI card.
Posted by Brian Blood as Colocation, Hardware, Servers at 10:15 AM CDT
No Comments »
In our previous adventures with Mac Minis as “blade” servers, I thought we might try installing Ubuntu/Debian on an Intel MacMini and seeing how the system performed against an OS X client based system.
Well, we did that and about a week later we wiped the machine and imaged off one of the other Minis and set it back up under OS X.
We had one of our techs scour all he could find on the net about installing Linux on an Intel MacMini and the biggest hurdle was getting something working in the EFI realm.
We ended up using rEFIt, a project on sourceforge, to allow us to dual boot into either Debian or Tiger. This had some issues, but in the end it worked out ok.
The USB Ethernet adapter also worked rather well right out of the box.
No, the real kicker was the on-board gigabit ethernet which is used on the backside primarily for database access. The Mini uses the Yukon based chipset for it’s GigE port and and this combination with the default ethernet driver installed by Debian induces a flow-control hang under certain loads.
Marcus Bointon hinted as much in comment #6 to my original article and so when the Debian Mini developed problems communication over that interface, I was pretty sure where to look.
Debian by default picks the “sky2″ driver for that PHY and it wasn’t cutting the mustard. Apparently this bug has been around for a couple of years (the chipset is also used on some other system boards) and the “workaround” is to recompile a different ethernet driver into the kernel and it solves the issue. Since running Debian on this system was merely a trial, we decided to punt instead of sinking more time in tinkering with it.
Under Debian, the Mini did actually perform about 10% better than when it was running Tiger. Ultimately, the OS turned out to not be the biggest factor in getting more performance out of the load balanced system as a whole. Tuning Apache and making some other improvements to the web application proved to be far more useful.
Posted by Brian Blood as Hardware, Linux, Servers at 11:57 PM CST
No Comments »
Server Monitor is an application that allows you to monitor the health of several Xserves over the network:
Sometimes the application gets a bit cranky about the connections it makes to the servers and reports that it can’t communicate or as you see here in the picture “reply not understood”. So we don’t really use it for serious monitoring other than as a cursory glance usually to check some items.
However, Apple really takes the cake with this knowledge-base article:
Xserve: Server Monitor does not authenticate with server over subnet
in which they claim that the way to fix the problems with their SOFTWARE, is to:
- Make the necessary changes to the username or password using Server Monitor.
- Quit Server Monitor.
- Shut down the Xserve that is the target of these changes.
- Remove the power cord from the back of the Xserve.
- Wait 30 seconds and plug the power cord back in.
- Power the server back on.
This sounds suspiciously similar to something an old tech friend of mine once told me:
There are sound scientifically proven reasons why one must sometimes sacrifice a chicken in order to get a SCSI chain to work.
Ugh.
Posted by Brian Blood as Colocation, Hardware, OS X Server, Servers, Soap Box at 11:33 PM CST
No Comments »
I forgot that I had my 2GB MicroCenter USB flash drive in the pocket of my sweat pants and they got put into the wash by my wife.
I was searching and searching the whole house trying to find it.
Finally, my wife pulled it out of the dryer, I plugged it into our Philips DVP5982 1080p Upscaling DVD Player and worked perfectly.
Amazing.
Posted by Brian Blood as Hardware at 10:38 PM CST
No Comments »
In the past couple of months, we’ve started to see the IBM/Hitachi 180 GB drives that came with the Apple Hard drive modules for the Xserve G4 and ATA Xraids fail.
We always have these in a software mirror RAID, so we don’t have any data loss, but we are proactively starting to replace these drives with new Seagate 250GB drives. These drives not only have better caches, but run cooler and with less power usage.
Posted by Brian Blood as Hardware, Servers at 9:15 AM CST
3 Comments »
It has come to my attention that the thumbscrews on the front panel of Xserves are a very important component of the overall structural integrity of the rack mounting system. they not only keep the Xserve from coming out of the case, but they also secure the forward portion of the server to the mounted top case. I’ve seen many Xserves that seemed as if they could pop out of their cases due to a small amount of bowing in the case. Having the thumbscrews tightened down adds another vertical support component.
Posted by Brian Blood as Colocation, Hardware, Servers at 11:54 AM CST
No Comments »
Was reading this article today on Google’s initiative to expand the economics of alternative energy:
Google expands into alternative energy
MICHAEL LIEDTKE, AP Business Writer
I love this paragraph here:
Toward that end, Google aims to produce one gigabyte of power from renewable energy at prices below the rates of electricity generated at coal-burning plants. One gigabyte power would be enough to supply the needs of a city the size of San Francisco.
I’m not exactly sure what one “gigabyte” of power is, but if Google is using them, I’m sure it’s a lot. 
I wonder if it’s comparable to the power you can generate by tapping into a clock tower that’s about to be struck by lightning.
Semi-related is the speculation that Google is building their own 10Gbit optical switches.
Guess they’ll combine their gigabytes of power with the gigabits of light. See: I knew lightning would be in there somewhere!
Posted by Brian Blood as Hardware, Soap Box at 9:14 AM CST
No Comments »
Every time we setup a server for a client, we always specify, at the minimum, mirrored drive pairs for boot and for data.
How those pairs are created has varied depending on the hardware involved.
PowerMac G5s (having only 2 native internal bays) get either a software RAID pair or a hardware RAID card. Or they get a drive bracket installed to increase internal capacity.
Xserve G4s are always built with 2 software RAID mirrors as the four bays in that system and not ability to use a hardware RAID card make it the most natural setup. Usually 2 x 60GB drives for the Boot and whatever size for the Data is appropriate for the application. A system where there will lost of media files will tend to get 2 x 500GB, otherwise, 2 x 250GB is now the sweet spot for longevity and reliability.
Xserve G5s I really don’t see any use in deploying if you aren’t using the Apple Hardware RAID card, but that also has some issues as well.
PowerMac G4s (yes we still use these) are a different story. With their 4 internal bays (5 if you want to squeeze one into where the Zip drive was spec’d to go, gives you a lot of options. We use PowerMac G4s almost exclusively at our office for things like development servers, log processing, internal project management, etc… Most of these have been with us for quite some time and have gone from one version of the OS to another as time progresses.
For some reason, I guess penuriousness is the most likely culprit, the boot drives on these systems were never more than whatever IDE drive we threw in at the point of setup. No mirroring or redundancy at all. Our data drives tended to be pairs of Ultra 2/3 SCSI disks software mirrored together.
Well I was determined that we should follow our own advice, so I’ve been on a crusade of upgrading/migrating disks/partitions/sharepoints on our internal servers so that all systems enjoy mirrored goodness.
I like the RAID cards over software RAID for 2 reasons: 1. I can easily partition the “disk” presented by the RAID card to the OS; 2. No hit on the CPU (which on a faster machine would be negligible) for performing the mirroring.
In the end, we’ve used the following devices/setups:
- ACARD AEC-6880M in mirror mode with 2 x 400GB IDE - PowerMac G4/500DP
- ACARD AEC-6890M in mirror mode with 2 x 320GB SATA - PowerMac G4/500DP
We still use 10K and 15K rpm SCSI disks in these systems to hold speed critical data like MySQL tables.
For some clients we’ve used:
- ACARD AEC-6880M in mirror mode with 2 x 250GB IDE - PowerMac G4/500DP
- ACARD AEC-6890M in mirror mode with 2 x 400GB SATA - PowerMac G5 DP
- HighPoint Tech RocketRAID 2224 with X4 external enclosure connected by Infiniband (This was FAST)
We have seen some issues with the ACARD devices, but we’re leaning towards power supply problems more than anything directly wrong with the cards. Definitely flash the cards to the latest firmware before deploying is a good tip.
As for drives, I tend to spec Seagate drives almost exclusively these days.
A mishmash of ideas and technologies, but with the inexpensive options available today, there really isn’t any reason to deploy anything (even user desktops) without some redundancy. An extra $150 at the initial setup (drive, card, whatever) can save you and your employee a lot of time (meaning money if you bill for your time) down the road when a drive WILL fail.
Posted by Brian Blood as Hardware, Servers at 1:39 PM CST
No Comments »
So, we’ve recently purchased a Xserve G5 so that we can upgrade the hardware that our mail server runs on from a Dual G4 Xserve.
The machine purchased came with a single 80GB SATA Drive Module, so I purchased two more ADMs from a supplier along with 3 of the Seagate ES (Enterprise Series) SATA drives at 320GB in size.
We also had been planning on upgrading a customers Xserve G5 that has a Hardware RAID PCI card and 3 x 80GB drives to 3 x 400 as they will be using this as a media server in their application.
We had quite a bit of trouble getting the LSI/Megaraid card to work reliably with these three drives.
I had done some research into the issue by searching my archives of the Apple OS X Server Mailing List, and found some statements that non-Apple firmware drives do not work with that RAID card. After some empirical testing we have confirmed this to a 100% certainty. So, if you are looking to upgrade drives in your Xserve G5 RAID set, you will need to find and install drives with Apple firmware. One possible source is that the Xserve RAID does NOT have this restriction and that you might be able to swap out drives that are in that device that have Apple Firmware and use them in the Xserve G5. Of course you’ve now created an “unsupported configuration” with your Xserve RAID and that’s a choice you can make.
The main symptoms we’ve been seeing is that the megaraid card doesn’t register itself with the system soon enough from a cold boot. And when it does, it does not (using the -showdevices command) show any of the drives attached to it. Warm booting after that sometimes will get the card and drives to show up, but after creating a RAID set and shutting down/rebooting, you have the same problems and the card starts sounding it’s alarm. Basically, totally unreliable as a subsystem. We’ve tried different cards, different Xserve G5s, different cables, different drives. Only one combination worked just as you would expect it to: Apple firmware Xserve drives.
Unfortunately, we’ve probably spent WAY more time in learning this tidbit than we would have saved if we had merely purchased Apple drives to begin with.
There is a possible source of bare SATA drives from Apple: Their 500GB drive they sell to be installed in the MacPro. It is $330 though, but that is cheaper than one with a new ADM.
UPDATE: I shared this blog post with a colleague who has an Xserve G5 and uses the RAID card and I knew that saying “100%” was going to be a mistake. He is using non-Apple Maxtor 300GB drives in his setup and reports no problems.
UPDATE: I did find 4 - 500GB SATA drives that had been pulled from MacPro machines. These looked to be good Seagate drives and they had the Apple logo. $100 each. Got them in, connected them up and unfortunately, the machine exhibited mostly the same symptoms as when the drives were not Apple drives. So I ordered some more 250GB drives from a source that has Xserve parts and threw them in. Worked.
Posted by Brian Blood as Hardware, Servers at 3:09 PM CDT
No Comments »
(or, the device formerly known as the ArrowPoint Content Smart™ Web switch)
Back in the heady days of the dot com boom, one needed to be able to assure you could handle a large amount of traffic for all those visitors that you just knew were coming to your web property. In order to do that, your web application needed to be able to scale, which meant load balancing amongst any number of servers. ArrowPoint was the darling of this market, founded in 1997 and scooped up by the Cisco mother ship in May 2000 for a whopping $6.1 Billion in stock. (yes, that’s billion; you know, real money) This was 491.94 times their current revenue. Obviously, Cisco wanted into this market in a big way.
So, ArrowPoint had a nice lineup of Layer 7 switches both for the big guys (the CS-800) and the not-so big (the CS-100 line). (SlashDot used a CS-100 at one point and then upgraded to a CS-800 after a DDOS killed it.)
These devices primary purpose was to distribute IP packets amongst a server farm based on whatever criteria you could think of. You could make your content rules as simple (basic IP address/any port) or as complex as you wanted (testing for existence of a specific cookie or portion of a URL on a specific domain)
After Cisco bought them, the primary change they made was turn the case their standard blue. They also added a few models with GBICs and more buffer ram on each port (CSS11155). They even added a model that used a PCMCIA Flash Disk (e.g. CSS11154-FD-AC) instead of the 2 or 4GB IDE drives that are used to hold the OS, known as the NS and the configuration and logs.
We had been using one of those first Cisco based versions (locked flash vers 4.0, operational flash 4.0) for the sell.com server farm. We were about to deploy a new load balanced app for another client, so we decided to do some refurbishing of our supply of CSSen. I picked up a couple off eBay, one of the Cisco versions and one of the older ArrowPoint versions for under $500 and started tearing them apart. These devices when they were new went for upwards of $20,000!!!
Some interesting tidbits
- We’ve seen a decent percentage of power supplies in these units fail. I’ve actually purchased older basic 8 port units off eBay merely to have spare power supplies available.
- Upgrading the RAM to the full 256MB is easy, cheap, fast and gives the CSS plenty of room to breath for packet processing.
- The IDE drives in these devices are getting fairly long in the tooth.
RAM:
There are two slots on the CSS motherboard for installed RAM. The slots are underneath (if yours has one) the daughtercard that is used for the additional ports (2 GBIC, 4 x 100FX, 4 x 100BT), so in order to upgrade the ram, you will need to pull this card out temporarily. It’s got about 3 screws and comes out without too much fuss.
The chip to use is a Micron 128MB DDR 100 MHz MT8LSDT1664HG-10EB1, so with 2 of these the CSS will have 256MB. I picked up a couple for sub $15 each.
Disk system:
The IDE drive in the non FD (Flash Disk) models is usually a Fujitsu 2, 4 or 6GB drive. The NS and logs take up a very small part of the space on these disks, so we decided to replace the only non-solid state part of the CSS (not counting the fans) with some newer, more reliable technology. I found a CompactFlash to IDE adapter for sub $20 and a 2GB CompactFlash card for about $60. I did some research into the long-term reliability and durability of CompactFlash. There are industrial-strength CF cards, but they are about 5-10 times as expensive. The major technological consideration of CF cards is the use of single-cell vs multi-cell memory. For long-term reliability, you want single-cell as the electronics on the card will actually monitor the health and adjust the storage of data within the cells as it finds problems and single-cell CF is also rated for a higher number of writes and has a higher MTBF. Good explanation here: DailyTech - Solid-state Drives Ready for Prime Time
So, with a 2GB Kingston Elite Pro “disk” installed, we merely use the Offline Diagnostic Menu accessible from the console port to format the new disk and use the boot from FTP function to pull down an updated NS (an ADI or ArrowPoint Distribution Image) onto the disk and it’s ready to start configuring.
The FD model of CSS comes with a PCMCIA to IDE sled in the place of the hard drive. Inserted into that slot is a 350MB SanDisk PCMCIA flash card. We’ve purchased the 1.2GB version of these cards and done the same process as above. Flash goodness all around.
One interesting note, I expected the see some decent amount of savings in amps when replacing an actual hard disk drive with a flash drive, but curiously, I didn’t. The device pulled about 0.92 amps (110V) with the hard drive and only went down to 0.85A with the flash drive. It’s interesting that a device of this type pulls so much current in the first place. Most of the switches we utilize typically draw in the 0.3A range or less. I guess that could be related to why we see a higher failure rate with the power supplies.
Summary
In the end, we ended up with some new/spare load balancers that have been cleaned up, upgraded and made more reliable. Not bad for a couple hundred dollars spent.
Posted by Brian Blood as Content Networking, Hardware, Routers and Firewalls at 5:56 PM CDT
No Comments »
