I set up a replication slave at our office to a MySQL server running at our colo and the master server is pretty busy. So busy that even with the compressed protocol option turned on the stream was taking a good 60-70 kbps out of the available bandwidth of our T1. Since it isn’t critical that this data be real-time slaved, I made a small shell script that can take parameters for starting and stopping either the sql or io threads: mysqlslavectl.sh
#!/bin/sh -
#
#
USER=mysqladmin
PASSWD=adminpwd
SCRIPT="/usr/local/mysql/bin/mysql -u$USER -p$PASSWD -e "
StartService ()
{
case $1 in
sql ) $SCRIPT "START SLAVE SQL_THREAD" ;;
io ) $SCRIPT "START SLAVE IO_THREAD" ;;
* ) echo "$0: unknown Start argument: $1";;
esac
}
StopService ()
{
case $1 in
sql ) $SCRIPT "STOP SLAVE SQL_THREAD" ;;
io ) $SCRIPT "STOP SLAVE IO_THREAD" ;;
* ) echo "$0: unknown Stop argument: $1";;
esac
}
CheckCommand ()
{
case $1 in
start ) StartService "$2" ;;
stop ) StopService "$2" ;;
* ) echo "$0: unknown argument: $1";;
esac
}
CheckCommand "$1" "$2"
I can call this like so:
mysqlslavectl.sh start io
mysqlslavectl.sh stop sql
I setup a couple of crontab entries, one to start the io thread at 8pm and one to stop the io thread at 6AM.
The sql thread will always be running.
Posted by Brian Blood as Database, MySQL at 6:30 PM CDT
No Comments »
Many years ago, before there were Xserves, in an attempt to save rack space in our cabinets, we experimented with ripping the guts out of a PowerMac G4 and stuffing them into a 2U server case. Here is a photo of one of those attempts:
It worked out fairly well and we had a couple of these running for several years.
The main piece to the puzzle was finding the right L riser card to give us access to the AGP slot and 2 of the PCI slots, one for a secondary Ethernet card and another for an ATTO SCSI card.
Posted by Brian Blood as Colocation, Hardware, Servers at 10:15 AM CDT
No Comments »
So apparently YouTube, according to a Pakistani telecommunications authority, carries content that is “deemed offensive to Islam.
So the ISP either purposefully or accidentally, added custom routing configurations to it’s routers to block YouTube. The unfortuate side-effect was that these BGP announcements were propagated to a large part of the world’s routers, taking YouTube offline for a good number of people.
This fellow has a pretty good summation/chronology of what happened:
Renesys Blog: Pakistan hijacks YouTube
Posted by Brian Blood as Routers and Firewalls, Soap Box at 11:25 AM CST
No Comments »
We have a few servers that are still running from being upgraded from the 10.2 and 10.3 days. Most all are running Tiger server, with one or two running Leopard.
Since all our XServe G4s run with dual mirrored pairs, we have quite a few of these software RAID sets.
The trick is that if a mirror pair becomes degraded, your server is now vulnerable because 10.4 disk utilities will not allow you to rebuild a v.1 raid set. You MUST convert the RAID set to v2 before it can be restored.
Unfortunately, the convertRAID verb for the command line diskutil, had some issues. Specifically if your drives had the OS 9 Drivers installed on them, or there wasn’t enough room to shrink the current partition, then the convertRAID operation would destroy the partition map of your disk.
As a result, the only way to get these volumes converted to v2 was to take the volume offline and run a Dusk Utility Restore operation from the v1 pair/disk to a new v2 pair/disk.
Since we have a handful of v1 RAID pairs that are the boot volume, being able to take a server down long enough to perform this operation is sometimes difficult.
The fine folks at SoftRAID have added a new feature to their latest version that allows you to convert RAID sets from the Apple RAID to a SoftRAID version and back. We’ve tested converting from v1 to SoftRAID format then to v2 and it works well. We had some strange behavior from the partition maps, but Mark James and the engineering staff gave us some tips on what to look for and this cleared those issues up. If you can’t run hardware RAID, get yourself a copy of SoftRAID.
As a lark, we also booted the server we used to test all of this on with Leopard Server and tried the diskutil convertRAID command to see if Apple had fixed that operation and it hallelujah it worked!
It even turned a single disk degraded v1 raid into a single member v2 raid set that could easily have another drive added to it for bringing it back to full redundancy. Good news this is as we won’t have to have a server with a boot volume that needs converting down for longer than it takes to boot from Leopard (a external FireWire drive of course), run the conversion, then reboot.
If you are running a server and do not have fully redundant (RAID is NOT backup) boot and data partitions, get thee to a store and buy another drive and add it in. The diskutil enableRAID command also works very well on a single disk.
Posted by Brian Blood as OS X Server, Servers at 10:40 AM CST
No Comments »
In our previous adventures with Mac Minis as “blade” servers, I thought we might try installing Ubuntu/Debian on an Intel MacMini and seeing how the system performed against an OS X client based system.
Well, we did that and about a week later we wiped the machine and imaged off one of the other Minis and set it back up under OS X.
We had one of our techs scour all he could find on the net about installing Linux on an Intel MacMini and the biggest hurdle was getting something working in the EFI realm.
We ended up using rEFIt, a project on sourceforge, to allow us to dual boot into either Debian or Tiger. This had some issues, but in the end it worked out ok.
The USB Ethernet adapter also worked rather well right out of the box.
No, the real kicker was the on-board gigabit ethernet which is used on the backside primarily for database access. The Mini uses the Yukon based chipset for it’s GigE port and and this combination with the default ethernet driver installed by Debian induces a flow-control hang under certain loads.
Marcus Bointon hinted as much in comment #6 to my original article and so when the Debian Mini developed problems communication over that interface, I was pretty sure where to look.
Debian by default picks the “sky2″ driver for that PHY and it wasn’t cutting the mustard. Apparently this bug has been around for a couple of years (the chipset is also used on some other system boards) and the “workaround” is to recompile a different ethernet driver into the kernel and it solves the issue. Since running Debian on this system was merely a trial, we decided to punt instead of sinking more time in tinkering with it.
Under Debian, the Mini did actually perform about 10% better than when it was running Tiger. Ultimately, the OS turned out to not be the biggest factor in getting more performance out of the load balanced system as a whole. Tuning Apache and making some other improvements to the web application proved to be far more useful.
Posted by Brian Blood as Hardware, Linux, Servers at 11:57 PM CST
No Comments »
Server Monitor is an application that allows you to monitor the health of several Xserves over the network:
Sometimes the application gets a bit cranky about the connections it makes to the servers and reports that it can’t communicate or as you see here in the picture “reply not understood”. So we don’t really use it for serious monitoring other than as a cursory glance usually to check some items.
However, Apple really takes the cake with this knowledge-base article:
Xserve: Server Monitor does not authenticate with server over subnet
in which they claim that the way to fix the problems with their SOFTWARE, is to:
- Make the necessary changes to the username or password using Server Monitor.
- Quit Server Monitor.
- Shut down the Xserve that is the target of these changes.
- Remove the power cord from the back of the Xserve.
- Wait 30 seconds and plug the power cord back in.
- Power the server back on.
This sounds suspiciously similar to something an old tech friend of mine once told me:
There are sound scientifically proven reasons why one must sometimes sacrifice a chicken in order to get a SCSI chain to work.
Ugh.
Posted by Brian Blood as Colocation, Hardware, OS X Server, Servers, Soap Box at 11:33 PM CST
No Comments »
I forgot that I had my 2GB MicroCenter USB flash drive in the pocket of my sweat pants and they got put into the wash by my wife.
I was searching and searching the whole house trying to find it.
Finally, my wife pulled it out of the dryer, I plugged it into our Philips DVP5982 1080p Upscaling DVD Player and worked perfectly.
Amazing.
Posted by Brian Blood as Hardware at 10:38 PM CST
No Comments »
I was setting up a MacMini running Leopard for a client and I was tweaking some of the options and controls for “Remote Management”. I had connected in with Remote Desktop Admin with an admin level account and wanted to tighten up some of the controls for who was authorized to connect. Using this screen here:
I clicked on the “Only these users” button, without having put the account I was connected as into the list, and got dropped from the connection. Be warned that all of these controls take IMMEDIATE effect. Thanks Apple.
Posted by Brian Blood as General at 1:19 PM CST
No Comments »
In the past couple of months, we’ve started to see the IBM/Hitachi 180 GB drives that came with the Apple Hard drive modules for the Xserve G4 and ATA Xraids fail.
We always have these in a software mirror RAID, so we don’t have any data loss, but we are proactively starting to replace these drives with new Seagate 250GB drives. These drives not only have better caches, but run cooler and with less power usage.
Posted by Brian Blood as Hardware, Servers at 9:15 AM CST
3 Comments »
It has come to my attention that the thumbscrews on the front panel of Xserves are a very important component of the overall structural integrity of the rack mounting system. they not only keep the Xserve from coming out of the case, but they also secure the forward portion of the server to the mounted top case. I’ve seen many Xserves that seemed as if they could pop out of their cases due to a small amount of bowing in the case. Having the thumbscrews tightened down adds another vertical support component.
Posted by Brian Blood as Colocation, Hardware, Servers at 11:54 AM CST
No Comments »
