We have a few servers that are still running from being upgraded from the 10.2 and 10.3 days. Most all are running Tiger server, with one or two running Leopard.
Since all our XServe G4s run with dual mirrored pairs, we have quite a few of these software RAID sets.
The trick is that if a mirror pair becomes degraded, your server is now vulnerable because 10.4 disk utilities will not allow you to rebuild a v.1 raid set. You MUST convert the RAID set to v2 before it can be restored.
Unfortunately, the convertRAID verb for the command line diskutil, had some issues. Specifically if your drives had the OS 9 Drivers installed on them, or there wasn’t enough room to shrink the current partition, then the convertRAID operation would destroy the partition map of your disk.
As a result, the only way to get these volumes converted to v2 was to take the volume offline and run a Dusk Utility Restore operation from the v1 pair/disk to a new v2 pair/disk.
Since we have a handful of v1 RAID pairs that are the boot volume, being able to take a server down long enough to perform this operation is sometimes difficult.
The fine folks at SoftRAID have added a new feature to their latest version that allows you to convert RAID sets from the Apple RAID to a SoftRAID version and back. We’ve tested converting from v1 to SoftRAID format then to v2 and it works well. We had some strange behavior from the partition maps, but Mark James and the engineering staff gave us some tips on what to look for and this cleared those issues up. If you can’t run hardware RAID, get yourself a copy of SoftRAID.
As a lark, we also booted the server we used to test all of this on with Leopard Server and tried the diskutil convertRAID command to see if Apple had fixed that operation and it hallelujah it worked!
It even turned a single disk degraded v1 raid into a single member v2 raid set that could easily have another drive added to it for bringing it back to full redundancy. Good news this is as we won’t have to have a server with a boot volume that needs converting down for longer than it takes to boot from Leopard (a external FireWire drive of course), run the conversion, then reboot.
If you are running a server and do not have fully redundant (RAID is NOT backup) boot and data partitions, get thee to a store and buy another drive and add it in. The diskutil enableRAID command also works very well on a single disk.
Posted by Brian Blood as OS X Server, Servers at 10:40 AM CST
No Comments »
Server Monitor is an application that allows you to monitor the health of several Xserves over the network:
Sometimes the application gets a bit cranky about the connections it makes to the servers and reports that it can’t communicate or as you see here in the picture “reply not understood”. So we don’t really use it for serious monitoring other than as a cursory glance usually to check some items.
However, Apple really takes the cake with this knowledge-base article:
Xserve: Server Monitor does not authenticate with server over subnet
in which they claim that the way to fix the problems with their SOFTWARE, is to:
- Make the necessary changes to the username or password using Server Monitor.
- Quit Server Monitor.
- Shut down the Xserve that is the target of these changes.
- Remove the power cord from the back of the Xserve.
- Wait 30 seconds and plug the power cord back in.
- Power the server back on.
This sounds suspiciously similar to something an old tech friend of mine once told me:
There are sound scientifically proven reasons why one must sometimes sacrifice a chicken in order to get a SCSI chain to work.
Ugh.
Posted by Brian Blood as Colocation, Hardware, OS X Server, Servers, Soap Box at 11:33 PM CST
No Comments »
One of our customers runs a system that has a shopping cart system and they like it when a customer upgrades their site to include full store functionality. This of course means an SSL certificate. And traditionally, this would mean an additional IP address that would need to added to the server to support the new certificate.
Entering wider spread usage is what is becoming known as the UCC (Unified Communications Certificate), which is just a fancy name for a regular X509 v3 certificate that utilizes the Subject Alternate Name extension. This extension allows the certificate creator to embed multiple alternate names that are cryptographically tied to the primary key that defines the certificate.
What this means in practical terms is that you could purchase a certificate whose primary name is www.mydomain.com and with the same certificate and IP address, support multiple variations of that domain name, such as secure.mydomain.com or www.mydomain.net, www.mydomain.org, etc… This is different than a wildcard certificate that is tied to a specific domain name and can be used on any third level host name as desired: *.mydomain.com.
Since many entities that have a web presence tend to pick up the .net/.biz/.org variations on their .com domain name, this type of certificate means that those additional names can be secured with the same certificate/IP address. This means simpler configuration and not having to burn an IP address for every single name variation.
I’ve tested GoDaddy’s version of this certificate that gives you a 5 name cert for $60/year on plain jane Apache under 10.4 client and it works without issue. I have two different vhosts with different ServerName directives and the SSL config under both simply use the same SSL
Example. Assume I have a cert that I have based on mydomain.com as the primary name and has the following names embedded in as Subject Alternate Name extensions:
- www.mydomain.com
- store.mydomain.com
- www.mydomain.net
- mydomain.net
- admin.mydomain.net
Also assume that I have two different facets of my web application. The public facing side that is served by the .com and the admin/extranet that is served under the .net variation. My Apache config would look like so:
Listen 443
NameVirtualHost *:443
<VirtualHost *:443>
ServerName www.mydomain.com
ServerAdmin support@mydomain.com
DocumentRoot “/www/mydomain-com/”
<IfModule mod_ssl.c>
SSLEngine On
SSLCertificateFile “/etc/httpd/ssl/mydomain-com.crt”
SSLCertificateKeyFile “/etc/httpd/ssl/mydomain-com.key”
SSLCertificateChainFile “/etc/httpd/ssl/gd_intermediate_bundle.crt”
</IfModule>
ServerAlias mydomain.com store.mydomain.com
</VirtualHost>
<VirtualHost *:443>
ServerName www.mydomain.net
ServerAdmin support@mydomain.net
DocumentRoot “/www/mydomain-net/”
<IfModule mod_ssl.c>
SSLEngine On
SSLCertificateFile “/etc/httpd/ssl/mydomain-com.crt”
SSLCertificateKeyFile “/etc/httpd/ssl/mydomain-com.key”
SSLCertificateChainFile “/etc/httpd/ssl/gd_intermediate_bundle.crt”
</IfModule>
ServerAlias mydomain.net admin.mydomain.net
</VirtualHost>
notice the certificate files are the same for both vhosts and there is no distinguishing between IP addresses on the VirtualHost directive. Nice, clean, simple and it Just Works.
I’ve tested the certificate in the following browsers and none of them complain at all:
IE 6, FF2, Camino 1.5, Safari 2/3, Opera 8.5
X.509 v3 has been around at least since 2002 so it should be well supported.
Now, OS X Server should work exactly the same way as the underlying Apache system pieces are the same. The only issue comes into play with the way that ServerMgr handles storing certificates, their keys and passphrases and how Apache integrates all of these items.
Using the Server Admin app, you will not be able to use one of these certificates to secure two different Hosts that you enter as the host name in Apache is used by the “getsslpassphrase” binary to locate the certificate, private key, and password for the host in question to start SSL.
Since the primary name of the certificate is NOT the name of this secondary vhost, the loading process will fail. Any usage of this type of certificate will require you to manually create whatever non-primary VirtualHosts that would be setup to take advantage of these additional names secure by the “UCC” cert.
In order to get these certificates to load you will need to remove the passphrase from the key file so that Apache doesn’t call the getpassphrase and then fail.
the basic command for doing this is:
openssl rsa -in mydomain.key -out mydomain.key.open
For those who are use to working with OS X Server and doing something a bit out of the ordinary, this should not be surprising.
More SSL Certificate vendors are starting to offer this type of certificate, but I like the extra feature that the folks at DigiCert include with ALL their certificates: unlimited usage on any number of servers.
The DigiCert products are more expensive than the ones at GoDaddy, but theirs are single root certificates and they also go through additional layers of validation more than the simple “can you get an email at the domain” verification that GoDaddy utilizes.
Posted by Brian Blood as OS X Server, Servers, Web App Development at 5:56 PM CST
No Comments »
Question to my Apple Authorized dealer that I just ordered Leopard
Server from to install on an Xserve G4:
> can you find out for me how I can get CD versions of the Leopard
> Server installer?
> we have some xserves that do not have DVD capable optical drives.
His Reply:
> From what I understand it doesn’t exist, one of the requirement of
> Leopard is a DVD drive. I’m guessing you’ll have to
> boot from another computer that has a DVD drive in Firewire Target
> mode.
My retort:
> From:
> http://www.apple.com/server/macosx/specs.html
> System Requirements
>
> Mac server or desktop computer with an Intel, PowerPC G5, or
> PowerPC G4 (867MHz or faster) processor; 1GB of physical RAM; 20GB
> of available disk space.
His Reply (looks like text pulled from somewhere)
> This is the first release of Mac OS X that’s not available in any
> form on CD, as all consumer-class computers that are capable of
> running Leopard also have at least a Combo Drive (DVD reading plus
> CD writing). Some Xserve models can run Mac OS X Server 10.5 but
> have only a CD-ROM drive; for such machines, you can perform a
> network installation using another computer running Leopard Server,
> or put the computer into Target Disk Mode and install Leopard
> Server from another computer that has a DVD reader.
Asking the Google, I find that text here:
http://db.tidbits.com/article/9243
So, installing Leopard Server on an Xserve G4 requires some sort of external piece of hardware. (Which of course, won’t be supplied by Apple (-; )
Target Disk Mode is not really an option as TGM on an Xserve G4 only exposes the device in Bay 1. Since we always build our Xserve G4s out with Mirrored arrays, we couldn’t start with a fresh pair of disks to install onto as a RAID pair. Perhaps a single drive RAID mirror might work and then add in second disk once booted up.
Ugh. I hate having to perform convoluted workarounds just to install an OS.
I guess all machines are equally supported for Leopard, just some are more equal than others.
UPDATE: A really good suggestion was made to me on the OS X Server mailing list of making an image of the Leopard disc, then restoring onto a partition of a external firewire drive. This will boot the machine and run the installer just as if from an disc.This can even be used to put multiple install partitions of any number of install discs.
Posted by Brian Blood as OS X Server, Servers, Soap Box at 7:23 PM CST
No Comments »
We setup our hosting environment in such a way that a site is tied to a Group and anyone in that group (usually just the one user) can do anything in that Sites folder. We provide both AFP and FTP access to our clients Site folders and although the controls for making AFP propagate the proper permissions (group read/write) are readily available, FTP is a different story. Users uploading files by FTP tend to have their documents set with rw-r-r. Under Panther Server, I had edited the ftpd.conf file, but had to LOCK that file as the servermgr software would overwrite the file and wipe out my umask directive even if I merely OPENED Server Admin utility. Of course, then every time I did, Server Admin would complain that the FTP service was broken. Very annoying.
With Tiger, however, I have a little more flexibility as I can now edit the LaunchDaemon plist file and give xtfp an additional command line argument whenever it is launched by launchd.
Here is the procedure:
launchctl unload /System/Library/LaunchDaemons/ftp.plist
edit /System/Library/LaunchDaemons/ftp.plist and add a new ProgramArgument:
This:
<key>ProgramArguments</key>
<array>
<string>xftpd</string>
<string>-a</string>
</array>
becomes:
<key>ProgramArguments</key>
<array>
<string>xftpd</string>
<string>-a</string>
<string>-u0002</string>
</array>
and then to complete the change:
launchctl load /System/Library/LaunchDaemons/ftp.plist
and that’s it.
That is now in my little toolkit of tweaks to make to OS X Server when installing. You do have to make sure that change isn’t wiped out by some OS update.
Posted by Brian Blood as OS X Server at 11:53 AM CDT
No Comments »
A poster to Apple Mac OS X Server mailing list confirmed the problem with the FTP server in Tiger Server post 10.4.9.
This is a big screwup by Apple.
— BEGIN POST —
I’ve been facing the very same issue at a customer’s place. FTP service was set with “FTP Root and Share Points” and was working fine until I apply the most recent security update. Now, when connecting the this ftp box, I’m sent to the file system root (/). Of course, I can connect but permissions don’t let me copy anything there. I had to twist this setup big way for it to -kind of- work. More investigations to come.
Well, after I tested this deeper this morning, I can tell you what happened.
The 2007-004 Security Update replaced the ftp.plist in /System/Library/LaunchDaemons from Mac OS X server with the version from Mac OS X *Client*. There is no check in the installer if the update installs on client or Server, and it is the same update for both.
But, of course, FTP services on client and server are *very* different. With the client ftp.plist from client on the server, it is ftpd which is launched, not xftpd.
The solution is to replace the ftp.plist with a previous version from Mac OS X Server. If you don’t have it, here is its content :
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.apple.xftpd</string>
<key>Program</key>
<string>/usr/libexec/xftpd</string>
<key>ProgramArguments</key>
<array>
<string>xftpd</string>
<string>-a</string>
</array>
<key>Sockets</key>
<dict>
<key>Listeners</key>
<dict>
<key>SockPassive</key>
<true/>
<key>SockServiceName</key>
<string>ftp</string>
<key>SockType</key>
<string>SOCK_STREAM</string>
</dict>
</dict>
<key>inetdCompatibility</key>
<dict>
<key>Wait</key>
<false/>
</dict>
</dict>
</plist>
Restart the server (relaunching the FTP service is not enough), and you should be up and running.
— END POST —
Thanks to Guillaume Gete
related url:
Apple Discussion Board thread
UPDATE: Looks like Apple has posted a Security Update to address this.
Posted by Brian Blood as OS X Server at 11:44 AM CDT
1 Comment »
We upgraded our main shared hosting Xserve this weekend from Panther Server to Tiger Server.
The next day we started to get some complaints about weird directory changes. Since we didn’t move anything around, we were sure these customers were crazy or something. ;-) We also had some issues with the permissions not being set correctly on newly uploaded items. Apache couldn’t see the items. I had fought with the OS X Server FTP umask settings in the past and ended up having to lock the ftpaccess file because even opening the FTP screen in Server Admin would cause my defumask setting to be blown away. (I wanted a umask setting of 0002)
In the course of investigating these issues, we found that the Tiger Upgrader kept the old FTP server launch info around and was using ftpd instead of xftpd. I figured this out by checking the entries for /System/Library/LaunchDaemons/ftp.plist on a system that had had a fresh 10.4 server install. Once I copied over the correct directives and did a launchctl unload and load, FTP service behavior returned to “normal”.
Ugh. Thanks, Apple.
Now, I have to go back to all the systems we’ve upgraded from 10.3 to 10.4 and make sure this nonsense hasn’t happened anywhere else.
Posted by Brian Blood as OS X Server at 3:49 PM CDT
No Comments »
For some reason, for a long while now net-snmp which the snmpd agent on OS X is based on has been broken in several aspects.
First: there is some wierd error that keeps coming up (sometimes) when you try and start snmpd:
nlist err: neither icmpstat nor _icmpstat found.
which I found the answer to fixing here.
Pretty simple, change this line in /System/Library/StartupItems/SNMP/SNMP from:
/usr/sbin/snmpd to /usr/sbin/snmpd -I -icmp
OK, that fixes that.
Second, I want to specify in the snmpd.conf file data for the “agentaddress”. I want to run snmpd on a different port and/or to restrict it to a backside interface.
so, every time I would put something like this into snmpd.conf:
agentaddress 16001
which is what the interactive snmpconf program put in there itself, I’d get this annoying error message in /var/log/snmpd.log
Error opening specified endpoint “16001″
Server Exiting with code 1
ugh.
I finally figured out how to make the stupid thing work. Don’t put the agentaddress specification into snmpd.conf; add it to the command that launches snmpd as it can be a command line option.
So finally to get snmpd on OS X to work:
/usr/sbin/snmpd -I -icmp
becomes:
/usr/sbin/snmpd -I -icmp 16001 or
/usr/sbin/snmpd -I -icmp 127.0.0.1:16001
So apparently the issue is somewhere in the code that picks up this information from the config file and not with snmpd in general.
Like I said: VERY annoying.
Update: John Welch of AFP548.com fame has a new article giving a primer on snmp and it’s use/setup in 10.5/Leopard Server. From what we’ve seen with Leopard Server, Apple has fixed some of the basic flaws in snmp. As in, you don’t have to do the nonsense above any longer.
Posted by Brian Blood as OS X Server, Servers at 4:22 PM CDT
No Comments »
In the course of rebuilding a customer’s Panther Xserve G5 on a 2 drive software raid to Tiger on a 3 drive hardware RAID, we needed to
migrate the data quickly and efficiently. We didn’t need to upgrade the OS, but simply do a fresh install.
What I wanted to do was to install the RAID card, hook two of the drives up to card, leaving one of the main drives connected to the system bus.
I intended to try and create a degraded RAID 5 set with the two drives, then copy the data from the main drive over. Then shutdown, hook up the third drive and have the raid card start to rebuild the array on the fly.
This would give me the fastest way of copying over the data from the system.
Alas, it was not to be. the megaraid cli program complained that I didn’t have enough members to create the RAID 5 set:
# megaraid -create R5 -drive 1 2
MEGARAID CLI version 1.0.12
Insufficient Drives 2 for RAID5
INSUFFICIENT/WRONG argument found to complete command
I ended up having to copy the data to both the other server under FWTD, and copying to a connected Firewire disk.
In the end, the RAID 5 device was created with all 3 drives and is running smoothly.
Posted by Brian Blood as OS X Server, Servers at 1:40 PM CST
No Comments »
A colo customer of ours wanted us to completely rebuild an Xserve G5 of theirs. It was running Panther server and started acting really squirrely. It was setup with an Apple software raid mirror of the drives in Bay 1 and Bay 3. There was an additional drive in Bay 2, but it wasn’t tied to anything.
The plan for rebuilding this box was to backup everything on the system, install the PCI Hardware RAID card, attach the three drives and then do a fresh Tiger Server install.
In the course of determining the best way to back up this box, we had the idea of putting the server into Firewire Target disk mode (FWTD) and attaching it to another server of ours with big fast disks. This turned out to be a pretty good solution, but I was pleasantly surprised by a feature.
We have all G4 Xserves and this G5 Xserve is the lone non-G4 box. So, based on my previous experience of using FWTD mode on G4 Xserves, I expected only the drive in the first of the three Bays to show up on the running server. Interestingly, when we connected the firewire cable, all of the disks including the Install CD in the CD drive of the G5 Xserve showed up on this other box.
Very cool.
Posted by Brian Blood as OS X Server, Servers at 3:52 PM CST
1 Comment »