One of our customers runs a system that has a shopping cart system and they like it when a customer upgrades their site to include full store functionality. This of course means an SSL certificate. And traditionally, this would mean an additional IP address that would need to added to the server to support the new certificate.

Entering wider spread usage is what is becoming known as the UCC (Unified Communications Certificate), which is just a fancy name for a regular X509 v3 certificate that utilizes the Subject Alternate Name extension. This extension allows the certificate creator to embed multiple alternate names that are cryptographically tied to the primary key that defines the certificate.

What this means in practical terms is that you could purchase a certificate whose primary name is www.mydomain.com and with the same certificate and IP address, support multiple variations of that domain name, such as secure.mydomain.com or www.mydomain.net, www.mydomain.org, etc… This is different than a wildcard certificate that is tied to a specific domain name and can be used on any third level host name as desired: *.mydomain.com.

Since many entities that have a web presence tend to pick up the .net/.biz/.org variations on their .com domain name, this type of certificate means that those additional names can be secured with the same certificate/IP address. This means simpler configuration and not having to burn an IP address for every single name variation.

I’ve tested GoDaddy’s version of this certificate that gives you a 5 name cert for $60/year on plain jane Apache under 10.4 client and it works without issue. I have two different vhosts with different ServerName directives and the SSL config under both simply use the same SSL

Example. Assume I have a cert that I have based on mydomain.com as the primary name and has the following names embedded in as Subject Alternate Name extensions:

1. www.mydomain.com
2. store.mydomain.com
3. www.mydomain.net
4. mydomain.net
5. admin.mydomain.net

Also assume that I have two different facets of my web application. The public facing side that is served by the .com and the admin/extranet that is served under the .net variation. My Apache config would look like so:


Listen 443
NameVirtualHost *:443

<VirtualHost *:443>
ServerName www.mydomain.com
ServerAdmin support@mydomain.com
DocumentRoot “/www/mydomain-com/”
<IfModule mod_ssl.c>
SSLEngine On
SSLCertificateFile “/etc/httpd/ssl/mydomain-com.crt”
SSLCertificateKeyFile “/etc/httpd/ssl/mydomain-com.key”
SSLCertificateChainFile “/etc/httpd/ssl/gd_intermediate_bundle.crt”
</IfModule>
ServerAlias mydomain.com store.mydomain.com
</VirtualHost>

<VirtualHost *:443>
ServerName www.mydomain.net
ServerAdmin support@mydomain.net
DocumentRoot “/www/mydomain-net/”
<IfModule mod_ssl.c>
SSLEngine On
SSLCertificateFile “/etc/httpd/ssl/mydomain-com.crt”
SSLCertificateKeyFile “/etc/httpd/ssl/mydomain-com.key”
SSLCertificateChainFile “/etc/httpd/ssl/gd_intermediate_bundle.crt”
</IfModule>
ServerAlias mydomain.net admin.mydomain.net
</VirtualHost>
notice the certificate files are the same for both vhosts and there is no distinguishing between IP addresses on the VirtualHost directive. Nice, clean, simple and it Just Works.

I’ve tested the certificate in the following browsers and none of them complain at all:

IE 6, FF2, Camino 1.5, Safari 2/3, Opera 8.5

X.509 v3 has been around at least since 2002 so it should be well supported.

Now, OS X Server should work exactly the same way as the underlying Apache system pieces are the same. The only issue comes into play with the way that ServerMgr handles storing certificates, their keys and passphrases and how Apache integrates all of these items.

Using the Server Admin app, you will not be able to use one of these certificates to secure two different Hosts that you enter as the host name in Apache is used by the “getsslpassphrase” binary to locate the certificate, private key, and password for the host in question to start SSL.

Since the primary name of the certificate is NOT the name of this secondary vhost, the loading process will fail. Any usage of this type of certificate will require you to manually create whatever non-primary VirtualHosts that would be setup to take advantage of these additional names secure by the “UCC” cert.

In order to get these certificates to load you will need to remove the passphrase from the key file so that Apache doesn’t call the getpassphrase and then fail.

the basic command for doing this is:
openssl rsa -in mydomain.key -out mydomain.key.open

For those who are use to working with OS X Server and doing something a bit out of the ordinary, this should not be surprising.

More SSL Certificate vendors are starting to offer this type of certificate, but I like the extra feature that the folks at DigiCert include with ALL their certificates: unlimited usage on any number of servers.

The DigiCert products are more expensive than the ones at GoDaddy, but theirs are single root certificates and they also go through additional layers of validation more than the simple “can you get an email at the domain” verification that GoDaddy utilizes.

Posted by Brian Blood as OS X Server, Servers, Web App Development at 5:56 PM UTC

No Comments »

Question to my Apple Authorized dealer that I just ordered Leopard
Server from to install on an Xserve G4:

> can you find out for me how I can get CD versions of the Leopard
> Server installer?
> we have some xserves that do not have DVD capable optical drives.

His Reply:
> From what I understand it doesn’t exist, one of the requirement of
> Leopard is a DVD drive. I’m guessing you’ll have to
> boot from another computer that has a DVD drive in Firewire Target
> mode.

My retort:
> From:
> http://www.apple.com/server/macosx/specs.html
> System Requirements
>
> Mac server or desktop computer with an Intel, PowerPC G5, or
> PowerPC G4 (867MHz or faster) processor; 1GB of physical RAM; 20GB
> of available disk space.

His Reply (looks like text pulled from somewhere)
> This is the first release of Mac OS X that’s not available in any
> form on CD, as all consumer-class computers that are capable of
> running Leopard also have at least a Combo Drive (DVD reading plus
> CD writing). Some Xserve models can run Mac OS X Server 10.5 but
> have only a CD-ROM drive; for such machines, you can perform a
> network installation using another computer running Leopard Server,
> or put the computer into Target Disk Mode and install Leopard
> Server from another computer that has a DVD reader.

Asking the Google, I find that text here:

http://db.tidbits.com/article/9243

So, installing Leopard Server on an Xserve G4 requires some sort of external piece of hardware. (Which of course, won’t be supplied by Apple (-; )

Target Disk Mode is not really an option as TGM on an Xserve G4 only exposes the device in Bay 1. Since we always build our Xserve G4s out with Mirrored arrays, we couldn’t start with a fresh pair of disks to install onto as a RAID pair. Perhaps a single drive RAID mirror might work and then add in second disk once booted up.

Ugh. I hate having to perform convoluted workarounds just to install an OS.

I guess all machines are equally supported for Leopard, just some are more equal than others.

UPDATE:  A really good suggestion was made to me on the OS X Server mailing list of making an image of the Leopard disc, then restoring onto a partition of a external firewire drive. This will boot the machine and run the installer just as if from an disc.This can even be used to put multiple install partitions of any number of install discs.

Posted by Brian Blood as OS X Server, Servers, Soap Box at 7:23 PM UTC

1 Comment »

We setup our hosting environment in such a way that a site is tied to a Group and anyone in that group (usually just the one user) can do anything in that Sites folder. We provide both AFP and FTP access to our clients Site folders and although the controls for making AFP propagate the proper permissions (group read/write) are readily available, FTP is a different story. Users uploading files by FTP tend to have their documents set with rw-r-r. Under Panther Server, I had edited the ftpd.conf file, but had to LOCK that file as the servermgr software would overwrite the file and wipe out my umask directive even if I merely OPENED Server Admin utility. Of course, then every time I did, Server Admin would complain that the FTP service was broken. Very annoying.

With Tiger, however, I have a little more flexibility as I can now edit the LaunchDaemon plist file and give xtfp an additional command line argument whenever it is launched by launchd.

Here is the procedure:

launchctl unload /System/Library/LaunchDaemons/ftp.plist

edit /System/Library/LaunchDaemons/ftp.plist  and add a new ProgramArgument:

This:

<key>ProgramArguments</key>
<array>
<string>xftpd</string>
<string>-a</string>
</array>

becomes:

<key>ProgramArguments</key>
<array>
<string>xftpd</string>
<string>-a</string>
<string>-u0002</string>
</array>

and then to complete the change:

 launchctl load /System/Library/LaunchDaemons/ftp.plist

and that’s it.

That is now in my little toolkit of tweaks to make to OS X Server when installing. You do have to make sure that change isn’t wiped out by some OS update.

Posted by Brian Blood as OS X Server at 11:53 AM UTC

No Comments »

A poster to Apple Mac OS X Server mailing list confirmed the problem with the FTP server in Tiger Server post 10.4.9.

This is a big screwup by Apple.

— BEGIN POST —

I’ve been facing the very same issue at a customer’s place. FTP service was set with “FTP Root and Share Points” and was working fine until I apply the most recent security update. Now, when connecting the this ftp box, I’m sent to the file system root (/). Of course, I can connect but permissions don’t let me copy anything there. I had to twist this setup big way for it to -kind of- work. More investigations to come.

Well, after I tested this deeper this morning, I can tell you what happened.

The 2007-004 Security Update replaced the ftp.plist in /System/Library/LaunchDaemons from Mac OS X server with the version from Mac OS X *Client*. There is no check in the installer if the update installs on client or Server, and it is the same update for both.

But, of course, FTP services on client and server are *very* different. With the client ftp.plist from client on the server, it is ftpd which is launched, not xftpd.

The solution is to replace the ftp.plist with a previous version from Mac OS X Server. If you don’t have it, here is its content :

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>Label</key>
        <string>com.apple.xftpd</string>
        <key>Program</key>
        <string>/usr/libexec/xftpd</string>
        <key>ProgramArguments</key>
        <array>
                <string>xftpd</string>
                <string>-a</string>
        </array>
        <key>Sockets</key>
        <dict>
                <key>Listeners</key>
                <dict>
                        <key>SockPassive</key>
                        <true/>
                        <key>SockServiceName</key>
                        <string>ftp</string>
                        <key>SockType</key>
                        <string>SOCK_STREAM</string>
                </dict>
        </dict>
        <key>inetdCompatibility</key>
        <dict>
                <key>Wait</key>
                <false/>
        </dict>
</dict>
</plist>

Restart the server (relaunching the FTP service is not enough), and you should be up and running.
— END POST —

Thanks to Guillaume Gete

related url:

Apple Discussion Board thread

UPDATE: Looks like Apple has posted a Security Update to address this.

Posted by Brian Blood as OS X Server at 11:44 AM UTC

1 Comment »

We upgraded our main shared hosting Xserve this weekend from Panther Server to Tiger Server.

The next day we started to get some complaints about weird directory changes. Since we didn’t move anything around, we were sure these customers were crazy or something.   We also had some issues with the permissions not being set correctly on newly uploaded items. Apache couldn’t see the items. I had fought with the OS X Server FTP umask settings in the past and ended up having to lock the ftpaccess file because even opening the FTP screen in Server Admin would cause my defumask setting to be blown away. (I wanted a umask setting of 0002)

In the course of investigating these issues, we found that the Tiger Upgrader kept the old FTP server launch info around and was using ftpd instead of xftpd. I figured this out by checking the entries for /System/Library/LaunchDaemons/ftp.plist on a system that had had a fresh 10.4 server install. Once I copied over the correct directives and did a launchctl unload and load, FTP service behavior returned to “normal”.

Ugh. Thanks, Apple.

Now, I have to go back to all the systems we’ve upgraded from 10.3 to 10.4 and make sure this nonsense hasn’t happened anywhere else.

Posted by Brian Blood as OS X Server at 3:49 PM UTC

No Comments »

For some reason, for a long while now net-snmp which the snmpd agent on OS X is based on has been broken in several aspects.

First: there is some wierd error that keeps coming up (sometimes) when you try and start snmpd:

nlist err: neither icmpstat nor _icmpstat found.

which I found the answer to fixing here.

Pretty simple, change this line in /System/Library/StartupItems/SNMP/SNMP from:

/usr/sbin/snmpd to /usr/sbin/snmpd -I -icmp

OK, that fixes that.

Second, I want to specify in the snmpd.conf file data for the “agentaddress”. I want to run snmpd on a different port and/or to restrict it to a backside interface.

so, every time I would put something like this into snmpd.conf:

agentaddress 16001

which is what the interactive snmpconf program put in there itself, I’d get this annoying error message in /var/log/snmpd.log

Error opening specified endpoint “16001″
Server Exiting with code 1

ugh.

I finally figured out how to make the stupid thing work. Don’t put the agentaddress specification into snmpd.conf; add it to the command that launches snmpd as it can be a command line option.

So finally to get snmpd on OS X to work:

/usr/sbin/snmpd -I -icmp

becomes:

/usr/sbin/snmpd -I -icmp 16001 or

/usr/sbin/snmpd -I -icmp 127.0.0.1:16001

So apparently the issue is somewhere in the code that picks up this information from the config file and not with snmpd in general.

Like I said: VERY annoying.

Update: John Welch of AFP548.com fame has a new article giving a primer on snmp and it’s use/setup in 10.5/Leopard Server. From what we’ve seen with Leopard Server, Apple has fixed some of the basic flaws in snmp. As in, you don’t have to do the nonsense above any longer.

Posted by Brian Blood as OS X Server, Servers at 4:22 PM UTC

No Comments »

In the course of rebuilding a customer’s Panther Xserve G5 on a 2 drive software raid to Tiger on a 3 drive hardware RAID, we needed to
migrate the data quickly and efficiently. We didn’t need to upgrade the OS, but simply do a fresh install.

What I wanted to do was to install the RAID card, hook two of the drives up to card, leaving one of the main drives connected to the system bus.

I intended to try and create a degraded RAID 5 set with the two drives, then copy the data from the main drive over. Then shutdown, hook up the third drive and have the raid card start to rebuild the array on the fly.

This would give me the fastest way of copying over the data from the system.

Alas, it was not to be. the megaraid cli program complained that I didn’t have enough members to create the RAID 5 set:

# megaraid -create R5 -drive 1 2

MEGARAID CLI version 1.0.12

Insufficient Drives 2 for RAID5

INSUFFICIENT/WRONG argument found to complete command

I ended up having to copy the data to both the other server under FWTD, and copying to a connected Firewire disk.
In the end, the RAID 5 device was created with all 3 drives and is running smoothly.

Posted by Brian Blood as OS X Server, Servers at 1:40 PM UTC

No Comments »

A colo customer of ours wanted us to completely rebuild an Xserve G5 of theirs. It was running Panther server and started acting really squirrely. It was setup with an Apple software raid mirror of the drives in Bay 1 and Bay 3. There was an additional drive in Bay 2, but it wasn’t tied to anything.

The plan for rebuilding this box was to backup everything on the system, install the PCI Hardware RAID card, attach the three drives and then do a fresh Tiger Server install.

In the course of determining the best way to back up this box, we had the idea of putting the server into Firewire Target disk mode (FWTD) and attaching it to another server of ours with big fast disks. This turned out to be a pretty good solution, but I was pleasantly surprised by a feature.

We have all G4 Xserves and this G5 Xserve is the lone non-G4 box. So, based on my previous experience of using FWTD mode on G4 Xserves, I expected only the drive in the first of the three Bays to show up on the running server. Interestingly, when we connected the firewire cable, all of the disks including the Install CD in the CD drive of the G5 Xserve showed up on this other box.

Very cool.

Posted by Brian Blood as OS X Server, Servers at 3:52 PM UTC

1 Comment »

We run cacti to do graphing of a lot of our resources. Obviously, traffic on switches, routers and servers, but also we do application level tracking. Things like # of online users or transactions submitted to a batch system or hits on an Apache web server, etc, etc. This amount of monitoring can cause cacti to get pretty busy. And for a long time, it hasn’t been an issue. Then last July, we really started doing a lot more monitoring for a new application architecture we are hosting for a client. This started to increase the load on our monitoring server quite a bit.

Back in December, you can see a break. This is when I upgraded this box from Panther server to Tiger Server. As you can see the load really jumped. So, I dug into the server to see what was causing the issue. I kept seeing this larger than acceptable load average, and just sitting there watching top, I couldn’t really see any one process taking up as much cpu as was warranted. In top, when you see the total cpu %, you can generally add up the cpu% of the top 4 or 5 processes and get close to what it is telling you the total cpu utilization is. However, top was showing about 45% cpu used and the top 4 processes, were only adding up to about 20%. Where was this extra cpu time going? I then noticed that most of the cpu usage was in the “system” % and I started watching the process list closer. Turns out there was a continuous stream of snmpget processes being launched and completed one right after another. Aha! So cacti is forking off snmpget processes to go and retrieve data from the devices we monitor. No single process was doing much heavy processing but the shear number of forking was causing all this load.

So. What to do with all these forked processes? My first thought was that we are monitoring a lot of SNMP v1 devices and that in SNMP v1 you could only request a single value, but that in SNMP v2, you can make a request for a range of values. So, I went on an upgrading process of either swapping out older v1-only devices for v2 capable or upgrading the IOS on some Cisco switches we run to a version that supports SNMP v2. Interestingly, there is a place on the Cisco site where you can directly download the latest versions of some older Cisco switches, like the venerable 2924XL devices.

So, getting rid of some of the SNMP v1 devices did have some impact in reducing the number of forked processes generated by cacti. You can see that reflected in the drop on the Load Average graph around the end of December. However, this did not solve the issue to the extent I’d hoped as you can see there was still considerable load on the box. So, the only thing left to update was the PHP on the server to a version with the php-snmp functions built-in so that no forking would be necessary. This meant: PHP 5.

I updated the server’s MySQL to MySQL 5.0.x. Then updated the cacti install to the latest version. Then downloaded the PHP 5.2 installer from Mr. Liyange’s site and made the upgrade and making the appropriate changes to the php.ini file.
I had done this same setup for a client before. PHP 5.2, cacti, etc and had had some issues with the built into PHP snmp functions. There is a function in the lib/snmp.php file in cacti called: snmp_get_method($version). It’s purpose is to find the best method of calling SNMP, based on the requested version of SNMP and the availability of certain functions or callable executable. The issue I had had was that when I was using the cacti interface to poll a device for interfaces to graph traffic for, php-snmp would fail and cacti would give a not very helpful snmp error. At that time, I merely added a line that forced cacti to use the snmp binaries. OK, now it was time to really track this down.

The first error I encountered in the php error log was when php was calling snmp2_get(), when I made cacti repoll a switch.

Could not open snmp connection: Unknown host

Which was obviously not correct. The second error message I saw in the base apache error log was:

No support for requested transport domain “udp”

so, I did some googling and found (primarily in the PHP bug tracking site) that php-snmp tended to work fine in the CLI version of PHP (which is what is used when cacti does it’s normal periodic polling), but that gave these errors when called from within the Apache module

I had also turned on debugging in SNMP, by adding:

doDebugging 1

to the snmp client conf file at /usr/share/snmpd/snmp.conf and watching all the relevant logs. SNMP outputs a LOT of info, and I had already gotten an idea of what the basic problem was, so I turned debugging back off.
OK, the basic issue here was that php-snmp works in PHP cli, but not in Apache. I guess you could call it a hack, but I merely added this line to the top of the snmp_get_method() function:

if (!empty($_SERVER['HTTP_HOST'])) return SNMP_METHOD_BINARY;

which is basically forcing cacti to call the cli version of snmp functions when there is a HTTP Host header, which is only going to be the case when this function is called from within Apache which is only when you are doing configs of your devices and data sources. (a lot of witches, there)All other times, the function continues on and chooses to use the php-snmp built-ins. So after all this debugging and tweaking, what’s the result? The polling process that cacti went through to poll all our devices by forking snmpget calls which used to take up to 3-4 minutes to complete, now takes (with only 2 concurrent poller processes) just under 14 seconds. As a result the load is now down considerably on that box:

MUCH better.

And the other things we have running on that server now run much more quickly.

Posted by Brian Blood as OS X Server, Servers at 11:25 AM UTC

2 Comments »

Like most hosting companies, I imagine, we have a public side network (10 and 100mbit connections) and a private side network (GigE)
We monitor the public side network switch ports for billing purposes and any local server to server traffic we try to keep on the private side network. Things like backups and database queries and mail routing, etc, etc..

Well, with the Software Update Server in Tiger Server we now have the ability to update a large set of systems using a single downloaded copy of updates. So using the nice little utility Software Update Enabler built by Andrew Wellington, we can point all our Xserves running Tiger Server to our single SUS through the backside network.

However….

The SUS is really just a customized instance of Apache whose content is updated by the swupd_sync process that is periodically run. It connects to the Apple SUS (akamai cache somewhere close) and compares the “index.sucatalog” file, which is merely a big plist xml file to it’s local copy and downloads the missing pieces.

When the swupd_sync process is done it replaces the host name/prefix directory portion of all the URLs that are in that file with it’s own hostname value like so:

<key>English</key>
<string>http://swcdn.apple.com/content/downloads/55/22/022-2725/252QW5yQ2d3BLTgrZn8CVB5KGj7YtyZt3b/022-2725.English.dist</string>
is transformed to:

<key>English</key>
<string>http://server.yoursus.com:8088/022-2725/252QW5yQ2d3BLTgrZn8CVB5KGj7YtyZt3b/022-2725.English.dist</string>

So here’s the problem.

That “server.yoursus.com” host name is more than likely the hostname of the public interface of your server. When a Tiger Xserve connects to our SUS (through it’s backside interface), it will read the index.sucatalog file, but when it goes to download the actual update files, it will be using those urls that swupd_sync generated. Which means the downloads will travel through the public side interfaces and completely negate the purpose of running the Updates through the backside network.

Now, I have asked the OS X Server product manager to allow a preference in the Software Update panel of Server Admin to set the host name that will be used by swpupd_sync in constructing those urls, but that feature hasn’t been added yet, so I had to come up with a workaround of my own to ensure that those urls in the index.sucatalog file refer to a private side network interface.

Enter launchd and sed

First thing to do was to come up with a script that altered those urls. I needed a quick and dirty find and replace on a text file. Not much out there that’s simpler at doing that than sed.

My simple shell script “alterswuphost.sh” for altering index.sucatalog which is in /usr/share/swupd/html
#!/bin/sh -
#

cd /usr/share/swupd/html;
sed -i .bkp ‘/server/s/server\.yoursus\.com/serverINTERNAL\.yoursus\.com/g’ index.sucatalog;
exit 0;

this says: for lines that contain “server”, find instances of “server.yoursus.com” and change them to “serverINTERNAL.yoursus.com”

the switch -i .bkp makes a backup of the file before running the find/replace

Now, I needed some way to know when I should run this script and this is where launchd came in to play.
I created a new LaunchAgent script to run whenever launchd noticed a change to the index.sucatalog file using the WatchPaths parameter:

<?xml version=”1.0″ encoding=”UTF-8″?>
<!DOCTYPE plist PUBLIC -//Apple Computer//DTD PLIST 1.0//EN http://www.apple.com/DTDs/PropertyList-1.0.dtd >
<plist version=”1.0″>
<dict>
<key>Label</key>
<string>info.networkjack.alterswupdhost</string>
<key>ProgramArguments</key>
<array>
<string>/Server/Scripts/alterswuphost.sh</string>
</array>
<key>WatchPaths</key>
<array>
<string>/Library/Logs/SoftwareUpdateServer.log</string>
</array>
<key>OnDemand</key>
<true/>
</dict>
</plist>

which I put into /Library/LaunchAgents/info.networkjack.alterswupdhost

and activated with:

launchctl load /Library/LaunchAgents/info.networkjack.alterswupdhost
so when swupd_sync updates it’s log file, launchd will follow it up with running this script that alters the hostnames with sed.

The Launch Agent watches “/Library/Logs/SoftwareUpdateServer.log” since that’s the most reliable of all the files to watch.

and that’s it!

I have this setup now and it seems to work ok.

Posted by Brian Blood as OS X Server at 2:36 AM UTC

4 Comments »