Apple, in it’s Infinite wisdom, has decided that we no longer need QuickTime Streaming as a function of OS X Server. A client recently purchased a new MacMini Server (Lion Server) and in the process of working through migrating data and services off some old XServe G4s we found that QTSS was missing.
There is the Darwin Streaming Server project as Mac OS Forge, so downloading and installing the latest build there (6.0.3) seemed like the proper course of action. One glitch though, the installer refuses to proceed against any OS Install that is OS X Server.
Since those checks are done with scripts, the workaround is to copy the DarwinStreamingServer.pkg installer bundle to your hard drive where it can be edited.
In the script: DarwinStreamingServer.pkg/Contents/Resources/VolumeCheck
on line 29 that first stanza of checks which begins with a check for $INSTALLED_SERVER_OS_VERS…
…comment that entire stanza out with # and the Installer will then let you install the DSS on your Lion Server install.
This might also prove useful:
chmod +ai “qtss allow list,search,readattr,readextattr,readsecurity” QTSSMoviesDir
Posted by Brian Blood as OS X Server, Servers at 11:41 AM UTC
1 Comment »
We have a development server for a client that was recently upgraded from Tiger Server to Leopard Server. This system holds the subversion repository and the staging sites for their hosted application. One of the configured pieces is that whenever someone commits into the SVN repository, we have a post-commit hook that sends a message to all the developers with the information from the revision commit. Email on this system is handling by the Apple built-in Postfix. When the system was upgraded, we noticed that we no longer received our SVN commit messages. Investigating this I found two things that needed fixing.
My first problem was that the logging that postfix was sending to syslogd was very sparse. so I checked through all the settings twice in Server Admin and directly in the main.cf and master.cf files. Took me a while, but I finally looked at the /etc/syslogd.conf file and found that the facility entry for mail was set to mail.warn. Checked the Server Admin setting for SMTP log level and set it to Debug.
Second problem: now that logging was fixed, I can see that the relayhost set in config was rejecting the messages. So not only were the original messages being rejected, the bounce messages were being bounced. Essentially anything being send was dying a quick death. I fixed the relayhost setting, tried another messages and BAM, message delivered.
Upgrading from Tiger to Leopard is an important step to take, but with all upgrades, one must really go through all your settings once again to verify their correctness.
Posted by Brian Blood as OS X Server, Servers at 3:54 PM UTC
No Comments »
A client needs to have a Site to Site VPN between a server at their office and a NetScreen at their colo.
I did a fresh new install of Leopard Server fully and cleanly updated to 10.5.8 running on a G4 MacMini to make sure I can configure both sides properly.
My test Server is on a clean public static IP address for the built-in ethernet.
Secondary ethernet using a USB Ethernet adapter for the private side of the network.
System has no issues until…..
I used the s2svpnadmin cli tool to create a new shared-secret IPSec tunnel to a NetScreen at our colo.
Very basic setup, nothing fancy (not like the tool lets you do anything fancy.)
After creating the config I start to get these entries in my system.log:
Mar 10 12:55:56 test1 vpnd: Server ‘TestColo’ starting…
Mar 10 12:55:56 test1 TestColo: 2010-03-10 12:55:56 CST Server ‘TestColo’ starting…
Mar 10 12:55:56 test1 vpnd: Listening for connections…
Mar 10 12:55:56 test1 TestColo: 2010-03-10 12:55:56 CST Listening for connections…
Mar 10 12:55:57 test1 ReportCrash: Formulating crash report for process vpnd
Mar 10 12:55:57 test1 com.apple.launchd (TestColo): Exited abnormally: Bus error
Mar 10 12:55:57 test1 com.apple.launchd (TestColo): Throttling respawn: Will start in 9 seconds
Mar 10 12:55:57 test1 ReportCrash: Saved crashreport to /Library/Logs/CrashReporter/vpnd_2010-03-10-125556_MacServe-Test1.crash using uid: 0 gid: 0, euid: 0 egid: 0
and looking at the crash report:
Process: vpnd 
Version: ??? (???)
Code Type: PPC (Native)
Parent Process: launchd 
Date/Time: 2010-03-10 12:55:56.252 -0600
OS Version: Mac OS X Server 10.5.8 (9L34)
Report Version: 6
Anonymous UUID: 7E25DC5D-7D93-42B5-8F69-F7C823244418
Exception Type: EXC_BAD_ACCESS (SIGBUS)
Exception Codes: KERN_PROTECTION_FAILURE at 0×0000000000000000
Crashed Thread: 0
Thread 0 Crashed:
0 ??? 0000000000 0 + 0
1 vpnd 0x0000444c accept_connections + 1280
2 vpnd 0x00002a08 main + 1572
3 vpnd 0x00001a48 start + 68
4 ??? 0000000000 0 + 0
Thread 0 crashed with PPC Thread State 32:
srr0: 0×00000000 srr1: 0x4200f030 dar: 0x000513b0 dsisr: 0×42000000
…. etc. etc.
I do NOT have the VPN service “running”.
I did find this post on Apple discussions:
and followed the posters directions for manually starting the tunnel.
I still get a bit of fussing, but no crash.
I checked the IPSec SA/SPD info with setkey -PD and some basic pings across the network and the tunnel is active.
The crashing doesn’t seem to be cpu arch dependent as my system is ppc and the OP on the Apple board is using a x86 machine.
Kind of a bummer. It looks like there is probably some really simple issue here as the crash apparently happens very early in the setup process: “accept_connections”.
Hopefully this will help someone in the future.
Oh and FYI:
Leopard Server IPSec parameters for a Shared Secret based VPN:
Phase 1: DiffieHellman Group 2, 3DES, MD5, lifetime: 28800
Phase 2: No Perfect Forward Secrecy; Encapsulated Packet (no AH); AES128 encryption; SHA1 hash; lifetime: 3600; Compression: Deflate (this is optional)
Posted by Brian Blood as OS X Server, Routers and Firewalls at 11:22 AM UTC
No Comments »
The new G4 MacMini with the SSD is running beautifully. However, there is one little detail I’d like to take care of to help prolong the life of the SSD: disable the atime updating in the file system.
When we build out Linux servers, one of the configuration changes we always make is to add a noatime flag to the mount options for the file systems. atime is the Last Access Timestamp and really is useless in a server environment.
After some empirical testing…..
# mount -uvw -o noatime /
/dev/disk0s3 on / (local, journaled)
no effect. Even produced this entry in the system.log:
Jan 8 14:19:27 vpn KernelEventAgent: tid 00000000 received
unknown event (256)
# mount -vuw -o noatime /
/dev/disk4 on / (hfs, local, journaled, noatime)
where it looks to be supported…
The test is to check the last access time with a ls -lu , then simply cat the file, then ls -lu again.
I guess I’ll need to upgrade the Mini to Leopard Server!
Posted by Brian Blood as Hardware, OS X Server at 3:27 PM UTC
No Comments »
In our continuing adventures of putting the highly versatile Apple MacMini to work in all sorts of applications:
A customer of ours has a specialized application that is extremely low bandwidth, but needs to be able to be accessed through a VPN connection to a protected network resource. All the usual suspects for providing this kind of service (Netscreen/SSG, Netopia) all have artificial limits on the number of concurrent VPN connections. In order to service the number of concurrent users (anywhere from 1 to 200), the customer would be looking at least $5,000 for a capable device. Again, this system is very low bandwidth, so these kinds of devices would be serious overkill.
In comes the MacMini running Tiger Server with the built-in VPN service.
However… the MacMini has always had a single weak spot: the hard drive. Even with a very low disk based usage that a dedicated VPN Server would entail that vulnerability to a disk failure has always bothered me and I wanted this VPN server to be on par in terms of reliability as much as possible compared to a hardware device. Even the MacMinis that we build out for customers in a load balanced environment, we always replace with a extended use rated drive, as the extra $100 is a nominal expenditure compared to the time that would be involved in dealing with a failed system.
In comes the Solid State Disk. A 32GB SSD with a IDE interface runs just under $100. For a system running primarily as a VPN Server, this is PLENTY of space as the only real ongoing writes will be log files. I chose a MLC (Multi-Level Cell) device over a SLC (Single Level Cell) based device as the differences in speed and longevity between the two variations of SSD just were not a factor in this application. A MLC SSD already has a useful lifetime/MTBF an order of magnitude greater than a standard Winchester based disk with it’s moving parts.
In the end I used a 1.33Ghz G4 (PowerPC, not Intel) MacMini with 1GB of RAM, as that CPU should be able to easily handle the tasks of handling a couple hundred concurrent VPN connections using 1-2Kbits/sec each. Again, VERY low bandwidth application.
I took the MacMini with it’s original 40GB Seagate drive and installed Tiger Server so I could get it updated and setup while I waited for the SSD to arrive from the online vendor. When I was ready to install the SSD, I used Disk Utility to save the installed system to a disk image. Installing the SSD was straightforward and not any different that replacing the internal drive on a MacMini. I then booted from CD, formatted the new SSD device and then performed a Restore operation of the installed Tiger Server onto the new SSD.
One thing that was very interesting to watch… SSDs perform their best with you throw a large amount of sequential reads/writes at them. When doing a Restore with Disk Utility it’s best to use the “Erase Destination” option as that will enable the imaging system the best opportunity to use a Block level Restore, instead of a File based Restore. A Block level Restore streams data at the disk and the SSD just ate this up. I Restored the 3.2GB installed System image in about 3 minutes. This is about 3 times faster than restoring back to the 40GB Seagate (I went back and tried it for comparison)
As always, when deploying a new type of setup we test the power utilization and found that 1) the G4 MacMini uses a ridiculously low amount of Amps; 2) the SSD didn’t really save much power usage.
Some anecdotes from my ammeter (we use 120V here in the States):
|Cold Start spike
Adding a USB Ethernet Adapter increased all values by about 0.06A
This is necessary for the backside network resource routing.
Transcend 32GB SSD
Transcend 32GB SSD upright
Transcend 32GB SSD in Apple System Profiler
And of course the obligatory speed of boot/login videos:
We have several more G4 MacMinis and are looking for appropriate applications to utilize them especially with an ultra-reliable drive installed.
Posted by Brian Blood as Colocation, Hardware, OS X Server, Routers and Firewalls, Servers at 1:41 PM UTC
No Comments »
This is from an exchange on the OS X Server mailing list.
The OP was trying to get mcrypt compiled into his PHP5 install on a 64-bit machine running 10.5.4
On Jul 24, 2008, at 4:53 PM, Someone wrote:
> I’m trying to compile my own PHP (which I’ve done plenty of times
> in the past on other systems), but
> I need libmcrypt support. Because 10.5 OS X Server is all 64 bit, I
> need a 64bit php binary, which
> means all my linked libraries need to be 64 bit as well.
> I tried to compile libmcrypt by hand and with the SVN version of
> macports, but both versions give me
> the following error when I do `make test` after compiling php with
> dyld: Symbol not found: _arcfour_LTX__is_block_algorithm
> Referenced from: /usr/local/lib/libmcrypt.4.dylib
> Expected in: flat namespace
> Googling for this error just brings up a forum post on entropy.org,
> which isn’t much help. How can I get my 64bit php to play nice?
> I’m using CFLAGS=’-arch x86_64′ before all my ./configure
> statements to get a x86_64 build.
Jul 25 14:20:59 gerry org.apache.httpd: httpd: Syntax error on line 1462 of /private/etc/apache2/httpd.conf: Syntax error on line 8 of /etc/apache2/sites/entropy-php.conf: Cannot load /usr/local/php5/libphp5.so into server: dlopen(/usr/local/php5/libphp5.so, 10): no suitable image found.
/usr/local/php5/libphp5.so: no matching architecture in universal wrapper
My response for solving the problem in a different way that I found on the Entropy.ch forums
How about going the other way…..
I ran into a similar situation setting up an Intel Xserve with a new install of Leopard Server.
I was trying to get the entropy.ch PHP module (which contains mcrypt FYI) running and it would not load due to “no appropriate image found” errors.
Apache on Leopard is a fat binary and was trying to load as a 64-bit app, which of course means that all plug-ins, loadable libraries need to be 64-bit as well and the Entropy PHP doesn’t currently have a 64-bit x86 image.
Since this is for an Apache 2 web server and I would be using the pre-forked model and any single one of my requests that would be running PHP code would be very unlikely to need anywhere near more than even 1GB of RAM, I forced Apache to run as a 32-bit process so that the PHP module would load.
I did this by using the lipo command to thin out Apple’s apache2 binary (after making a backup of course) to just contain a 32-bit x86 image.
Here is a small shell script that will make a backup and run the lipo command:
mv httpd httpd.ub;
lipo -info httpd.ub;
lipo -thin i386 httpd.ub -output httpd.i386;
lipo -info httpd.i386;
ln -s httpd.i386 httpd;
Posted by Brian Blood as OS X Server, Servers, Web App Development at 9:02 PM UTC
No Comments »
We had the occasion to setup shared calendaring for our office. Here is a quick to-do list:
- Server must be a full Open Directory system (not standalone)
- Setup DNS name for calendar: ical.networkjack.info
- Setup iCal Server on OS X Server, including setting up vhost for calendar
- Turn on Calendaring (in Advanced tab in Workgroup Manager) for all users who will be calendar
Client Setup (Mac OS X 10.5)
- Add server as a Directory using Directory Utility, so that you can search
- Add account for iCal Server within iCal (Prefs -> Accounts)
- username : password
- Setup delegation to other users (this is where the Directory comes in)
That should be it for the most part.
Posted by Brian Blood as OS X Server at 12:03 PM UTC
No Comments »
With all the recent attention on the DNS exploit and the delay in response by Apple for providing a response, I thought I would undertake a deeper review some of our dns systems. We were already protected from the exploit as we do not provide recursive service with any of our unpatched dns servers.
Turns out that not only has Apple not patched the BIND install as of July 29, they haven’t really kept up with the installed config files, specifically the root servers hints file (/var/named/named.ca)
DNS is a distributed system, but a DNS resolver has to be given a place to start in it’s search for your name resolution request. That’s what the 13* root servers are about. They are the top of the tree so to speak. They are strategically placed around the world and load balanced and are THE critical part of the Internet infrastructure.
The hints file is populated with names/IP addresses of the 13 root servers preset like so:
A.ROOT-SERVERS.NET. 3600000 A 220.127.116.11
this goes on down from A to M
Occasionally, an IP address of one of the root server will change. There have been two updates in the past 4 years: B and L
B.ROOT-SERVERS.NET. 3600000 A 18.104.22.168
L.ROOT-SERVERS.NET. 3600000 A 22.214.171.124
; updated Jan 29, 2004
B.ROOT-SERVERS.NET. 3600000 A 126.96.36.199
; updated Nov, 1, 2007
L.ROOT-SERVERS.NET. 3600000 A 188.8.131.52
In Tiger Server, both B and L are out of date even with all updates and security patches applied
Leopard Server (up to 10.5.4) still shows the L root server out of date.
Update your /var/named/named.ca with the new entries preferably with this file:
then stop/start the named process
The hints file from internic will also include IPv6 information.
The guys at Renesys wrote a good article about the potential dangers of not querying the correct root servers.
* – The count is 13 and 13 being the count because that’s the maximum amount of dns records that can be crammed into a single IP packet response.
Posted by Brian Blood as OS X Server, Servers at 12:47 PM UTC
No Comments »
We have a few servers that are still running from being upgraded from the 10.2 and 10.3 days. Most all are running Tiger server, with one or two running Leopard.
Since all our XServe G4s run with dual mirrored pairs, we have quite a few of these software RAID sets.
The trick is that if a mirror pair becomes degraded, your server is now vulnerable because 10.4 disk utilities will not allow you to rebuild a v.1 raid set. You MUST convert the RAID set to v2 before it can be restored.
Unfortunately, the convertRAID verb for the command line diskutil, had some issues. Specifically if your drives had the OS 9 Drivers installed on them, or there wasn’t enough room to shrink the current partition, then the convertRAID operation would destroy the partition map of your disk.
As a result, the only way to get these volumes converted to v2 was to take the volume offline and run a Dusk Utility Restore operation from the v1 pair/disk to a new v2 pair/disk.
Since we have a handful of v1 RAID pairs that are the boot volume, being able to take a server down long enough to perform this operation is sometimes difficult.
The fine folks at SoftRAID have added a new feature to their latest version that allows you to convert RAID sets from the Apple RAID to a SoftRAID version and back. We’ve tested converting from v1 to SoftRAID format then to v2 and it works well. We had some strange behavior from the partition maps, but Mark James and the engineering staff gave us some tips on what to look for and this cleared those issues up. If you can’t run hardware RAID, get yourself a copy of SoftRAID.
As a lark, we also booted the server we used to test all of this on with Leopard Server and tried the diskutil convertRAID command to see if Apple had fixed that operation and it hallelujah it worked!
It even turned a single disk degraded v1 raid into a single member v2 raid set that could easily have another drive added to it for bringing it back to full redundancy. Good news this is as we won’t have to have a server with a boot volume that needs converting down for longer than it takes to boot from Leopard (a external FireWire drive of course), run the conversion, then reboot.
If you are running a server and do not have fully redundant (RAID is NOT backup) boot and data partitions, get thee to a store and buy another drive and add it in. The diskutil enableRAID command also works very well on a single disk.
Posted by Brian Blood as OS X Server, Servers at 10:40 AM UTC
1 Comment »
Server Monitor is an application that allows you to monitor the health of several Xserves over the network:
Sometimes the application gets a bit cranky about the connections it makes to the servers and reports that it can’t communicate or as you see here in the picture “reply not understood”. So we don’t really use it for serious monitoring other than as a cursory glance usually to check some items.
However, Apple really takes the cake with this knowledge-base article:
Xserve: Server Monitor does not authenticate with server over subnet
in which they claim that the way to fix the problems with their SOFTWARE, is to:
- Make the necessary changes to the username or password using Server Monitor.
- Quit Server Monitor.
- Shut down the Xserve that is the target of these changes.
- Remove the power cord from the back of the Xserve.
- Wait 30 seconds and plug the power cord back in.
- Power the server back on.
This sounds suspiciously similar to something an old tech friend of mine once told me:
There are sound scientifically proven reasons why one must sometimes sacrifice a chicken in order to get a SCSI chain to work.
Posted by Brian Blood as Colocation, Hardware, OS X Server, Servers, Soap Box at 11:33 PM UTC
No Comments »