A client needs to have a Site to Site VPN between a server at their office and a NetScreen at their colo.
I did a fresh new install of Leopard Server fully and cleanly updated to 10.5.8 running on a G4 MacMini to make sure I can configure both sides properly.
My test Server is on a clean public static IP address for the built-in ethernet.
Secondary ethernet using a USB Ethernet adapter for the private side of the network.
System has no issues until…..
I used the s2svpnadmin cli tool to create a new shared-secret IPSec tunnel to a NetScreen at our colo.
Very basic setup, nothing fancy (not like the tool lets you do anything fancy.)
After creating the config I start to get these entries in my system.log:
Mar 10 12:55:56 test1 vpnd[1614]: Server ‘TestColo’ starting…
Mar 10 12:55:56 test1 TestColo[1614]: 2010-03-10 12:55:56 CST Server ‘TestColo’ starting…
Mar 10 12:55:56 test1 vpnd[1614]: Listening for connections…
Mar 10 12:55:56 test1 TestColo[1614]: 2010-03-10 12:55:56 CST Listening for connections…
Mar 10 12:55:57 test1 ReportCrash[1615]: Formulating crash report for process vpnd[1614]
Mar 10 12:55:57 test1 com.apple.launchd[1] (TestColo[1614]): Exited abnormally: Bus error
Mar 10 12:55:57 test1 com.apple.launchd[1] (TestColo): Throttling respawn: Will start in 9 seconds
Mar 10 12:55:57 test1 ReportCrash[1615]: Saved crashreport to /Library/Logs/CrashReporter/vpnd_2010-03-10-125556_MacServe-Test1.crash using uid: 0 gid: 0, euid: 0 egid: 0
and looking at the crash report:
Process: vpnd [1614]
Path: /usr/sbin/vpnd
Identifier: vpnd
Version: ??? (???)
Code Type: PPC (Native)
Parent Process: launchd [1]
Date/Time: 2010-03-10 12:55:56.252 -0600
OS Version: Mac OS X Server 10.5.8 (9L34)
Report Version: 6
Anonymous UUID: 7E25DC5D-7D93-42B5-8F69-F7C823244418
Exception Type: EXC_BAD_ACCESS (SIGBUS)
Exception Codes: KERN_PROTECTION_FAILURE at 0×0000000000000000
Crashed Thread: 0
Thread 0 Crashed:
0 ??? 0000000000 0 + 0
1 vpnd 0×0000444c accept_connections + 1280
2 vpnd 0×00002a08 main + 1572
3 vpnd 0×00001a48 start + 68
4 ??? 0000000000 0 + 0
Thread 0 crashed with PPC Thread State 32:
srr0: 0×00000000 srr1: 0×4200f030 dar: 0×000513b0 dsisr: 0×42000000
…. etc. etc.
I do NOT have the VPN service “running”.
I did find this post on Apple discussions:
http://discussions.apple.com/thread.jspa?threadID=1491028#7116067
and followed the posters directions for manually starting the tunnel.
I still get a bit of fussing, but no crash.
I checked the IPSec SA/SPD info with setkey -PD and some basic pings across the network and the tunnel is active.
The crashing doesn’t seem to be cpu arch dependent as my system is ppc and the OP on the Apple board is using a x86 machine.
Kind of a bummer. It looks like there is probably some really simple issue here as the crash apparently happens very early in the setup process: “accept_connections”.
Hopefully this will help someone in the future.
Oh and FYI:
Leopard Server IPSec parameters for a Shared Secret based VPN:
Phase 1: DiffieHellman Group 2, 3DES, MD5, lifetime: 28800
Phase 2: No Perfect Forward Secrecy; Encapsulated Packet (no AH); AES128 encryption; SHA1 hash; lifetime: 3600; Compression: Deflate (this is optional)
Posted by Brian Blood as OS X Server, Routers and Firewalls at 11:22 AM UTC
No Comments »
We have some Windows-based servers that we colocate for some clients.
We’ve always insisted that those devices sit behind some sort of protection and for a long time, we’ve used a Cisco 2621 as a screening router for a smaller subnet of our main address space. Any traffic that wanted to reach the protected IPs was routed through this device and we applied access list screening both inwards and outwards.
Over time, this device become unable to handle the traffic that was pushed through it and we decided to replace it. We had a 10-user model NetScreen 5GT that was untasked and since we had only a handful of devices on that protected subnet, we found a new home for the 5GT as a transparent firewall for those systems.
The protected subnet was compartmentalized with the use of a non-tagging VLAN on a our main Cisco customer attach switch, so segregation of a broadcast domain was not an issue. We merely needed to configure the 5GT into Layer 2 mode and setup the right policies for both directions of traffic.
I like to filter Bogons on our network so I started there. Now in this context, any traffic originating from the Untrusted side and having a source IP that existed on the Trusted could also be considered a Bogon so I made sure that rule was in place as well. Since Defined Addresses must be defined in terms of a security zone, I had to setup our Protected IPs in both Zones so I could define the correct policy
One problem I did run up against is the way Sessions are handled in ScreenOS. The max Sessions that can be tracked by this model of NetScreen is 2064, and in a busy period after installing the device we did get close to reaching the limit. The solution was to drop the timeout value for POP3 (one of the servers is a Mail Server) and HTTP/HTTPS in the Predefined Services section down to a very low value. This would ensure that there would be a faster turnover of entries in the Sessions table and keep it further away from the limit. This does mean a bit more work for the CPU, but the NetScreen’s ASICs are up to the challenge.
It has turned out to be a very good switchout of better hardware and management access policies in the ScreenOS Web management is much easier than with the Cisco ACL approach. My main gripe there is that to make a change to an access-list, I have to remove it from the interface, remote it from the router, then add the new access-list back to the router, then reapply it to the interface. A very tedious chore.
Posted by Brian Blood as Routers and Firewalls at 12:47 AM UTC
No Comments »
The new G4 MacMini with the SSD is running beautifully. However, there is one little detail I’d like to take care of to help prolong the life of the SSD: disable the atime updating in the file system.
When we build out Linux servers, one of the configuration changes we always make is to add a noatime flag to the mount options for the file systems. atime is the Last Access Timestamp and really is useless in a server environment.
After some empirical testing…..
Under Tiger:
# mount -uvw -o noatime /
/dev/disk0s3 on / (local, journaled)
no effect. Even produced this entry in the system.log:
Jan 8 14:19:27 vpn KernelEventAgent[34]: tid 00000000 received
unknown event (256)
Leopard:
# mount -vuw -o noatime /
/dev/disk4 on / (hfs, local, journaled, noatime)
where it looks to be supported…
The test is to check the last access time with a ls -lu , then simply cat the file, then ls -lu again.
—
I guess I’ll need to upgrade the Mini to Leopard Server!
Posted by Brian Blood as Hardware, OS X Server at 3:27 PM UTC
No Comments »
This is a continuation of the saga of helping a customer of ours with their MySQL on Windows issues.
The basic premise is that MySQL 5 running under Windows has problems with large numbers of connections/open files.
We initially presented our client with 2 choices for solving their problem:
- Setup MySQL on a different server running Linux
- Move their database backend from MySQL to Microsoft SQL Server
Option 1 would require a non-trivial capital outlay and time to setup and migrate the data over
Option 2 held more interest for them as a longer term solution as they were interested in some of the powerful reporting and analysis tools. However, the server that all of this was running on turned out to present a problem in the following way:
The system is a Dell PowerEdge 2950 with dual dual-core Xeons (with HyperThreading enabled, resulting in 8 logical cpus), 8GB RAM and running Windows 2003 Server x64. For a general ballpark estimate of pricing for MS SQL Server, I went to the Dell website and configured a similar machine and went to select a SQL Server as an add-on. Due to the way that Microsoft prices SQL Server for a web environment, you must license it for each processor you have installed in your server. A dual-processor licensed SQL Server 2005 running under Windows x64 turned out to be in excess of $14,000 on that page.
Needless to say, the customer had a bit of sticker shock. Not only would they have to plop down FOURTEEN grand, they would still need man-hours to work out kinks in transitioning the backend of the app from MySQL. Getting a fully loaded separate server for half that cost running Linux/MySQL was looking more attractive.
I was chatting with a friend about the customers dilemma and he had a brilliant suggestion: Run a Linux instance under VMWare on the existing hardware.
I broached the idea with the client and they were game to give it a shot. The server was at this point extremely underutilized and was only limited by the specific software implementations. They were already running MySQL here anyway, so putting a different wrapper around it in order to escape those limits was worth investigating.
The server needed some more disk space so they ordered 6 more of the 146GB 2.5″ SAS drive modules like the 2 that were already installed in a mirrored pair. These are great little devices for getting a decent number of spindles into a database server.
While they did that I installed the latest Release Candidate of VMWare Server 2.0 and went about setting up a Debian based Linux install with the latest MySQL 5.0 binary available. During testing we had some interesting processor affinity issues for the VM environment and after a very good experience and excellent responses in the VMWare community forums, I tweaked the config to have that VirtualMachine attach itself to processors 5-8 which correspond to all of the logical cpus on the second socket. This left the first socket’s worth of cpus for OS and IIS/ASP usage. Doing this would help avoid the cache issues that occur with multi-cpu systems and multi-threaded processes.
I ended up configuring the VM for 3.5GB RAM, 2 processors and tweaking the MySQL config to make good use of those resources for their INNOdb based tables.
The system has been running very reliably for over 2 months and is making very good use of the resources on that system. Load Average in the VM stays well under 1.0 and the cpus in the Windows system are easily handling all the different loads.
Overall I spent probably 8-10 hours installing and tweaking the config and helping them migrate their data. This was much less expensive to the other options and they got to make use of their existing hardware and software configurations.
A very positive experience and outcome.
Posted by Brian Blood as Database, Hardware, Linux, MySQL, Servers at 2:41 PM UTC
No Comments »
In our continuing adventures of putting the highly versatile Apple MacMini to work in all sorts of applications:
A customer of ours has a specialized application that is extremely low bandwidth, but needs to be able to be accessed through a VPN connection to a protected network resource. All the usual suspects for providing this kind of service (Netscreen/SSG, Netopia) all have artificial limits on the number of concurrent VPN connections. In order to service the number of concurrent users (anywhere from 1 to 200), the customer would be looking at least $5,000 for a capable device. Again, this system is very low bandwidth, so these kinds of devices would be serious overkill.
In comes the MacMini running Tiger Server with the built-in VPN service.
However… the MacMini has always had a single weak spot: the hard drive. Even with a very low disk based usage that a dedicated VPN Server would entail that vulnerability to a disk failure has always bothered me and I wanted this VPN server to be on par in terms of reliability as much as possible compared to a hardware device. Even the MacMinis that we build out for customers in a load balanced environment, we always replace with a extended use rated drive, as the extra $100 is a nominal expenditure compared to the time that would be involved in dealing with a failed system.
In comes the Solid State Disk. A 32GB SSD with a IDE interface runs just under $100. For a system running primarily as a VPN Server, this is PLENTY of space as the only real ongoing writes will be log files. I chose a MLC (Multi-Level Cell) device over a SLC (Single Level Cell) based device as the differences in speed and longevity between the two variations of SSD just were not a factor in this application. A MLC SSD already has a useful lifetime/MTBF an order of magnitude greater than a standard Winchester based disk with it’s moving parts.
In the end I used a 1.33Ghz G4 (PowerPC, not Intel) MacMini with 1GB of RAM, as that CPU should be able to easily handle the tasks of handling a couple hundred concurrent VPN connections using 1-2Kbits/sec each. Again, VERY low bandwidth application.
I took the MacMini with it’s original 40GB Seagate drive and installed Tiger Server so I could get it updated and setup while I waited for the SSD to arrive from the online vendor. When I was ready to install the SSD, I used Disk Utility to save the installed system to a disk image. Installing the SSD was straightforward and not any different that replacing the internal drive on a MacMini. I then booted from CD, formatted the new SSD device and then performed a Restore operation of the installed Tiger Server onto the new SSD.
One thing that was very interesting to watch… SSDs perform their best with you throw a large amount of sequential reads/writes at them. When doing a Restore with Disk Utility it’s best to use the “Erase Destination” option as that will enable the imaging system the best opportunity to use a Block level Restore, instead of a File based Restore. A Block level Restore streams data at the disk and the SSD just ate this up. I Restored the 3.2GB installed System image in about 3 minutes. This is about 3 times faster than restoring back to the 40GB Seagate (I went back and tried it for comparison)
As always, when deploying a new type of setup we test the power utilization and found that 1) the G4 MacMini uses a ridiculously low amount of Amps; 2) the SSD didn’t really save much power usage.
Some anecdotes from my ammeter (we use 120V here in the States):
|
40GB
Seagate |
32GB SSD |
| Max CPU/Disk |
0.25A |
0.24A |
| Cold Start spike |
0.26A |
0.25A |
| Idle |
0.13A |
0.12A |
Adding a USB Ethernet Adapter increased all values by about 0.06A
This is necessary for the backside network resource routing.
Some pictures
 Transcend 32GB SSD |
 Transcend 32GB SSD upright |
 Transcend 32GB SSD in Apple System Profiler |
And of course the obligatory speed of boot/login videos:
G4MiniSSD-Boot
G4MiniSSD-Login
We have several more G4 MacMinis and are looking for appropriate applications to utilize them especially with a ultra-reliable drive installed.
Posted by Brian Blood as Colocation, Hardware, OS X Server, Routers and Firewalls, Servers at 1:41 PM UTC
No Comments »
There are times when managing ECMSquared installs where a customer will want to migrate to a new piece of hardware.
One the biggest chores is migrating all of the stored mailbox data. When we migrate a customer from EIMS (Eudora Internet Mail Server) or another product to ECM, the migration must go through a IMAP to IMAP process using imapsync. With a hardware only change, there is no format change, so it’s just a matter of copying data. Well, how do you efficiently copy tens of gigabytes of mail data with tens of thousands of folders and hundreds of thousands of files?
rsync
but not just any old rsync. Here is the command I ended up using after culling through a bunch of web sites looking for the most efficient way of making the connection:
rsync -av -e “ssh -c blowfish” –delete root@newserverIP:/var/mail/data/1* /var/mail/data/ &
rsync -av -e “ssh -c blowfish” –delete root@newserverIP:/var/mail/data/2* /var/mail/data/ &
rsync -av -e “ssh -c blowfish” –delete root@newserverIP:/var/mail/data/3* /var/mail/data/ &
Several items of note here:
- the -a switch is an “archive” option meaning that rsync will maintain all file permissions and dates as best as possbile.
- the -e switch and -c blowfish tell rsync that when using ssh to communicate with the other system to have ssh use the blowfish cipher. Blowfish is a very fast block level cipher that is well suited to bulk data transfers.
- the –delete switch tells rsync to delete anything in the target that is no longer in the source. This will be become important on the second pass (see below)
- The multiple forked rsync calls can all run simultaneously as each set of directories picked up by numerical wildcards (ECM Maildir data is stored primarily by site id) will have some different folder contents/data sets to work on. The system will settle down and balance itself out between network and disk activity amongst the various processes. This makes sure all parts of the system are utilized to their fullest.
- We make these calls based on having a passwordless ssh pre-setup between the servers.
- We make two passes with this set of commands. Once to pre-load the data to the new server, and then once again at the cutover point. The second time there should be much less data to actually move over the network.
Posted by Brian Blood as Mail Server, Servers at 9:08 PM UTC
No Comments »
We have customers that we provide support for both unix and windows based systems. We like put metrics for these systems into our cacti monitoring system, especially performance based values. Here is an explanation I provided to a customer as we recently deployed a Linux based system for their MySQL database alongside their ASP.NET based web app:
–
On unix based systems, the metric of Load Average is based on the number of processes that have asked the kernel for cpu time and are currently waiting for that to be made available to them.
Ideally, you want your hardware to be powerful enough so that your Load Average is always below 1.0, meaning that when a process asks for cpu time it gets it without having to wait. It’s called an average because the cpu scheduler in the kernel reports these as an average over the past 1 minute, past 5 minutes and past 15 minutes and that is what you are seeing in that graph.
This is a completely different perspective than the graph that shows specific percent CPU utilization that we have setup for the Windows system.
It is possible to have 100% cpu utilization of a system, but a load average of <1.0. In this scenario, there is only One process that is asking for time and it is using all the cpu it can. The Load Avg is 1.0 or less because there aren’t other threads/processes that need the cpu as well during that period.
The more cpus, the more processes can concurrently request/have access to cpu time before the Load Average starts to reach 1.0, even if those processes are maxing out their individual cpus.
With more powerful cpus, the quicker a process will complete a task before the next task gets its time, so the “run queue” is emptied quicker, thereby keeping the Load Avg lower
Posted by Brian Blood as General, Hardware, Servers at 12:00 AM UTC
No Comments »
This is from an exchange on the OS X Server mailing list.
The OP was trying to get mcrypt compiled into his PHP5 install on a 64-bit machine running 10.5.4
On Jul 24, 2008, at 4:53 PM, Someone wrote:
> I’m trying to compile my own PHP (which I’ve done plenty of times
> in the past on other systems), but
> I need libmcrypt support. Because 10.5 OS X Server is all 64 bit, I
> need a 64bit php binary, which
> means all my linked libraries need to be 64 bit as well.
>
> I tried to compile libmcrypt by hand and with the SVN version of
> macports, but both versions give me
> the following error when I do `make test` after compiling php with
> them:
>
> dyld: Symbol not found: _arcfour_LTX__is_block_algorithm
> Referenced from: /usr/local/lib/libmcrypt.4.dylib
> Expected in: flat namespace
>
> Googling for this error just brings up a forum post on entropy.org,
> which isn’t much help. How can I get my 64bit php to play nice?
>
> I’m using CFLAGS=’-arch x86_64′ before all my ./configure
> statements to get a x86_64 build.
>
Jul 25 14:20:59 gerry org.apache.httpd[625]: httpd: Syntax error on line 1462 of /private/etc/apache2/httpd.conf: Syntax error on line 8 of /etc/apache2/sites/entropy-php.conf: Cannot load /usr/local/php5/libphp5.so into server: dlopen(/usr/local/php5/libphp5.so, 10): no suitable image found.
Did find:
/usr/local/php5/libphp5.so: no matching architecture in universal wrapper
—–
My response for solving the problem in a different way that I found on the Entropy.ch forums
—–
How about going the other way…..
I ran into a similar situation setting up an Intel Xserve with a new install of Leopard Server.
I was trying to get the entropy.ch PHP module (which contains mcrypt FYI) running and it would not load due to “no appropriate image found” errors.
Apache on Leopard is a fat binary and was trying to load as a 64-bit app, which of course means that all plug-ins, loadable libraries need to be 64-bit as well and the Entropy PHP doesn’t currently have a 64-bit x86 image.
Since this is for an Apache 2 web server and I would be using the pre-forked model and any single one of my requests that would be running PHP code would be very unlikely to need anywhere near more than even 1GB of RAM, I forced Apache to run as a 32-bit process so that the PHP module would load.
I did this by using the lipo command to thin out Apple’s apache2 binary (after making a backup of course) to just contain a 32-bit x86 image.
Here is a small shell script that will make a backup and run the lipo command:
#!/bin/sh -
#
#
cd /usr/sbin;
mv httpd httpd.ub;
lipo -info httpd.ub;
lipo -thin i386 httpd.ub -output httpd.i386;
lipo -info httpd.i386;
ln -s httpd.i386 httpd;
Posted by Brian Blood as OS X Server, Servers, Web App Development at 9:02 PM UTC
No Comments »
We had the occasion to setup shared calendaring for our office. Here is a quick to-do list:
Prerequisites:
- Server must be a full Open Directory system (not standalone)
Basic Steps:
- Setup DNS name for calendar: ical.networkjack.info
- Setup iCal Server on OS X Server, including setting up vhost for calendar
- Turn on Calendaring (in Advanced tab in Workgroup Manager) for all users who will be calendar
Client Setup (Mac OS X 10.5)
- Add server as a Directory using Directory Utility, so that you can search
- Add account for iCal Server within iCal (Prefs -> Accounts)
- username : password
- ical.networkjack.info:8008
- Setup delegation to other users (this is where the Directory comes in)
That should be it for the most part.
Posted by Brian Blood as OS X Server at 12:03 PM UTC
No Comments »
We have some devices in our server room in our office that due to the AC not running full blast on nights and Sundays, can get a little warm, even with spot cooling. To ensure better air flow, I’ve added a hi-tech piece of equipment to a PowerMac G4.
I normally would have added a PCI slot fan, but this server doesn’t have any available.
Posted by Brian Blood as Hardware, Servers at 10:52 AM UTC
No Comments »
