So apparently YouTube, according to a Pakistani telecommunications authority, carries content that is “deemed offensive to Islam.

So the ISP either purposefully or accidentally, added custom routing configurations to it’s routers to block YouTube. The unfortuate side-effect was that these BGP announcements were propagated to a large part of the world’s routers, taking YouTube offline for a good number of people.

This fellow has a pretty good summation/chronology of what happened:

Renesys Blog: Pakistan hijacks YouTube

Posted by Brian Blood as Routers and Firewalls, Soap Box at 11:25 AM CST

No Comments »

Here is a list of the basics that every system administrator should implement:

1. Set your Reverse DNS. Don’t leave it empty.
2. Have geographically separated DNS servers
3. MTAs should have properly formed HELO names
4. rDNS should match the HELO on your MTA
5. HELO should resolve to your IP address
6. MX records must point to A records
7. Filter Bogons at the first opportunity in your network architecture and at appropriate routing points.
8. More to come

Posted by Brian Blood as General, Mail Server, Routers and Firewalls, Servers, Soap Box at 3:53 PM CST

No Comments »

(or, the device formerly known as the ArrowPoint Content Smart™ Web switch)

Back in the heady days of the dot com boom, one needed to be able to assure you could handle a large amount of traffic for all those visitors that you just knew were coming to your web property. In order to do that, your web application needed to be able to scale, which meant load balancing amongst any number of servers. ArrowPoint was the darling of this market, founded in 1997 and scooped up by the Cisco mother ship in May 2000 for a whopping $6.1 Billion in stock. (yes, that’s billion; you know, real money) This was 491.94 times their current revenue. Obviously, Cisco wanted into this market in a big way.

So, ArrowPoint had a nice lineup of Layer 7 switches both for the big guys (the CS-800) and the not-so big (the CS-100 line). (SlashDot used a CS-100 at one point and then upgraded to a CS-800 after a DDOS killed it.)

These devices primary purpose was to distribute IP packets amongst a server farm based on whatever criteria you could think of. You could make your content rules as simple (basic IP address/any port) or as complex as you wanted (testing for existence of a specific cookie or portion of a URL on a specific domain)

After Cisco bought them, the primary change they made was turn the case their standard blue. They also added a few models with GBICs and more buffer ram on each port (CSS11155). They even added a model that used a PCMCIA Flash Disk (e.g. CSS11154-FD-AC) instead of the 2 or 4GB IDE drives that are used to hold the OS, known as the NS and the configuration and logs.

We had been using one of those first Cisco based versions (locked flash vers 4.0, operational flash 4.0) for the sell.com server farm. We were about to deploy a new load balanced app for another client, so we decided to do some refurbishing of our supply of CSSen. I picked up a couple off eBay, one of the Cisco versions and one of the older ArrowPoint versions for under $500 and started tearing them apart. These devices when they were new went for upwards of $20,000!!!

Some interesting tidbits

• We’ve seen a decent percentage of power supplies in these units fail. I’ve actually purchased older basic 8 port units off eBay merely to have spare power supplies available.
• Upgrading the RAM to the full 256MB is easy, cheap, fast and gives the CSS plenty of room to breath for packet processing.
• The IDE drives in these devices are getting fairly long in the tooth.

RAM:

There are two slots on the CSS motherboard for installed RAM. The slots are underneath (if yours has one) the daughtercard that is used for the additional ports (2 GBIC, 4 x 100FX, 4 x 100BT), so in order to upgrade the ram, you will need to pull this card out temporarily. It’s got about 3 screws and comes out without too much fuss.

The chip to use is a Micron 128MB DDR 100 MHz MT8LSDT1664HG-10EB1, so with 2 of these the CSS will have 256MB. I picked up a couple for sub $15 each.

Disk system:

The IDE drive in the non FD (Flash Disk) models is usually a Fujitsu 2, 4 or 6GB drive. The NS and logs take up a very small part of the space on these disks, so we decided to replace the only non-solid state part of the CSS (not counting the fans) with some newer, more reliable technology. I found a CompactFlash to IDE adapter for sub $20 and a 2GB CompactFlash card for about $60. I did some research into the long-term reliability and durability of CompactFlash. There are industrial-strength CF cards, but they are about 5-10 times as expensive. The major technological consideration of CF cards is the use of single-cell vs multi-cell memory. For long-term reliability, you want single-cell as the electronics on the card will actually monitor the health and adjust the storage of data within the cells as it finds problems and single-cell CF is also rated for a higher number of writes and has a higher MTBF. Good explanation here: DailyTech - Solid-state Drives Ready for Prime Time

So, with a 2GB Kingston Elite Pro “disk” installed, we merely use the Offline Diagnostic Menu accessible from the console port to format the new disk and use the boot from FTP function to pull down an updated NS (an ADI or ArrowPoint Distribution Image) onto the disk and it’s ready to start configuring.

The FD model of CSS comes with a PCMCIA to IDE sled in the place of the hard drive. Inserted into that slot is a 350MB SanDisk PCMCIA flash card. We’ve purchased the 1.2GB version of these cards and done the same process as above. Flash goodness all around.

One interesting note, I expected the see some decent amount of savings in amps when replacing an actual hard disk drive with a flash drive, but curiously, I didn’t. The device pulled about 0.92 amps (110V) with the hard drive and only went down to 0.85A with the flash drive. It’s interesting that a device of this type pulls so much current in the first place. Most of the switches we utilize typically draw in the 0.3A range or less. I guess that could be related to why we see a higher failure rate with the power supplies.

Summary

In the end, we ended up with some new/spare load balancers that have been cleaned up, upgraded and made more reliable. Not bad for a couple hundred dollars spent.

Posted by Brian Blood as Content Networking, Hardware, Routers and Firewalls at 5:56 PM CDT

No Comments »

Datacenter Power. It seems you can never have enough.

We have our colocation inside an Equinix IBX. It is an excellent facility. Unfortunately, about 2 years ago, our cage got a new neighbor. They have added rack after rack of new servers to accommodate their ever increasing traffic. Which means they have effectively used up all the allocated power feeds for our section of the colo.
So as we started to fill our own cabinets, we found that we were quickly using up the 2 x 20A 110V feeds they had allocated to each of our cabinets. Our partner in colocation, sell.com was also at this time upgrading their farm to the latest dual xeon models. These boxes were pulling a LOT more amps than the previous P3 generation.

Very quickly, we became experts on how much amperage we could squeeze out of our existing feeds and what systems required how much power.

Here are some anecdotal amperage readings we took from our fancy amp reading tool.

Dell PowerEdge 2850

Specs: Dual Xeon 3.6GHz/1MB; 6 x 73 GB SCSI Hard Drive (10K RPM); Dual Power supplies

• PS A & B both active
  • PS A - 1.15A
  • PS A & B - 2.35A
• PS A only - 2.30A

Dell PowerEdge 1650

Specs: Dual PIII 1.4Ghz; 2GB RAM; 3 x 36GB SCSI 10K rpm; Dual 275W Power supplies

• PS A & B both active
  • PS A - 0.7A
  • PS A & B
    • Nominal operation - 1.41A
    • Warm Boot - 1.44A Peak
    • Cold Boot (drives spinning up) - 1.56A
• PS A only
  • Nominal operation - 1.37A

Apple Power Mac G4

Specs: G4/533 Dual - 1.5GB RAM - 2 x 18GB SCSI (15K rpm)

• Peak Startup - 1.27A
• Max load on SCSI drives - big copy operation - 1.18A

Apple Xserve G4

Specs: Dual 1.0 Ghz G4, 2GB RAM 2×60GB & 2 x 180GB

• heavy cpu/disk load - 1.52A
  • simultaneous diskutil zero on all disks (booted from CD)
  • Max CPU - multiple threads of cat /dev/urandom > /dev/null & ssh/rsa keygen operations
• all 4 disks idle - 1.37A
• Insert 180GB ADM - peak 1.41A, settled back down to 1.32A
• Insert second 180GB ADM - peak 1.48A, settled down to 1.38A
• keygen and cat large data file generated by /dev/urandom, copied to Software RAID mirror 60GB - spikes to 1.56A

Apple Xserve G5

Specs: Dual 2.0Ghz G5, 3GB RAM, 3 x 80GB SATA

• Nominal operation - 1.8A
• Max Cold Boot - 2.16A

Apple Mac Mini

Specs: Intel 1.66Ghz Core Duo, 2GB RAM, 60GB E-Rated Hitachi drive E7K100 model

• Nominal operation - 0.29A
• Max cpu, disk activity - 0.37A

IBM 4000R

Specs: Dual 833Mhz PIII - Single Power supply - 2 x 18GB SCSI (10K rpm)

• Cold Boot (drives spinning up) - 1.0A
• heavy cpu/disk load - multiple instances of cpuburn and cat’ing /dev/urandom to a file - 0.9A
• Nominal operation - 0.75A max

IBM eServer x330

Specs: Two Intel Pentium III (Coppermine) 864MHz processors, 1GB RAM, Single Power Supply, Single 36GB SCSI drive

• Connecting Power Peak: 0.29A
• Stdby Steady: 0.11A
• Power On Peak: 0.78A
• SCSI spinup: 0.98A
• Powered low load: 0.63A
• Loaded (6.0+ Load Average with disk): 0.80A
• Disk activity only: 0.72 peakA
• Reasonable Load + Disk Activity: 0.79A
• heavy cpu/disk load - multiple instances of cpuburn and cat’ing /dev/urandom to a file - 0.82A

IBM eServer x336

Specs: Dual 3.0Ghz Xeon, 4GB RAM, Dual 575W PowerSupplies, Dual 146GB SCSI drives

• Connecting Power Peak: 1.06A
• Stdby Steady: 0.79A
• Power On Peak: 2.5A
• Powered low load: 2.12A
• Loaded (7.0+ with disk): 3.25A
• Disk activity only: 2.40A
• Reasonable Load + Disk Activity: 2.85A peak
• heavy cpu/disk load - multiple instances of cpuburn and cat’ing /dev/urandom to a file - 3.2A

Cisco 11151 Load Balancing switch - 0.89A

Cisco 3548XL Switch - 0.32A

Dave from NetApp has some interesting things to say about power in the datacenter.

Posted by Brian Blood as Colocation, Routers and Firewalls, Servers at 6:51 PM CST

No Comments »

I’ve always been a fan of Netopia.

I’ve used Timbuktu remote control software since forever (1990) and we still use it today on our servers even with the availability of VNC and Apple Remote Desktop (I repeat myself).

Back in the early 90’s when ISDN connections were all the rage, at a previous employer, we used Netopia ISDN 440 Internet Routers extensively. We interconnected several branch offices in Texas to create an AppleTalk WAN using AURP (AppleTalk Update-base Routing Protocol). We had a hub and spoke system to create our own “VPN tunnels”, so the corporate office in Dallas could print to a Laserwriter printer in a branch office in Houston. It was cool stuff that was easy to setup. We also used those AppleTalk tunnels to interconnect first our QuickMail Servers using the AppleTalk ADSP File Transfer plugin for the Communnications Toolbox, then our AppleShare IP 5.0 mail servers taking an inbound SMTP over IP from our ISP connection then to an SMTP over AppleTalk to the specific branch office for that employee. One of the cooler features of that mail server and oh the heady days of the AppleTalk Network Browser.

So when Netopia came out with the R-series of routers for T1 and DSL deployments in the late 90’s, they were the old reliable friend updated for the latest technology. They were a bit outdated in their typical configuration with ony a 10mbit WAN port and an 8-port hub, but we usually downlinked that to a faster switch with more ports anyway and didn’t really need anything faster than 10 mbits total throughput anyway.

A year or two ago, when we needed to update two of our site to site VPN tunnels and support roadwarrior access from home to office and colo networks, we thought we would give them a try. We tried a R9100 at our office connected to an R9100 at the colo and while it was easy to setup, unfortunately, it was dog-slow.

We’ve since switched to using NetScreens, but since they have at LEAST one million different options, we’ve only really gotten the site to site vpn setup working and we really miss the very easy PPTP (especially for Mac OS X clients) VPN connections that the Netopia OS/firmware provides.

So when a customer recently asked about setting up VPN access into the backside network of their colo systems, I went back and looked at what the Netopia line had to offer. What I found was the 3386-ENT. Which turned out to be JUST what I was looking for: A 10/100 WAN port, 4 port 10/100 switch for the LAN, smaller form factor (which is important in a colo cabinet) and still the very easy to use Netopia management interface. These devices were introduced back in March 2003, so they’ve had a good amount of time to update the firmware and make them stable systems. And firmware updates are always free. Which is nice.

I was able to pick one up off eBay for $75 shipped and when it arrived (and got a correct power supply as the one sent to me was for a mobile phone or something - ALWAYS label your power supply with what it goes to) I was pleasantly surprised at the feature set of this new device.

A word about the 3300 line; Back in September 2001 purchased Cayman systems and these devices are actually borne of that line of Cayman dsl modems/routers. So to directly compare them and their performance with the R-series isn’t exactly right on, but I’m going to choose to beieve it’s close enough.

So, it turns out that based on specs alone, this little box should kick butt. The processor families are probably not the same (see above para), but the R9100 has a 33mhz cpu with 4MB ram and the 3386-ENT has a 168mhz cpu with 8MB ram. I’m hopeful that translates into much better performance.

It turns out to be true. I get the router installed and configured for VPN access, then connect to the box from my PowerBook G4 over my FIOS line at home. It took about 5 minutes to set that up. 5 mintes. It took me more than 5 minutes to just to get the right docs open for the NetScreen. Once connected, I was able to easily connect to ip addresses of devices on that backside of the network and work just like I was there connected right up to the LAN. Exactly like a VPN is meant to work.

Performance testing

I’ve found that the best way to easily test the speed of a link is run a Timbuktu (more Netopia!!!) file transfer. It’s always been the most efficient and also shows you nice data rate numbers on how fast things are going. My FIOS line at the house is a 15mbit down/2mbit up connection, so the best thing to try to max out the system would be to pull a file off a box through it’s backside IP.

Trying that I got about 3.5mbit throughput. Pushing a file up to the server, I got about 210K which is essentially maxing out the upstream speed of my FIOS. Still 3.5mbit is pretty darn good. And I was ssh’ed into another box at the same time and didn’t notice any delays or higher latencies in typing or output from top.

Before I did all this, I setup our cacti system to poll the 3386 for both the throughput on the WAN port and the current cpu usage.

So you don’t have to go searching yourself, the SNMP OID for current CPU usage on any Netopia router is: .1.3.6.1.4.1.304.1.3.1.3.1.0

When I was transferring the big files to/from the servers, apparently the 3386 was haulin the chili:

so, not a high-end NetScreen 208 or Cisco VPN concentrator, but a very good performer for a sub-$100 router that is easy to setup.

Other notes

Looking through the screens on this box shows a surprising amount of breadth in feature-sets for such a small device. (make sure you update to the latest firmware)

• VRRP for handling redundant gateways
• QoS - called Prioritize Delay Sensitive data in the interface. Either on or off, but still a nice touch in being able to make sure your VoIP traffic doesn’t get the jitters.
• NTP client settings and update frequency
• WAN link rate limit, in case for some strange reason (which I have encountered), you want to go slower.
• Stateful Inspection of packets to ward off certain types of IP attacks.
• Local NAT (called IP passthrough). allows you to use the WAN IP of the device inside the LAN. Useful for deploying servers internally and providing external access without having to reconfigure hostnames/IP addresses in applications.
• Diffserv for creating specific QoS rules.
• VLAN support for LAN side
• MultiNAT for when your ISP gives you a static 8-IP subnet and you want to implement some interesting routing.
• syslog support

Again, these are not all deeply configurable like a Cisco or a NetScreen, but having the basic features of some of these technologies can sometimes really save your bacon when a client wants something a bit unorthodox.

Overall, I’m pretty happy with the device and I’m waiting on another to arrive from an eBay seller. I hope this is the start of another beautiful friendship with our friends from Netopia (I mean Cayman, wait, Netopia. Right?)

Posted by Brian Blood as Routers and Firewalls at 12:38 AM CDT

No Comments »