• 75.153.176.1  (ns1.dns.telus.com)
• 75.153.176.9 (ns2.dns.telus.com)

I think that either the Optik router or the TELUS DNS servers are a bottleneck here.  I guess  that for some reason, iTunes needs to do a lot of DNS lookups when it transfers data?

Anyways I changed my clients to use the following DNS servers:

• 8.8.8.8
• 8.8.4.4
• 208.67.222.222
• 208.67.220.220

The first two are provided by Google, the last two are provided by OpenDNS.  iTunes download speed went from ~200KB/sec to ~1.5MB/sec.  On Mac OS X, change the setting in System Preferences → Network → (Select your Network Connection) → Advanced → DNS.

A football match is time-bound.  This rightly requires that a football coach focus on winning the game in the required timeframe, not on beating the spread.  I can’t think of any type of business that’s similarly time-bound (in fact, I think a competent economist could argue that in an efficient market, all corporations are growth-bound).  In an environment that isn’t time-bound, it makes sense that the best metric of a corporation’s value is not its position or “score”, but the first and second derivatives of its position (that is, velocity and acceleration).  And isn’t this what market analysts are really getting at when they try to define expectations for a corporation?

Bonus amateur economist theorizing: there’s been a lot of talk lately about executive pay and the perverse incentives it creates.  For what it’s worth, I think the best rationalization of executive pay comes from tournament theory applied to the workplace.  This is probably best explained by Tim Harford in various places, including this YouTube video from the Logic of Life.  To summarize: lavish CEO pay is not only a reward for the CEO’s labour, it’s an incentive for his underlings to aspire to the top position.  This creates its own set of perverse incentives, prioritizing individual success over the success of the corporation.

I also added a Java implementation of Graham’s Scan to find the convex hull of a 2-D coordinate set.

Preparing for the Worst

First, establish a backup plan!  Once you register the device, you’ll have access to the HP WebOS site.  From there you can download WebOS Doctor, a Java app that will let you restore your TouchPad should something untoward happen to it.

I haven’t used WebOS Doctor yet and I hope I never have to.  But as a backup-backup plan, let’s make a copy of the filesystem.  Follow the instructions here to get terminal (root) access to the TouchPad. After logging in, I ran this command:

tar cvfz /media/internal/downloads/backup.tar.gz / --exclude '/media/internal/downloads/*

to backup the filesystem.  It’s far from a perfect backup, and it may never be useful, but I like having it around just in case.  Let me know if you need a copy, it’s about 300MB.

Setting up a Cross-Compiler

Wow, setting up a cross compiler has gotten a lot easier since the last time I checked.  All I had to do was download the ARM cross-compiler from here and install it.

The cross-compiler build tools are set up with weird names like “arm-none-linux-gnueabi-ar”.  There’s got to be an easier way to configure this but I just ran this command to set up shortened symlinks:

rweeks@faithless:/bitsafe/CodeSourcery/Sourcery_G++_Lite/bin$ for CMD in *; do CMD_TRIM=`echo $CMD | sed -e 's/arm-none-linux-gnueabi-//'`; ln -s $CMD $CMD_TRIM; done

Then you can control whether you’re using your native buildtools or the cross-compiler buildtools just by setting your $PATH.

Building OpenSSH

I figure a good first-day milestone is to get OpenSSH running on the TouchPad.  I’ll try to work my way up to a full-on Android distro .  I pretty much followed the instructions here with some minor changes for the cross-compiler.

First I defined a directory where I want to put all the build output:

mkdir /bitsafe/arm-openssh-server/output

The commands I used to build zLib:

./configure --prefix=/bitsafe/arm-openssh-server/output

make && make install

The commands I used to build openSSL (this will take a while):

./Configure --prefix=/bitsafe/arm-openssh-server/output linux-armv4

make && make install

The commands I used to build openSSH:

./configure --host=arm-none-linux-gnueabi --prefix=/bitsafe/arm-openssh-server/output --with-zlib=$PWD/../zlib-1.2.5 --with-ssl-dir=../openssl-1.0.0d

make && make install

The OpenSSH build will fail to install due to the cross-compiler (it can’t strip the output files).  But that should be OK.

Deploying OpenSSH to the TouchPad

OK, everything should be built at this point.  You can double-check that you’re using the cross-compiler by, eg.

find . -type f -print0 | xargs -0 file

Where you see executable files, it should indicate that they have been built for the ARM architecture.  I wrapped up everything in a tarball:

tar cvfz openssh-server.tar.gz arm-openssh-server/

This grabbed my “output” directory, as well as the source directories for zLib, openSSL and openSSH (in case any files failed to install correctly into output/)

I had a vastly complicated procedure in mind to copy the tarball to my TouchPad, but it turns out that the TouchPad just mounts as a vfat file system which was mounted automatically by my build machine.  The whole thing was just a drag-and-drop, which was nice.

Extract the tar file like so:

tar xvfz openssh-server.tar.gz --no-same-owner

You’ll get a bunch of errors on the extraction because it can’t create symbolic links: I think this is because you’re extracting to a filesystem that doesn’t support symlinks.  No big deal.

Test that the cross-compiler worked!

cd /media/internal/downloads/arm-openssh-server/zlib-1.2.5
./example
zlib version 1.2.5 = 0x1250, compile flags = 0x55
uncompress(): hello, hello!
gzread(): hello, hello!
gzgets() after gzseek:  hello!
inflate(): hello, hello!
large_inflate(): OK
after inflateSync(): hello, hello!
inflate with dictionary: hello, hello!

Starting OpenSSH Server

Harder than it sounds!  The root filesystem is mounted read-only by default, but you can hack your way around that with:

mount -w -o remount /

Then, add the sshd user and group to /etc/passwd and /etc/group

Setup an ssh key using ssh-keygen and set the “HostKey” property in sshd_config to point to the private key.

If you didn’t remount the root FS as read-write, you need to set UsePrivilegeSeparation to true.

Start sshd like so:

root@RussHPTouchPad:/media/internal/downloads/arm-openssh-server/openssh-5.8p2# $PWD/sshd -f sshd_config -ddd

(This is required because by setting up the “prefix” properties during the SSH build, all the commands will now be looking for their config in /bitsafe/…)

Last thing is to undo all the custom iptables config:

iptables -F

iptables -P INPUT ACCEPT

iptables -X ALLOWED_PACKETS

iptables -X ICMPFLOOD

iptables -X INVALID_PACKETS

OpenSSH 5.8 Running on TouchPad:

backfire:bin rweeks$ ssh -p 2222 root@192.168.10.128
The authenticity of host '[192.168.10.128]:2222 ([192.168.10.128]:2222)' can't be established.
RSA key fingerprint is de:c2:1f:6a:30:e9:33:cc:85:f6:28:07:62:f9:8b:9b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[192.168.10.128]:2222' (RSA) to the list of known hosts.
root@192.168.10.128's password:
lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory
lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory
debug1: permanently_set_uid: 0/0
Environment:
  USER=root
  LOGNAME=root
  HOME=/home/root
  PATH=/usr/bin:/bin:/usr/sbin:/sbin:/bitsafe/arm-openssh-server/output/bin
  MAIL=/var/mail/root
  SHELL=/bin/sh
  TZ=:/var/luna/preferences/localtime
  SSH_CLIENT=192.168.10.107 52206 2222
  SSH_CONNECTION=192.168.10.107 52206 192.168.10.128 2222
  SSH_TTY=/dev/pts/1
  TERM=xterm-color
root@RussHPTouchPad:/var/home/root#

This is the original paper describing the RTree and its implementation.  I’ve built a Java implementation based on it, here.

I used this RTree implementation to generate the simulated network topologies shown here.

By “pseudo-realistic” I mean that cities are much more likely to be adjacent to nearby cities, and the edges are weighted such that a shortest-path algorithm will prefer many short hops over a few long hops.  Specifically, the weight of an edge is n*log(n) where n is the great-circle distance between the two incident cities.

This algorithm tends to produce disconnected graphs.  It often doesn’t define any edges between the Americas and Africa or Europe.  Usually I just go and insert those edges manually.

Comments are extremely welcome re. how to make this simulated topology more realistic.  One thing I’m thinking of is defining a set of vertices as “aggregation nodes” and then auto-generating a certain number of devices adjacent to each aggregation node, where the number of aggregated devices per aggregation node is proportional to the population of the aggregation node city.

• network-16384.xml.gz
• network-8192.xml.gz
• network-4096.xml.gz
• network-2048.xml.gz
• network-1024.xml.gz
• network-512.xml.gz
• network-256.xml.gz
• network-128.xml.gz
• network-64.xml.gz
• network-32.xml.gz

network-16384

network-8192

network-4096

network-2048

network-1024

network-512

network-256

network-128

network-64

network-32

Firstly, a couple pics:

Shown above is the new, small NAS (“faithless”) and the old one (“backfire”).  The old one had a few disadvantages: it was huge, noisy and power-hungry.  For storage, I had 3x250GB SATA drives, each set up with EXT4.  Two of the drives were removable (one is shown partly removed in the photo).  In theory, one of the drives would be kept off-site.  I would rsync from client machines (mostly my laptop) to the fixed internal drive, and then rsync from there to the removable drive in the chassis.  Then, I would swap the removable drive with the off-site drive.

It’s a great theory but in practice I didn’t back-up to the file server as often as I should have and I practically never swapped the off-site drive.  Finally, I was worried (maybe unreasonably) about latent sector error with EXT4 due to its limited checksumming.

A Word About Latent Sector Error

So, the SATA spec tells us that we should expect an error every 10¹³ bits.  10¹³ works out to about 2⁴³ bits, or 2⁴⁰ bytes.  That’s 1TB¹.  It’s a little worrying to me that, if I fill up my 1TB drive, there’s a good chance of at least one error lurking somewhere.

EXT4 checksums its journal, which AFAIK will protect against errors caused by sync failures (ie. power failure during disk I/O).  But it’s not going to protect against latent sector errors.  To do that, you need checksumming on all the file data, along the lines of what ZFS or BTRFS provides.

Oh, ZFS, We Hardly Knew Ye

ZFS is an amazing filesystem.  I don’t think the benchmarks at Phoronix give it enough credit: ZFS’s default behaviour is to flush to persistent storage on every sync, which is the only way to guarantee data consistency in the absence of battery-backed on-disk cache.  Further, the one/two punch of the intent log and the adaptive read cache mean that, if you’ve set up your hybrid storage pools correctly, you can get some mind-boggling performance without sacrificing the fault-tolerance of your storage pool.

The problem with ZFS is that it runs best in Solaris.  I don’t like where Solaris is heading, and I have no reason to believe that it supports the hardware platform I’m using for the new NAS.  There have been a few attempts to port ZFS to Linux, but the most promising effort is still in its infancy.  So it’s not an option for this project.

BTRFS is a good option.  Its performance numbers are impressive and it offers the same RAID-0 and RAID-1 configurations as ZFS, plus redundant data checksums.  It’s got snapshot capabilities similar to ZFS.  I’ll miss the ZFS send/recv functionality, rsync will get me part of the way but it’s not a great solution.  I’m eagerly anticipating a proper fsck utility for BTRFS.

RAID-1 Fault Tolerance

If you look at the Wikipedia article on RAID, it claims that N-way mirroring (RAID-1) offers a Hamming distance of N.  This is a bit of an exaggeration, to me, because it assumes that checksumming is being done at a higher level.  What I mean is, if you have a 2-disk mirror and one of the disks has a latent sector error, to my knowledge there is no clear way to figure out which disk has gone bad, because you have no way of telling what the data should have looked like.  In an N-way mirror, your RAID controller (if it is clever) could probably find agreement between the other N-1 disks in the volume and reset the bad sector (isolate it, more likely), but even if that functionality exists I don’t think it would be present in a consumer grade $20 controller.  In my opinion, RAID-1 offers no fault tolerance at all because there is no checksum to verify against.  You effectively get a checksum when you go with RAID-5 or RAID-6, which is pretty much the route that ZFS/BTRFS has taken.

Enough With The Preamble, Already

There are 5 disks inside the NAS: 1x128GB OCZ Vertex SSD and 4x1TB WD Caviars.  The SSD is set up as the boot drive (Ubuntu 10.10); it also hosts all the application software.  As an aside, I will never again build a computer without an SSD.  The performance boost is incredible.  Boot time is sub-10-seconds.

The CPU is a dual-core Intel Atom D525 (1.8GHz).  I maxxed out the motherboard with 4GB of DDR3-1333.

The four 1TB drives are set up in a multi-device BTRFS filesystem, RAID-10 with redundant metadata.  I’ve disabled write caching (sudo hdparm -W 0 /dev/sd[b,c,d,e]).  With write caching enabled, I can do sequential writes up to 120MB/sec from SSD to the BTRFS filesystem.  With write caching enabled, that drops to a still-respectable 60MB/sec.  I expect that single-threaded sequential I/O, heavily biased towards writes, will be the primary use of this system.

I originally shared the filesystem through NFS but I’m accessing it mostly from OS X devices, and it seems like the OS X NFS client really, really sucks.  I couldn’t do better than about 10MB/sec over NFS and large sequential writes cause the client to become unresponsive for long periods of time.  Anyways, it’s still exported over NFS with the ‘sync’ option but I mostly access it via rsync+SSH.  In that mode of operation, I can get about 20MB/sec sequential write over my home network.

To solve the off-site backup problem, I think I’ll get a 1TB external drive and rsync from the BTRFS filesystem to the external drive.  eSATA would be really convenient here, but I’ll probably have to stick with USB due to hardware limitations.  ZFS send/receive would also be really useful to incrementally transfer the fileystem to an external disk.  The absence of this functionality in BTRFS means I’ll probably stick with rsync.

1: Useful to remember: 2¹⁰=1KB, 2²⁰=1MB, 2³⁰=1GB, 2⁴⁰=1TB.  If you’re going to complain about how 1GB=1000MB, please, don’t.  We’re approximating here.