Halvar just posted a very good hypothesis, but one that does not include the fact that a forged request can also include a glue record that updates the cache on the attacked server, with a very long TTL. What this gibberish means is that you can attack a DNS in ten seconds and gain control of all queries sent to that DNS for all subdomains of any second-level domain like google.com. You “hijack” xkdjlsfjls.google.com and you have in effect hijacked the entirety of google.com for all users of the target DNS.

I’d post the Matasano article — which has a very good explanation — but I’d rather not worry about copyright infringement lawsuits.

I’m using Kubuntu Hardy, and my computer has two 400 GB SATA hard disks. Yes, that’s all the storage I have at hand; as of three days ago, it was RAIDed using the multipath devices (md) kernel module, split in two LVM volumes: /and /home. Oh, and two same-size byte-aligned swap partitions, one on each disk, swapon‘ed pri=0.

I had been salivating over the thought of using ZFS in my workstation because of several killer features:

• The first one that comes to mind is end-to-end data integrity thanks to checksumming — I’ve already had many disks go bad on me, while others corrupted my data silently (which is, believe it or not, the most insidious thing ever, because after you’ve noticed it, backups won’t help you with that — you’ve probably already papered over your backups with new, bad data).
• The second one is compression. Together with tightly packed data, compression promises to increase performance and reduce disk utilization.
• The third one is the advanced transactional algorithm that yields an always-consistent disk structure. Unlike log-based filesystems, ZFS does copy-on-write and ripples the changes up through the filesystem tree; before the topmost node is updated, the changes don’t affect consistency; when the topmost node is updated, the disk is consistent as well. Never fsck again!

“Damn, gotta get me some of that, I thought”

Getting ZFS was actually a piece of cake: I went to the Mercurial repository for the project, selected the tip view, and downloaded a nice tarball. I then installed a couple of dependencies according to the README, and hit scons in a terminal window. Five commands were built:

• zfs-fuse, the daemon that serves FUSE requests. The FUSE module is an odd beast: applications futzing with a FUSE-mounted filesystem talk to the kernel VFS, which talks to FUSE, which talks to the daemon backing that particular mount. This userspace-kernelspace-userspace-kernelspace–userspace overhead, you will see, is a big deal.
• zfs and zpool, the main management commands that use IPC to talk to zfs-fuse.
• two others that you won’t care and I won’t care either.

A cursory inspection with such important system binaries was in order, so I ldd the daemon and the commands. zfs-fuse links to /usr/lib/libz*.so*. Not good, chicken and egg problem, linking to a library in a filesystem that will not be available before zfs-fuse is running? I rebuild it using a modified SConstruct file so it statically links that library in.

I had decided that my filesystem layout would be:

• 1 GB swap partition on each disk
• 1 GB / filesystem, composed of two RAID1 partitions (one on each disk), formatted with ext3 (in case of catastrophe, it’s nice to have something the kernel can boot without initial RAM disks)
• 398 GB ZFS volume, where I planned to drop /usr, /home and /var

But I didn’t have extra hard disks to make the switch. No problem, croupier, everything I have on red please — and spin that wheel! I installed ZFS directly on my running system. How did I do it? Well, if you must know:

• I offlined the second disk with mdadm.
• I swapoff‘ed its swap partition. At this point the disk is no longer busy.
• I repartitioned the disk (if the disk is non-busy, the kernel rereads the partition table just fine).
• Then I relied on the first all the time.

Yes, realtime no-boot filesystem switchover — or at least I thought it would be that easy (I was very wrong).

Then I mkfs.ext3ed the new 1 GB root filesystem, and mkswap‘ed the swap one. A couple of rsyncs later (which I scripted for consistency and repeatability), I had a new, working /. I mounted it and went in it, to remove mdadm.conf and lvm.conf lines that could prove problematic on next boot. At this point I was panicking because it was superstitiously conceivable that, after a reboot, md would want to rebuild the arrays and destroy the second disk.

I then copied the ZFS binaries in /sbin and ran it. A cursory lsof inspection told me that the ZFS socket was on /etc/zfs/zfs_socket. zpool create quickly gave me the 392 GB of disk space that were previously empty in the second disk, in which I created subvolumes, with adjusted mount points to end up under a temporary tree structure under /newfs. Curiously, after creating a subvolume, it’s not mounted, but a zfs mount -a works as you probably would expect.

I enabled compression in the root volume (subvolumes inherit attributes) and started rsyncing /var, /usr and /home into each subvolume. Cue the movie 32 hours later to have an idea of how slow it was. It was unbelievably slow — un-frigging-believable, with both CPUs nearly pegged and regularly hovering at 150% combined user+system. The worst part is, I was seeing disk throughput in the 2-3 MB/s range, using iostat 1 and zpool iostat 1. Keep in mind that performance (high write throughput, low responsiveness/latency during massive reads) is marketed as a ZFS selling point — and I don’t doubt the Sun guys… on Solaris, not Linux!

During that lengthy process I started finding out several things that would prove crucial later on:

• FUSE does not support mmap in the Linux kernel that my distribution uses. Many, many applications rely on that feature to work.
• There was no initscript for ZFS. I would have to write an initscript from scratch. On Kubuntu, where initscripts are being (1) phased out and (2) completely different to my beloved RPM distros.

At this point I was a bit nervous, if you’ll allow me to understate. But I wrote the initscript anyway:

#! /bin/sh

BEGIN INIT INFO

Provides: zfs

Required-Start: mountall

Required-Stop: sendsigs

Should-Start:

Should-Stop:

Default-Start:

Default-Stop:

Short-Description: Enable/disable the ZFS-FUSE subsystem

Description: Control ZFS-FUSE subsystem

END INIT INFO

PIDFILE=/var/run/zfs-fuse.pid LOCKFILE=/var/lock/zfs/zfs_lock

. /lib/init/vars.sh

. /lib/lsb/init-functions . /lib/init/mount-functions.sh

export PATH=/sbin:/bin unset LANG ulimit -v unlimited

do_start() { test -x /sbin/zfs-fuse || exit 0 PID=cat "$PIDFILE" 2> /dev/null if [ "$PID" != "" ] then if kill -0 $PID 2> /dev/null then echo "ZFS-FUSE is already running" exit 3 else # pid file is stale, we clean up shit log_action_begin_msg "Cleaning up stale ZFS-FUSE PID files" rm -f /var/run/sendsigs.omit.d/zfs-fuse "$PIDFILE" log_action_end_msg 0 fi fi

pre_mountall

log_action_begin_msg "Starting ZFS-FUSE process"
zfs-fuse -p "$PIDFILE"
ES_TO_REPORT=$?
if [ 0 = "$ES_TO_REPORT" ]
then
    true
else
    log_action_end_msg 1 "code $ES_TO_REPORT"
    post_mountall
    exit 3
fi

for a in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
do
    PID=`cat "$PIDFILE"`
     [ "$PID" != "" ] && break
    sleep 1
done

if [ "$PID" = "" ]
then
    log_action_end_msg 1 "ZFS-FUSE did not start or create $PIDFILE"
    post_mountall
    exit 3
else
    log_action_end_msg 0
fi

log_action_begin_msg "Immunizing ZFS-FUSE against OOM kills and sendsigs signals"
mkdir -p /var/run/sendsigs.omit.d
cp "$PIDFILE" /var/run/sendsigs.omit.d/zfs-fuse
echo -17 > "/proc/$PID/oom_adj"
ES_TO_REPORT=$?
if [ 0 = "$ES_TO_REPORT" ]
then
    log_action_end_msg 0
else
    log_action_end_msg 1 "code $ES_TO_REPORT"
    post_mountall
    exit 3
fi

log_action_begin_msg "Mounting ZFS filesystems"

zfs mount -a
ES_TO_REPORT=$?
if [ 0 = "$ES_TO_REPORT" ]
then
    log_action_end_msg 0
else
    log_action_end_msg 1 "code $ES_TO_REPORT"
    post_mountall
    exit 3
fi

if [ -x /usr/bin/renice ] ; then
    log_action_begin_msg "Increasing ZFS-FUSE priority"
    /usr/bin/renice -15 -g $PID > /dev/null
    ES_TO_REPORT=$?
    if [ 0 = "$ES_TO_REPORT" ]
    then
        log_action_end_msg 0
    else
        log_action_end_msg 1 "code $ES_TO_REPORT"
        post_mountall
        exit 3
    fi
    true
fi

post_mountall

}

do_stop () { test -x /sbin/zfs-fuse || exit 0 PID=cat "$PIDFILE" 2> /dev/null if [ "$PID" = "" ] ; then # no pid file, we exit exit 0 elif kill -0 $PID 2> /dev/null; then # pid file and killable, we continue true else # pid file is stale, we clean up shit log_action_begin_msg “Cleaning up stale ZFS-FUSE PID files” rm -f /var/run/sendsigs.omit.d/zfs-fuse “$PIDFILE” log_action_end_msg 0 exit 0 fi

pre_mountall

log_action_begin_msg "Syncing disks"
sync
log_action_end_msg 0

log_action_begin_msg "Unmounting ZFS filesystems"
zfs unmount -a
ES_TO_REPORT=$?
if [ 0 = "$ES_TO_REPORT" ]
then
    log_action_end_msg 0
else
    log_action_end_msg 1 "code $ES_TO_REPORT"
    post_mountall
    exit 3
fi

post_mountall # restore /var/lock and /var/run to their right places

log_action_begin_msg "Terminating ZFS-FUSE process gracefully"
kill -TERM $PID

for a in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
do
    kill -0 $PID 2> /dev/null
    [ "$?" != "0" ] && break
    sleep 1
done

if kill -0 $PID 2> /dev/null
then
    log_action_end_msg 1 "ZFS-FUSE refused to die after 15 seconds"
    post_mountall
    exit 3
else
    rm -f /var/run/sendsigs.omit.d/zfs-fuse "$PIDFILE"
    log_action_end_msg 0
fi

log_action_begin_msg "Syncing disks again"
sync
log_action_end_msg 0

}

case “$1″ in start) do_start ;; stop) do_stop ;; status) PID=cat "$PIDFILE" 2> /dev/null if [ "$PID" = "" ] ; then echo “ZFS-FUSE is not running” exit 3 else if kill -0 $PID then echo “ZFS-FUSE is running, pid $PID” zpool status exit 0 else echo “ZFS-FUSE died, PID files stale” exit 3 fi fi ;; restart|reload|force-reload) echo “Error: argument ‘$1′ not supported” >&2 exit 3 ;; *) echo “Usage: $0 start|stop|status” >&2 exit 3 ;; esac

:

The script should explain itself.

There were two problems, though. I derived my script from the NFS one and, in the process, I discovered that NFS was symlinked to be started at slot 31 in level 6 and 0. This means that the initscripts subsystem would call that script with a start argument when in reality, the action was in the stop block. Since I couldn’t figure out what kind of magic the Upstart initscripts compatibility subsystem does to get a stop block to run when a start block is requested by its configuration, I just created two glue scripts: one to start ZFS no matter what, and one to stop ZFS no matter what:

-rwxr-xr-x 1 root root 481 2008-06-18 04:09 /etc/init.d/mountzfs
-rwxr-xr-x 1 root root 488 2008-06-18 04:09 /etc/init.d/umountzfs

Then I studied the Kubuntu boot sequence very carefully, and used some elbow grease (update-rc.d) to symlink them to get the results I wanted:

lrwxrwxrwx 1 root root 19 2008-06-18 03:52 /etc/rc0.d/S35umountzfs -> ../init.d/umountzfs
lrwxrwxrwx 1 root root 19 2008-06-18 03:52 /etc/rc6.d/S35umountzfs -> ../init.d/umountzfs
lrwxrwxrwx 1 root root 18 2008-06-18 03:52 /etc/rcS.d/S36mountzfs -> ../init.d/mountzfs

Trust me, writing the script was the easy part — figuring out how it interacts with the rest of the system was much harder.

Finally, I rebooted to my new root filesystem on the second disk. If you thought that my system booted correctly, you would be very, very wrong indeed. Eighty percent of the boot sequence were red [ fail ]s and sh: command not found errors. At the end, the system dropped me into a recovery console, where I could finally switch the ZFS mount points to their final destinations. Then, just to try out: zfs mount -a.

/homemounted. /var couldn’t be mounted, because the boot process graciously created incredibly important missing directories in it. And then, deadlock.

Crap, what was wrong?

Alt+SysRq+R. Boot again. What’s wrong? No idea. Try strace. The friggin’ command is in /usr. Hypotheses ran through my head for two hours. With me in front of a very, very broken system. I tried everything under the Sun that I could get my hands on — which is not much when you don’t have a CD-ROM drive, mind you.

And (summarizing two hours) then, I tried this: zfs set mountpoint=/tmp/usr vault/usr ; mkdir -p /tmp/usr ; zfs mount vault/usr.

Miracle of miracles, it worked. I copied the entire cast of characters of Linux Debugging: The Movie into the very tightly packed /. I strace --ffed the hell out of zfs-fuse and I found the problem. The moronic mount.fuse subcommand, that actually connects the kernel and user endpoints, tries to read /usr/lib/locale/locale-archive right in the middle of mounting the filesystem! Instant deadlock that you can only get out of by using the SysRq OOM key (yes, zfs-fuse is actually a great OOM candidate — 1.5 GB VM size on this 1.0 GB RAM computer; yes, I discovered that on my own before I wrote the OOM immunization code in the initscript).

I then discovered two things: zfs-fuse didn’t deadlock when started from the recovery command, but it did lockup when starting it from the initscript. What you can’t see is that the version of the initscript that I initially wrote was sourcing the LANG variable from a configuration script in /etc. OK, so how do you solve locale problems? Instant fixup: unset LANG before running the command.

OK, so do I have a booting system now, or what? Wrong again. Some processes get started before the actual mounting of filesystems, and the ZFS subsystem can’t actually be started earlier in the boot process without creating an initramfs dependency or another, different, chicken-and-egg problem. So I moved what I could move from the ZFS volume’s /var into the /var directory of the / filesystem. I ended up with this structure backed up by ZFS (and the rest, you can safely assume, in a very tightly crammed ext3 filesystem):

zfs list
NAME              USED  AVAIL  REFER  MOUNTPOINT
vault             294G  69,8G    18K  none
vault/home        290G  69,8G   290G  /home
vault/usr        3,36G  69,8G  3,36G  /usr
vault/var         842M  69,8G    18K  none
vault/var/cache   515M  69,8G   515M  /var/cache
vault/var/lib     282M  69,8G   282M  /var/lib
vault/var/tmp    44,5M  69,8G  44,5M  /var/tmp

Boot again. Oh, yeah, I’m enjoying the 3-minute boot time on this formerly-a-screamer machine. D-Bus fails to start. D-Bus is actually very required for many things in Kubuntu, but I manage to start a GUI session up, if only to Google up what was wrong with it. That was probably not the best moment to find out that just starting the KDE 3.5 session took over ten minutes. All of this with less than 1 MB/s from the disk, according to iostat and 160% CPU usage, according to top.

Then I discovered the zfs-fuse Google group. It’s a fantastic place where everyone (including Ricardo Correia) received me very well and had lots of tips. Only there did I find out what was wrong with D-Bus — a bug that manifests itself only with FUSE filesystems, for which a patch exists and works.

At this point I’m extremely exhausted from this marathon session, so I basically just try to backport the patch into the dbus source package for my distribution. You’ve probably heard that Debian (and, by extension, Ubuntu) has a fantastic build system — it failed on me. Not only was apt not working (remember the mmap issue?), but dpkg-source also failed while trying to apply the patches for the source package. Oh, yes, I manage to solve this problem by learning, on-the-spot, how the apt build “system” actually works, and manually replicating the entire process that should be automated. Many thanks to the gents at #debian in Freenode for their kind responses to my questions.

Bam, built dbus (it’s yours if you want it). Installed it. Started it. And the chain of daemons that were depending on it just start up and take life. Neat trick, Upstart!

Back to performance questions and ZFS. Do you know what the real performance killer is? You’ll never guess it…

…icons! While GTK+ applications take marginally more time to start under a ZFS regime, KDE applications take an order of magnitude more. Before, on a warm working set, a KDE application took about 2 seconds to start. Today, Kmail takes in excess of five minutes to start. Why? Here’s why — multiply that by fifty thousand and you’ll get the idea. Each icon that the application requests results in thousands and thousands of access() and stat() calls. FUSE doesn’t use a kernel cache by default (there are several reasons for that), so the only cache that backs those requests up is the ARC cache, which is an impressive caching regime and technical achievement but, in this case, it’s very much like caching your car keys somewhere in Europe, because of the transatlantic userspace-kernelspace-userspace-kernelspace-userspace barrier. Per-call. When this is taking place, the CPUs remain pegged at 190%, eaten by ZFS alive, and the 12 case fans jump to 11.000 RPM.

The zfs-fuse Google groups guys came up with a couple of suggestions (all documented in the list, which I’m too lazy to link to again). These all are compile-time options, so a ZFS rebuild is in order for every one of them:

• scons debug=0. A very slight CPU usage decrease.
• Increasing the ARC cache. I doubled it from 128 to 256 MB. Turns out it’s not a caching problem and it doesn’t help at all.
• Mount option big_writes for FUSE filesystems. Here’s what I did about that:

Recompiled ZFS, this time enabling a FUSE mount option named big_writes that I’ve read about in the Google group. Yes, the daemon needs to be recompiled, and it’s not fast. No, I’m not actually jumping to the part where I actually compiled ZFS with big_writes first, then booted, only to find out that I needed a new kernel. Oh, wait, I just did. Fortunately, I did back zfs-fuse up.

Next up? Latest 2.6.26-rc6 kernel, because of:

• Hey, writable mmap is there for FUSE filesystems! Yeah! Now I can have apt-get back!
• big_writes.

When was the last time a kernel compile took four hours for you? Mine was yesterday. But it’s actually fun — the process hasn’t changed that much from 1998, and the distro already comes with a nice .config that you can reuse with make oldconfig. And, this time, you get to do out-of-tree kernel builds! Yay!

Well, I ticked the wrong option in make menuconfig anyway, because my kernel modules don’t fit my puny /, now at 400 MB free. Jeez, four hours. Google some more. Turns out I turned a debugging option on.

After this, FUSE userspace itself was due for a recompile. Another odyssey, whose fruits you can reap here (warning: CVS checkout).

OK, redo the initial RAM disk, adjust GRUB configuration, reboot with the latest kernel. It’s all good. More surprisingly, I’m actually getting some of my performance back. Some of it. As in “Kmail no longer takes five minutes to start — only three”.

And, most importantly, applications that depend on mmap now work correctly. My boot process isn’t an epic [ fail ] anymore — and that’s incredibly reassuring.

This is the point where my journey turns into smooth sailing. I zpool scrubed my new baby. After five hours, with the solid guarantee that my data was OK and nothing’d been lost or corrupted during the rsync, I nuked my first disk, replicated the new partition structure on it. A nice RAID1 array for the final /. A short rsync for the / filesystem. A quick mkswap for the new swap partition. A fast adjustment in /etc/fstab and another one in mdadm.conf for the new array. Reinstall and reconfigure GRUB on the first disk. And, finally, I leave the best for the latest:

zpool attach vault /dev/by-id/second-disk-huge-partition /dev/by-id/first-disk-huge-partition

Man, that rocked. It was unbelievably fast — like, disk-platter fast, around 40 to 50 MB per second, and the system didn’t get that much more slow when it was resilvering the first disk. Which kind of makes lots of sense, because zfs-fuse is now crossing the userspace-kernelspace barrier just once per operation. How do I know this? Well, strace: I know that what zfs-fuse does is, it opens the disk partition in direct I/O mode and then manages it for itself, responding to FUSE requests — but the resilvering process doesn’t involve FUSE at all, it’s just the two disks practically chatting with each other through zfs-fuse. Now I know for sure that ZFS will give me platter speeds. It’s just a matter of time (and maybe me pestering Ricardo Correia to collaborate with me on this same issue).

Questions that I haven’t solved yet? Sure, there are a lot. Two that haunt me:

• No root filesystem on ZFS. Others on the Google group have managed it. Me? I didn’t want to mess with /etc/zfs inside the initramfs, thank you very much.
• I know this for sure: the only active cache now is the userspace ARC cache from ZFS; I read the FUSE kernel code, and it clearly flushes files from the cache when programs open() them. Honestly, if I could wish for something to just become true overnight, I’d wish for the ARC to be moved into the kernel and to have it replace the page cache, but that won’t happen anytime soon. There’s a FUSE kernel_cache option, but I’m wary of enabling it. When I have been sufficiently reassured that the option won’t corrupt my precious data, I will enable it. That will be a couple of hours of reading someone else’s code, so I’m inclined to defer it for a few days. But, in theory, this should give me platter speeds instead of giving my 12 case fans ’speed’. At the hefty cost of RAM for two redundant caches.
• Do filesystem readahead and Linux disk scheduler algorithms mess up in some way with ZFS’ control of the platter? The data integrity question is closed, because the writes are submitted with barriers, but I’m worried that the Linux I/O scheduler is second-guessing the decisions of ZFS’ one.
• The /etc/init.d/sendsigs omit.d protocol I’m using on the initscript plain fails. I had to shunt the script with an exit 0 right before the killall5 in sendsigs because killall5 plain hung instead of ignoring ZFS as it should have done — and it needs to ignore ZFS because ZFS is unmounted later. This won’t be a problem once we get our own kernelspace ZFS implementation.

OK, that was my journey. I’m on ZFS now, my machine’s rock-solid (if a bit CPU-tired) and my data’s never been so safe. I also got compression, which saved me about 6 GB. Furthermore, I’ve given you the initscript, the steps and the software (except ZFS, but you can compile that yourself).

Go wild.

Following these paragraphs, two images will show up, one below the other. If you pasted a 5″ by 5″ square cardboard hole on the screen of a 24-inch monitors/TVs, this is the difference you would see between an SD set and an HD set. In detail, here’s what you need to know about these images:

1. The first video is a standard definition video. It has 352 lines and it’s in widescreen format, made to fit the aspect ratio of old televisions (4:3 aspect ratio). It was very probably captured from a high-definition source and resized to standard definition so the video file wouldn’t be so large. The compression algorithm used is XviD, and the file (42 minutes of video) weighs in at 350 MB. As it happens, this is the stereotypical video standard used to distribute series and movies over BitTorrent.
2. The second video is what they call 720p (non-interlaced 720 lines of resolution). It was very likely captured directly from a digital source into a file, then recompressed into h.264 — a more advanced codec than XviD which, if used here, would make the file two to three times larger. This is the stereotypical HD broadcast via BitTorrent, and it weighs at 1.1 GB.
3. The first video image was resized to match the resolution of the second one, because this is the process your TV would use to show you standard definition. They were subsequently cropped to fit the interesting area because they were huge shots and this Web page isn’t as wide.
4. In case you care to know, these are still shots from the Dexter intro.

Here it is in uncompressed PNG form so you can see the quality difference:

Do you see the difference?

Points to notice:

1. Eyebrows. The SD shot shows a shape only. The HD one shows the shape of each hair.
2. Chin and facial hair. It’s a blur in the SD shot. Each hair is clearly visible in the HD one.
3. Forehead. Do the wrinkles disappear in the SD shot?
4. Eyes. They look blurred in the SD shot. They look sharp as razors in the HD one.

You can find the other ninety-nine differences on your own.

Please note that even though the HD broadcast was already recompressed a second time, the quality is still amazing. It’s very likely that standard 720p video “from the tap” (digital cable box + digital HDMI cable) looks even better. Imagine how it would look at 1080p!

HD. I never thought it would look so good. And now that I got myself a 24″ wide-gamut 1920×1200 monitor… dang, I can finally appreciate it.

Pero no es sólo eso. Antes de la ley, el crimen en general estaba por los cielos en esa ciudad. Hoy, los pocos delitos que se cometen no son ni una fracción de aquellos.

Pocas cosas hay contra las que no se puede discutir, y una de ellas son los hechos — duros y fríos hechos. Kennesaw ha demostrado contundentemente que, a menos armas de fuego, más crimen.

The script

Stash it in /usr/lib/nagios/plugins of the Dirvish backup machine, naming it check_dirvish. This script assumes that your Dirvish vaults are in /mnt/backup, so tune it if that isn’t true in your case:

#!/bin/bash

for a in /mnt/backup/* ; do if [ -f ls -d "$a/"* 2> /dev/null | grep -v /dirvish | sort -g | tail -1/rsync_error ] ; then echo "CRITICAL: latest backup in vault $a failed" exit 2 else /bin/true fi done echo "OK: All backups OK"

The security setup

Create a nagios user on your Dirvish backup machine, and set up SSH passwordless authentication.

Now, if your Dirvish vaults are accessible only to root, set up sudo to allow Nagios to run this script as root:

nagios ALL = NOPASSWD: /usr/lib/nagios/plugins/check_dirvish

The Nagios setup

Finally, set Nagios up:

define command{
        command_name    ssh_dirvish_sudo
        command_line    /usr/lib/nagios/plugins/check_by_ssh -t 29 -H $HOSTADDRESS$ -C 'sudo /usr/lib/nagios/plugins/check_dirvish'
        }
define service{
        use                             generic-service
        host_name                       gabriela
        service_description             Backups
        check_command                   ssh_dirvish_sudo
        }

Of course, the sudo call is only needed if the Dirvish vaults are restricted for the nagios user.

El cheapo keyboards and their rubber hills

Regular cheap keyboards (and most expensive ones) use a dome-type switch under every key — it’s a hill-shaped rubber thingie that, when you press, activates a sensor below, and when you release, pushes the key back.

They suck.

Model Ms and their clickety-clickety switches

Almost all Model Ms, in contrast, use mechanical-type switches with springs (yes, there’s a spring under each key). This is the way they work:

1. When you press a key, the spring underneath it opposes an increasing amount of resistance until it sort of “gives way” (the spring actually “breaks” halfway) — and when it does, it stops resisting, clicks loudly and sends the keypress to the computer.
2. When you release the key, the spring recovers its original form, and the key bounces back, with another click.

Many people despise Model Ms precisely for the noise and the springs’ resistance. I’m sure these people aren’t typists or heavy keyboard users — I usually refer to them as ignorants — because those characteristics are exactly why you want a Model M.

And here’s why you want one: faster typing speed, more accuracy, and less frustration.

Model Ms are more accurate

Accuracy determines how fast you type, and how frustrated you get in front of the computer. Make many mistakes a day, you’ll hate your job. Type fast with no errors, you’ll be happy.

When a key on a Model M clicks, you can be 100% sure that the key was registered. If a key didn’t click (because you pressed it too lightly), then there’s a 0% chance the key registers. Dome-type keyboards aren’t that way — there’s no click, so you can’t be sure without watching the screen — and that makes your hands exert extra pressure just to be sure keypresses get through. Mind you, not that much extra pressure — you can’t consciously tell — but the effort builds up and wears your arm muscles.

Give it a shot, if you have a cheap keyboard. Press a key lightly — lightly enough that the key “gives way” but not strongly enough for the computer to detect. You’ll see this is not only possible, but a rather frequent event. That kind of half-ass keypress isn’t possible in a Model M. Either it clicks and it works, or it doesn’t. Ever.

The feedback is more than just click-click — you actually feel it under your finger

Plus, you not only get very clear audible feedback. As stated before, the springs actually give way when the key is activated, so you can tell beyond a shadow of doubt when you activated a key, even using headphones with music at 120dB, because you feel the spring give way, and that is an unequivocal signal that your keypress has gone through. You don’t need to double-check on your screen, ever.

And that feedback is key to computing without frustration

These characteristics of mechanical-type keyboards like the Model M not only (dramatically) reduce typing errors, but they let you determine without even glancing at your screen when you’ve made a common mistake, then quickly backspace over it. This makes a brutal difference if you’re into activities that require uninterrupted concentration or simultaneously reading and typing massive amounts of text. Let me give you just one example why:

Have you accidentally hit two keys simultaneously with one fingertip? I don’t know about you, but it happens to me occasionally — especially when I move my hand back from the mouse.

With dome-type keyboards, you need to check the screen to see if one or two letters were detected (because of the half-ass keypress problem I explained above). Checking the screen before typing again sucks; yes, I know it’s only a couple seconds — however, multiply that by hundreds of mistakes a day, and you get an idea of how frustrating it gets. And every time you do it, you lose your laser-sharp focus on what you were doing.

With a Model M, you can instantly tell and correct that type of mistake — because of the tactile/audible response combo — and then you just hit the Backspace key the appropriate number of times. Two clicks/bumps in your finger, one backspace; Four clicks/bumps, two backspaces. No need to check the screen and interrupt your flow — be it dictation, thoughts or computer programming — just backspace one or two times and continue typing.

And, boy, do they last!

As for their durability… I bought my Model M used from a bank in a used-equipment sale. That keyboard had been used for a decade since 1980 — at a bank terminal on a mainframe-based character terminal system. You know, bank teller monkeys pounding on its keys all day long, nine-to-six, nonstop, for ten years. And it still works like a charm. How’s that for durability?

Test one!

So, if you’d rather spend $80 unstressing on Friday nights and $3 on your keyboard, be my guest. But if you’re into doing your job quickly while avoiding stress in the first place, you should give one Model M a spin for a week (or, if you can’t find any, perhaps Das Keyboard is your next-best bet), then tell me if you went back to your old keyboard. You won’t regret it.

• The medicine: every single episode carries a medical mystery that (while not always accurate) is plausible and keeps me clawing at the TV. But (most interesting) you do not need to know medicine to understand House.
• Cameron and Thirteen: gorgeous, both of them. I’m liking Thirteen more every day. But I’d tap them both any day.
• Cuddy vs. House: he always takes potshots at her, but most of the time she delivers stunningly strong comebacks.
• The Wilson / House dynamic: how can someone be so patient with a stubborn asshole of a friend, and at the same time be able to make the asshole think? It’s about the ping-pong of ideas, insults and remarks between them.
• Gregory House himself: he is an ugly, misogyinistic motherfucker who is always on drugs and booze, sneers at people, makes offensive jokes and remarks, hurts everyone around him, makes me crack up at least five times per episode (every single of his remarks has that kind of sideline humor embedded in it, plus the delivery both in facial expressions and tone of voice)… and he is always right, and he always wins.

House fucking rules. It’s the best TV series I’ve ever watched in my life.

Be free. Today.