The NX server requires a font named fixed from the X core font subsystem when starting up. Fedora 9 no longer has the X core font subsystem by default — they solved it by compiling fixed and cursor right into the X server. Therefore, the X fonts package is no longer installed by default.

Without said fonts, the NX server fails. Once you’ve installed them, NX works again.

You’re welcome.

Why 64 instead of 32 bits? Integer arithmetic. atomic_add_64, the function in the ZFS sources that accounts for the most CPU time, is over twenty instructions of 32 bits assembler guarded by mutexes. The same function in 64 bits is one instruction. One.

I’m using Kubuntu Hardy, and my computer has two 400 GB SATA hard disks. Yes, that’s all the storage I have at hand; as of three days ago, it was RAIDed using the multipath devices (md) kernel module, split in two LVM volumes: /and /home. Oh, and two same-size byte-aligned swap partitions, one on each disk, swapon‘ed pri=0.

I had been salivating over the thought of using ZFS in my workstation because of several killer features:

• The first one that comes to mind is end-to-end data integrity thanks to checksumming — I’ve already had many disks go bad on me, while others corrupted my data silently (which is, believe it or not, the most insidious thing ever, because after you’ve noticed it, backups won’t help you with that — you’ve probably already papered over your backups with new, bad data).
• The second one is compression. Together with tightly packed data, compression promises to increase performance and reduce disk utilization.
• The third one is the advanced transactional algorithm that yields an always-consistent disk structure. Unlike log-based filesystems, ZFS does copy-on-write and ripples the changes up through the filesystem tree; before the topmost node is updated, the changes don’t affect consistency; when the topmost node is updated, the disk is consistent as well. Never fsck again!

“Damn, gotta get me some of that, I thought”

Getting ZFS was actually a piece of cake: I went to the Mercurial repository for the project, selected the tip view, and downloaded a nice tarball. I then installed a couple of dependencies according to the README, and hit scons in a terminal window. Five commands were built:

• zfs-fuse, the daemon that serves FUSE requests. The FUSE module is an odd beast: applications futzing with a FUSE-mounted filesystem talk to the kernel VFS, which talks to FUSE, which talks to the daemon backing that particular mount. This userspace-kernelspace-userspace-kernelspace–userspace overhead, you will see, is a big deal.
• zfs and zpool, the main management commands that use IPC to talk to zfs-fuse.
• two others that you won’t care and I won’t care either.

A cursory inspection with such important system binaries was in order, so I ldd the daemon and the commands. zfs-fuse links to /usr/lib/libz*.so*. Not good, chicken and egg problem, linking to a library in a filesystem that will not be available before zfs-fuse is running? I rebuild it using a modified SConstruct file so it statically links that library in.

I had decided that my filesystem layout would be:

• 1 GB swap partition on each disk
• 1 GB / filesystem, composed of two RAID1 partitions (one on each disk), formatted with ext3 (in case of catastrophe, it’s nice to have something the kernel can boot without initial RAM disks)
• 398 GB ZFS volume, where I planned to drop /usr, /home and /var

But I didn’t have extra hard disks to make the switch. No problem, croupier, everything I have on red please — and spin that wheel! I installed ZFS directly on my running system. How did I do it? Well, if you must know:

• I offlined the second disk with mdadm.
• I swapoff‘ed its swap partition. At this point the disk is no longer busy.
• I repartitioned the disk (if the disk is non-busy, the kernel rereads the partition table just fine).
• Then I relied on the first all the time.

Yes, realtime no-boot filesystem switchover — or at least I thought it would be that easy (I was very wrong).

Then I mkfs.ext3ed the new 1 GB root filesystem, and mkswap‘ed the swap one. A couple of rsyncs later (which I scripted for consistency and repeatability), I had a new, working /. I mounted it and went in it, to remove mdadm.conf and lvm.conf lines that could prove problematic on next boot. At this point I was panicking because it was superstitiously conceivable that, after a reboot, md would want to rebuild the arrays and destroy the second disk.

I then copied the ZFS binaries in /sbin and ran it. A cursory lsof inspection told me that the ZFS socket was on /etc/zfs/zfs_socket. zpool create quickly gave me the 392 GB of disk space that were previously empty in the second disk, in which I created subvolumes, with adjusted mount points to end up under a temporary tree structure under /newfs. Curiously, after creating a subvolume, it’s not mounted, but a zfs mount -a works as you probably would expect.

I enabled compression in the root volume (subvolumes inherit attributes) and started rsyncing /var, /usr and /home into each subvolume. Cue the movie 32 hours later to have an idea of how slow it was. It was unbelievably slow — un-frigging-believable, with both CPUs nearly pegged and regularly hovering at 150% combined user+system. The worst part is, I was seeing disk throughput in the 2-3 MB/s range, using iostat 1 and zpool iostat 1. Keep in mind that performance (high write throughput, low responsiveness/latency during massive reads) is marketed as a ZFS selling point — and I don’t doubt the Sun guys… on Solaris, not Linux!

During that lengthy process I started finding out several things that would prove crucial later on:

• FUSE does not support mmap in the Linux kernel that my distribution uses. Many, many applications rely on that feature to work.
• There was no initscript for ZFS. I would have to write an initscript from scratch. On Kubuntu, where initscripts are being (1) phased out and (2) completely different to my beloved RPM distros.

At this point I was a bit nervous, if you’ll allow me to understate. But I wrote the initscript anyway:

#! /bin/sh

BEGIN INIT INFO

Provides: zfs

Required-Start: mountall

Required-Stop: sendsigs

Should-Start:

Should-Stop:

Default-Start:

Default-Stop:

Short-Description: Enable/disable the ZFS-FUSE subsystem

Description: Control ZFS-FUSE subsystem

END INIT INFO

PIDFILE=/var/run/zfs-fuse.pid LOCKFILE=/var/lock/zfs/zfs_lock

. /lib/init/vars.sh

. /lib/lsb/init-functions . /lib/init/mount-functions.sh

export PATH=/sbin:/bin unset LANG ulimit -v unlimited

do_start() { test -x /sbin/zfs-fuse || exit 0 PID=cat "$PIDFILE" 2> /dev/null if [ "$PID" != "" ] then if kill -0 $PID 2> /dev/null then echo "ZFS-FUSE is already running" exit 3 else # pid file is stale, we clean up shit log_action_begin_msg "Cleaning up stale ZFS-FUSE PID files" rm -f /var/run/sendsigs.omit.d/zfs-fuse "$PIDFILE" log_action_end_msg 0 fi fi

pre_mountall

log_action_begin_msg "Starting ZFS-FUSE process"
zfs-fuse -p "$PIDFILE"
ES_TO_REPORT=$?
if [ 0 = "$ES_TO_REPORT" ]
then
    true
else
    log_action_end_msg 1 "code $ES_TO_REPORT"
    post_mountall
    exit 3
fi

for a in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
do
    PID=`cat "$PIDFILE"`
     [ "$PID" != "" ] && break
    sleep 1
done

if [ "$PID" = "" ]
then
    log_action_end_msg 1 "ZFS-FUSE did not start or create $PIDFILE"
    post_mountall
    exit 3
else
    log_action_end_msg 0
fi

log_action_begin_msg "Immunizing ZFS-FUSE against OOM kills and sendsigs signals"
mkdir -p /var/run/sendsigs.omit.d
cp "$PIDFILE" /var/run/sendsigs.omit.d/zfs-fuse
echo -17 > "/proc/$PID/oom_adj"
ES_TO_REPORT=$?
if [ 0 = "$ES_TO_REPORT" ]
then
    log_action_end_msg 0
else
    log_action_end_msg 1 "code $ES_TO_REPORT"
    post_mountall
    exit 3
fi

log_action_begin_msg "Mounting ZFS filesystems"

zfs mount -a
ES_TO_REPORT=$?
if [ 0 = "$ES_TO_REPORT" ]
then
    log_action_end_msg 0
else
    log_action_end_msg 1 "code $ES_TO_REPORT"
    post_mountall
    exit 3
fi

if [ -x /usr/bin/renice ] ; then
    log_action_begin_msg "Increasing ZFS-FUSE priority"
    /usr/bin/renice -15 -g $PID > /dev/null
    ES_TO_REPORT=$?
    if [ 0 = "$ES_TO_REPORT" ]
    then
        log_action_end_msg 0
    else
        log_action_end_msg 1 "code $ES_TO_REPORT"
        post_mountall
        exit 3
    fi
    true
fi

post_mountall

}

do_stop () { test -x /sbin/zfs-fuse || exit 0 PID=cat "$PIDFILE" 2> /dev/null if [ "$PID" = "" ] ; then # no pid file, we exit exit 0 elif kill -0 $PID 2> /dev/null; then # pid file and killable, we continue true else # pid file is stale, we clean up shit log_action_begin_msg “Cleaning up stale ZFS-FUSE PID files” rm -f /var/run/sendsigs.omit.d/zfs-fuse “$PIDFILE” log_action_end_msg 0 exit 0 fi

pre_mountall

log_action_begin_msg "Syncing disks"
sync
log_action_end_msg 0

log_action_begin_msg "Unmounting ZFS filesystems"
zfs unmount -a
ES_TO_REPORT=$?
if [ 0 = "$ES_TO_REPORT" ]
then
    log_action_end_msg 0
else
    log_action_end_msg 1 "code $ES_TO_REPORT"
    post_mountall
    exit 3
fi

post_mountall # restore /var/lock and /var/run to their right places

log_action_begin_msg "Terminating ZFS-FUSE process gracefully"
kill -TERM $PID

for a in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
do
    kill -0 $PID 2> /dev/null
    [ "$?" != "0" ] && break
    sleep 1
done

if kill -0 $PID 2> /dev/null
then
    log_action_end_msg 1 "ZFS-FUSE refused to die after 15 seconds"
    post_mountall
    exit 3
else
    rm -f /var/run/sendsigs.omit.d/zfs-fuse "$PIDFILE"
    log_action_end_msg 0
fi

log_action_begin_msg "Syncing disks again"
sync
log_action_end_msg 0

}

case “$1″ in start) do_start ;; stop) do_stop ;; status) PID=cat "$PIDFILE" 2> /dev/null if [ "$PID" = "" ] ; then echo “ZFS-FUSE is not running” exit 3 else if kill -0 $PID then echo “ZFS-FUSE is running, pid $PID” zpool status exit 0 else echo “ZFS-FUSE died, PID files stale” exit 3 fi fi ;; restart|reload|force-reload) echo “Error: argument ‘$1′ not supported” >&2 exit 3 ;; *) echo “Usage: $0 start|stop|status” >&2 exit 3 ;; esac

:

The script should explain itself.

There were two problems, though. I derived my script from the NFS one and, in the process, I discovered that NFS was symlinked to be started at slot 31 in level 6 and 0. This means that the initscripts subsystem would call that script with a start argument when in reality, the action was in the stop block. Since I couldn’t figure out what kind of magic the Upstart initscripts compatibility subsystem does to get a stop block to run when a start block is requested by its configuration, I just created two glue scripts: one to start ZFS no matter what, and one to stop ZFS no matter what:

-rwxr-xr-x 1 root root 481 2008-06-18 04:09 /etc/init.d/mountzfs
-rwxr-xr-x 1 root root 488 2008-06-18 04:09 /etc/init.d/umountzfs

Then I studied the Kubuntu boot sequence very carefully, and used some elbow grease (update-rc.d) to symlink them to get the results I wanted:

lrwxrwxrwx 1 root root 19 2008-06-18 03:52 /etc/rc0.d/S35umountzfs -> ../init.d/umountzfs
lrwxrwxrwx 1 root root 19 2008-06-18 03:52 /etc/rc6.d/S35umountzfs -> ../init.d/umountzfs
lrwxrwxrwx 1 root root 18 2008-06-18 03:52 /etc/rcS.d/S36mountzfs -> ../init.d/mountzfs

Trust me, writing the script was the easy part — figuring out how it interacts with the rest of the system was much harder.

Finally, I rebooted to my new root filesystem on the second disk. If you thought that my system booted correctly, you would be very, very wrong indeed. Eighty percent of the boot sequence were red [ fail ]s and sh: command not found errors. At the end, the system dropped me into a recovery console, where I could finally switch the ZFS mount points to their final destinations. Then, just to try out: zfs mount -a.

/homemounted. /var couldn’t be mounted, because the boot process graciously created incredibly important missing directories in it. And then, deadlock.

Crap, what was wrong?

Alt+SysRq+R. Boot again. What’s wrong? No idea. Try strace. The friggin’ command is in /usr. Hypotheses ran through my head for two hours. With me in front of a very, very broken system. I tried everything under the Sun that I could get my hands on — which is not much when you don’t have a CD-ROM drive, mind you.

And (summarizing two hours) then, I tried this: zfs set mountpoint=/tmp/usr vault/usr ; mkdir -p /tmp/usr ; zfs mount vault/usr.

Miracle of miracles, it worked. I copied the entire cast of characters of Linux Debugging: The Movie into the very tightly packed /. I strace --ffed the hell out of zfs-fuse and I found the problem. The moronic mount.fuse subcommand, that actually connects the kernel and user endpoints, tries to read /usr/lib/locale/locale-archive right in the middle of mounting the filesystem! Instant deadlock that you can only get out of by using the SysRq OOM key (yes, zfs-fuse is actually a great OOM candidate — 1.5 GB VM size on this 1.0 GB RAM computer; yes, I discovered that on my own before I wrote the OOM immunization code in the initscript).

I then discovered two things: zfs-fuse didn’t deadlock when started from the recovery command, but it did lockup when starting it from the initscript. What you can’t see is that the version of the initscript that I initially wrote was sourcing the LANG variable from a configuration script in /etc. OK, so how do you solve locale problems? Instant fixup: unset LANG before running the command.

OK, so do I have a booting system now, or what? Wrong again. Some processes get started before the actual mounting of filesystems, and the ZFS subsystem can’t actually be started earlier in the boot process without creating an initramfs dependency or another, different, chicken-and-egg problem. So I moved what I could move from the ZFS volume’s /var into the /var directory of the / filesystem. I ended up with this structure backed up by ZFS (and the rest, you can safely assume, in a very tightly crammed ext3 filesystem):

zfs list
NAME              USED  AVAIL  REFER  MOUNTPOINT
vault             294G  69,8G    18K  none
vault/home        290G  69,8G   290G  /home
vault/usr        3,36G  69,8G  3,36G  /usr
vault/var         842M  69,8G    18K  none
vault/var/cache   515M  69,8G   515M  /var/cache
vault/var/lib     282M  69,8G   282M  /var/lib
vault/var/tmp    44,5M  69,8G  44,5M  /var/tmp

Boot again. Oh, yeah, I’m enjoying the 3-minute boot time on this formerly-a-screamer machine. D-Bus fails to start. D-Bus is actually very required for many things in Kubuntu, but I manage to start a GUI session up, if only to Google up what was wrong with it. That was probably not the best moment to find out that just starting the KDE 3.5 session took over ten minutes. All of this with less than 1 MB/s from the disk, according to iostat and 160% CPU usage, according to top.

Then I discovered the zfs-fuse Google group. It’s a fantastic place where everyone (including Ricardo Correia) received me very well and had lots of tips. Only there did I find out what was wrong with D-Bus — a bug that manifests itself only with FUSE filesystems, for which a patch exists and works.

At this point I’m extremely exhausted from this marathon session, so I basically just try to backport the patch into the dbus source package for my distribution. You’ve probably heard that Debian (and, by extension, Ubuntu) has a fantastic build system — it failed on me. Not only was apt not working (remember the mmap issue?), but dpkg-source also failed while trying to apply the patches for the source package. Oh, yes, I manage to solve this problem by learning, on-the-spot, how the apt build “system” actually works, and manually replicating the entire process that should be automated. Many thanks to the gents at #debian in Freenode for their kind responses to my questions.

Bam, built dbus (it’s yours if you want it). Installed it. Started it. And the chain of daemons that were depending on it just start up and take life. Neat trick, Upstart!

Back to performance questions and ZFS. Do you know what the real performance killer is? You’ll never guess it…

…icons! While GTK+ applications take marginally more time to start under a ZFS regime, KDE applications take an order of magnitude more. Before, on a warm working set, a KDE application took about 2 seconds to start. Today, Kmail takes in excess of five minutes to start. Why? Here’s why — multiply that by fifty thousand and you’ll get the idea. Each icon that the application requests results in thousands and thousands of access() and stat() calls. FUSE doesn’t use a kernel cache by default (there are several reasons for that), so the only cache that backs those requests up is the ARC cache, which is an impressive caching regime and technical achievement but, in this case, it’s very much like caching your car keys somewhere in Europe, because of the transatlantic userspace-kernelspace-userspace-kernelspace-userspace barrier. Per-call. When this is taking place, the CPUs remain pegged at 190%, eaten by ZFS alive, and the 12 case fans jump to 11.000 RPM.

The zfs-fuse Google groups guys came up with a couple of suggestions (all documented in the list, which I’m too lazy to link to again). These all are compile-time options, so a ZFS rebuild is in order for every one of them:

• scons debug=0. A very slight CPU usage decrease.
• Increasing the ARC cache. I doubled it from 128 to 256 MB. Turns out it’s not a caching problem and it doesn’t help at all.
• Mount option big_writes for FUSE filesystems. Here’s what I did about that:

Recompiled ZFS, this time enabling a FUSE mount option named big_writes that I’ve read about in the Google group. Yes, the daemon needs to be recompiled, and it’s not fast. No, I’m not actually jumping to the part where I actually compiled ZFS with big_writes first, then booted, only to find out that I needed a new kernel. Oh, wait, I just did. Fortunately, I did back zfs-fuse up.

Next up? Latest 2.6.26-rc6 kernel, because of:

• Hey, writable mmap is there for FUSE filesystems! Yeah! Now I can have apt-get back!
• big_writes.

When was the last time a kernel compile took four hours for you? Mine was yesterday. But it’s actually fun — the process hasn’t changed that much from 1998, and the distro already comes with a nice .config that you can reuse with make oldconfig. And, this time, you get to do out-of-tree kernel builds! Yay!

Well, I ticked the wrong option in make menuconfig anyway, because my kernel modules don’t fit my puny /, now at 400 MB free. Jeez, four hours. Google some more. Turns out I turned a debugging option on.

After this, FUSE userspace itself was due for a recompile. Another odyssey, whose fruits you can reap here (warning: CVS checkout).

OK, redo the initial RAM disk, adjust GRUB configuration, reboot with the latest kernel. It’s all good. More surprisingly, I’m actually getting some of my performance back. Some of it. As in “Kmail no longer takes five minutes to start — only three”.

And, most importantly, applications that depend on mmap now work correctly. My boot process isn’t an epic [ fail ] anymore — and that’s incredibly reassuring.

This is the point where my journey turns into smooth sailing. I zpool scrubed my new baby. After five hours, with the solid guarantee that my data was OK and nothing’d been lost or corrupted during the rsync, I nuked my first disk, replicated the new partition structure on it. A nice RAID1 array for the final /. A short rsync for the / filesystem. A quick mkswap for the new swap partition. A fast adjustment in /etc/fstab and another one in mdadm.conf for the new array. Reinstall and reconfigure GRUB on the first disk. And, finally, I leave the best for the latest:

zpool attach vault /dev/by-id/second-disk-huge-partition /dev/by-id/first-disk-huge-partition

Man, that rocked. It was unbelievably fast — like, disk-platter fast, around 40 to 50 MB per second, and the system didn’t get that much more slow when it was resilvering the first disk. Which kind of makes lots of sense, because zfs-fuse is now crossing the userspace-kernelspace barrier just once per operation. How do I know this? Well, strace: I know that what zfs-fuse does is, it opens the disk partition in direct I/O mode and then manages it for itself, responding to FUSE requests — but the resilvering process doesn’t involve FUSE at all, it’s just the two disks practically chatting with each other through zfs-fuse. Now I know for sure that ZFS will give me platter speeds. It’s just a matter of time (and maybe me pestering Ricardo Correia to collaborate with me on this same issue).

Questions that I haven’t solved yet? Sure, there are a lot. Two that haunt me:

• No root filesystem on ZFS. Others on the Google group have managed it. Me? I didn’t want to mess with /etc/zfs inside the initramfs, thank you very much.
• I know this for sure: the only active cache now is the userspace ARC cache from ZFS; I read the FUSE kernel code, and it clearly flushes files from the cache when programs open() them. Honestly, if I could wish for something to just become true overnight, I’d wish for the ARC to be moved into the kernel and to have it replace the page cache, but that won’t happen anytime soon. There’s a FUSE kernel_cache option, but I’m wary of enabling it. When I have been sufficiently reassured that the option won’t corrupt my precious data, I will enable it. That will be a couple of hours of reading someone else’s code, so I’m inclined to defer it for a few days. But, in theory, this should give me platter speeds instead of giving my 12 case fans ’speed’. At the hefty cost of RAM for two redundant caches.
• Do filesystem readahead and Linux disk scheduler algorithms mess up in some way with ZFS’ control of the platter? The data integrity question is closed, because the writes are submitted with barriers, but I’m worried that the Linux I/O scheduler is second-guessing the decisions of ZFS’ one.
• The /etc/init.d/sendsigs omit.d protocol I’m using on the initscript plain fails. I had to shunt the script with an exit 0 right before the killall5 in sendsigs because killall5 plain hung instead of ignoring ZFS as it should have done — and it needs to ignore ZFS because ZFS is unmounted later. This won’t be a problem once we get our own kernelspace ZFS implementation.

OK, that was my journey. I’m on ZFS now, my machine’s rock-solid (if a bit CPU-tired) and my data’s never been so safe. I also got compression, which saved me about 6 GB. Furthermore, I’ve given you the initscript, the steps and the software (except ZFS, but you can compile that yourself).

Go wild.

What you’re about to see is just a twenty-line snippet representing standard pathological behavior of a widely used KDE application that comes with Kubuntu. I’m at a loss of words, because what you’re about to see repeats itself (I kid you not) at least a twenty thousand times:

access("/usr/share/icons/crystalproject/24x24/devices/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/24x24/filesystems/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/24x24/filesystems/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/24x24/mimetypes/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/24x24/mimetypes/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/32x32/actions/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/32x32/actions/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/32x32/apps/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/32x32/apps/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/32x32/devices/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/32x32/devices/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/32x32/filesystems/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/32x32/filesystems/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/32x32/mimetypes/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)

Now, let me ask you: what kind of engineering is this? Oh, I’ll just look for the icon named mail_todo of types XPM, SVG, SVGZ, PNG, JPG across thousands of directories and if I can’t find it, I just won’t draw it.

Modern desktop environments package all that shit up in a single icon cache and use mmap to quickly find it. Well, guess again, loser who invented that band-aid. I’m using a filesystem that doesn’t support mmap. Now my filesystem has to consume inordinate amounts of CPU to find each one of those two hundred icons your application uses, because you couldn’t find a better algorithm to find the motherfucking icon.

Oh, by the way, every time I delete an e-mail from KMail’s listing, the entire process is repeated again. I guess I can safely omit the part where Kicker (the panel) and Kopete all do the same things for each icon that they display. Woops, I didn’t omit it.

Just another tale of living in the bleeding edge of Linux computing.

In the interest of full disclosure: I’m using the CPU-heavy ZFS filesystem from Sun. I’m testing it because it offers unsurpassed reliability. Yes, it’s CPU-heavy. Yes, it runs through FUSE. But that’s not the point — the point is that this wouldn’t matter if KDE apps (and, I’m sure, many others as well) wouldn’t suck so much to find a motherfucking icon. After the latest torvalds kernel recompile and another ZFS recompile for performance (which failed), I have gained quite a lot of performance: now KMail doesn’t take 10 minutes to launch — it only takes four. On a dual-core Xeon, with 1 GB RAM and 7200 RPM dual platters. Four motherfucking minutes. I now fully agree with Cox when he said userspace does stupid things.

rudd-o@karen: ~ $
history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head
76 cd
38 mv
36 svn
34 python
31 rm
30 ls
19 git
14 dpkg
14 aptitude
13 find

Please note: I have history set up so it doesn’t record repeated commands. So this count is a count of unique, different commands each.

Want the graphwatch version? Here it is:

cd 70 #################################################
      mv 36 #########################
  python 34 #######################
     svn 32 ######################
      rm 29 ####################
      ls 29 ####################
     git 16 ###########
    dpkg 14 #########
aptitude 14 #########
    nano 13 #########

Here’s what I did to generate that graph:

1. history > x
2. graphwatch 'cat x | awk '\''{a[$2]++}END{for(i in a){print a[i] ” ” i}}’\” | sort -rn | head | awk ‘\” { print $2 ” ” $1 } ‘\”’ ” “

graphwatch is a nice Python script that takes a properly quoted shell pipeline that will be run through bash and a field separator (in this case, a sngle empty space). My extra awk invocation there swaps the first and second fields so the first column is the label, and the second column is the value — graphwatch plots a nice set of proportional horizontal bars.

But that’s not it. The command is run once every second, so if the contents of the file x (or the output of the supplied command) change, the graph is re-plotted. Very nifty to watch tabular numeric output in graphical form, in real-time — imagine using it with iostat and you’ll get the idea.

Here’s the code. I will release it as a proper package with enhancements (it’s a quick hack right now) later on:

#!/usr/bin/env python

from time import sleep from subprocess import PIPE,Popen from sys import argv,exit,stdout import os

try: cmd = argv[1] sep = argv[2] except IndexError, e: print "usage: graphwatch 'shell pipeline' 'separator'" exit(os.EX_USAGE) print "Running this command in a shell:" print ">", cmd

def getstdout(cmdline): p = Popen(["bash","-c",cmdline],stdout=PIPE) output = p.communicate()[0] if p.returncode != 0: raise Exception, "Command %s return code %s"%(cmdline,p.returncode) return output

try: while True:

    output = getstdout(cmd).strip()
    stdout.write(getstdout("clear"))

    labels = []
    values = []
    for l in output.splitlines():
        try:
            label,value= l.strip().split(sep,1)
            label,value = label.strip(),value.strip()
        except ValueError, e: label,value = None,l.strip()
        labels.append(label)
        try: values.append(float(value))
        except ValueError, e: values.append(0.0)

    try:
        maximum_value = max(values + [maximum_value] )
    except NameError, e:
        maximum_value = max(values)

    try:
        longest_label = max( [len(l) for l in labels if l is not None ] )
        label_fmt = "%" + str(longest_label) + "s"
    except ValueError, e:
        longest_label = -1 # to compensate for the extra -2 below
        pass # nothing will happen later

    longest_number = max( [ len(str(int(i))) for i in values if type(i) is float ] )
    number_fmt = "%" + str(longest_number) + ".0f"

    column_width = 61
    leftover_for_plot = column_width - longest_label - longest_number - 2


    for value,label in zip(values,labels):
        if label is not None:
            print label_fmt%label,
        print number_fmt%value,
        print "#"*int(value/maximum_value*leftover_for_plot)

    sleep(1)

except KeyboardInterrupt, e: exit(0)

Proposed solution: make PackageKit do exactly what synaptic has been doing for the past three years: when a package install process blocks on file descriptor 0, unhide a hidden VTE widget. As it stands now, none of the publicly facing Ubuntu or Debian PacketKit pages have proposed any other solution to the problem, and it appears there’s no other proposed solution on the mailing list. (Take care to note that this has nothing to do with debconf.)

I absolutely disagree with you.  This goddamned practice of letting packages interrupt the upgrade process willy-nilly on the packager’s whim is idiotic and should have dealt with years ago (preferably with death by fire).  Don’t tell me that the Debian project had a breakthrough invention when they “invented” the “hey, let’s interrupt package installation to ask the user a dpkg-configure question or to run a postinstall script”. That is not a breakthrough; it’s a practice that should have long been abandoned.

The reason why dpkg has this brain damage is pretty clear: the installation/upgrade process and the post-install configuration process aren’t clearly separated. It boils down to a single question: it’s fine by me if you want to include a package that needs extra user input for configuration — so, why don’t you ask me for the configuration at first-run time, and ship a package with sane defaults?

The RPM take on this issue has a huge advantage: by limiting user interaction during upgrades/installations, it forces packagers to distribute sanely preconfigured packages. That’s the reason you can install Dovecot, Postfix, and SquirrelMail on Fedora, and have a working mail server right out of the box — save, perhaps, for changing the listening interface setting on the Postfix configuration file.

The Debian way, on the other hand, creates a moral incentive for the packager to punt configuration questions to the user. Honestly, you can’t expect end-users to interact with a shell. The minute I show my dad a Synaptic VTE question, is the minute my dad stops using Linux.

This is one of the things that suck big time in dpkg. RPM-based distros don’t have this problem. On Fedora, I can make an unattended upgrade and it will work 100% of the time, knowing that the configuration files won’t be replaced but I will have a way to diff those configuration files. Not so with Ubuntu. Not so with Debian. I’m being candid here: I’ve had to answer tens of questions after a Debian dist-upgrade. I’ve never had to answer a single question after a smart upgrade. Yes, I know I could tell dpkg-configure not to ask me questions, but I wonder when Debian will default to that behavior.

What’s more, the RPM way has a fantastic consequence: upgrades are fully transactional. You can’t make a transactional upgrade if you’re interrupting the upgrade process by opening stdin to ask bullshit questions or other hackery. Honest, the need to run apt-get -f install bullshit is something I’ve never, ever in my life experienced with Fedora’s or Red Hat’s RPM. And it’s a weekly fucking occurrence in Ubuntu!

Just now, I had temporarily turned Nagios2 off with /etc/init.d/nagios2 stop, and the goddamned apt-get -f install won’t finish because it insists on dpkg-configuring Nagios2, and the dpkg-configure process fails because it attempts to shut down Nagios2 and that fails, leaving my system in a circle-jerk of partial configuredness, that I wouldn’t know how to fix if I didn’t have ten years of Linux experience.

Oh, and for the record, the other reasons are:

1. You can’t have two versions of the package with the same name installed in the system (I have visited this argument before). Not very used in the RPM world either, but I have put that use into practice.
2. You can’t do an rpm -Va, and have dpkg check for permissions, dates, times, and file sizes/sums. What the hell were the dpkg developers thinking? This is practically the first thing I use to audit a server right before I take it under my wing, just to get a clear idea of what the people before me did with it.
3. When you remove a package with dpkg, the configuration files remain unless you purge it. Why can’t dpkg do like RPM — erasing configuration files unless they were modified by the user? Two different remove operations are one too many.
4. dpkg happily lets you destroy your system.

And, quoting you:

This has the added benefit of making it possible for any user on any distro. to unhide the VTE window when something goes wrong–and we all know that something goes wrong with package managers all the time.

I dunno about your experiences with Red Hat or Fedora, but when I use a GUI package manager on Fedora, and something goes wrong, what I’m interested in is the output of stderr. All GUI package managers neatly show these messages nowadays. And I rarely see any error dialog box with stderr contents. So don’t start with the “something goes wrong with package managers all the time” bullshit.

So, let’s amend your last paragraph: let’s make PackageKit redirect /dev/null into the dpkg subprocess. That should take care of things neatly, and screw all braindead packages out there. What’s more, if the package somehow refuses to continue installing, let’s send dpkg a SIGKILL after a while, then prompt the user to change to a non-braindead distribution. So PackageKit doesn’t support your favorite brain damaged behavior? Sucks to be you.

Jeez, how much more of this obsolescent crap and stupid policy do we have to swallow, just because it’s the policy of the most widely used distro? I thought open source and free software were about meritocracy, not the tyranny of the majority. And I wish that the portion of the brain of the developer that conceived this “installation is interruptible” brain damage would just necrotize thanks to an aneurysm.

These packages are built from the source code of the next release of Ubuntu, Hardy. They work flawlessly on my home machines (yes, plural). I built these packages myself for my usage:

• libpulse-browse0_0.9.8-1ubuntu3_i386.deb
• libpulse-mainloop-glib0_0.9.8-1ubuntu3_i386.deb
• libpulse0_0.9.8-1ubuntu3_i386.deb
• libpulsecore5_0.9.8-1ubuntu3_i386.deb
• pulseaudio-esound-compat_0.9.8-1ubuntu3_i386.deb
• pulseaudio-module-gconf_0.9.8-1ubuntu3_i386.deb
• pulseaudio-module-hal_0.9.8-1ubuntu3_i386.deb
• pulseaudio-module-lirc_0.9.8-1ubuntu3_i386.deb
• pulseaudio-module-x11_0.9.8-1ubuntu3_i386.deb
• pulseaudio-module-zeroconf_0.9.8-1ubuntu3_i386.deb
• pulseaudio-utils_0.9.8-1ubuntu3_i386.deb
• pulseaudio_0.9.8-1ubuntu3_i386.deb

Download them all in a folder, then just open a terminal window there and type:

sudo dpkg -i *deb

That should upgrade you to the latest PulseAudio. I was gonna write a tutorial, but then you would have had to download hundreds of megabytes of development packages.

Fast-forward to my contemporary Kubuntu system. I just installed (using dpkg -i) a series of packages, and dpkg happily unpacked them all on my filesystem, only to nonchalantly tell me — when the files were already replaced on my (now very broken) computer — that it could not configure these packages, because the version of my C library is too old.

With the new files are already on my system! A tad too late to blow up on my face, isn’t it? Now I have a badly broken system, and I can’t back out without resorting to manually downloading and installing old packages.

Question to the dpkg developers: If the version of my C library is too fucking old — and dpkg knows this fact — why the fuck does dpkg proceed with file installation, only to barf in the middle of the installation process, leaving me with a broken computer? If the dependencies don’t fucking fit, don’t fucking let me install the packages in the first place!

It’s not like transactional package installation is rocket science to get right, is it? Otherwise, how did the RPM guys figure this out so well?

In b4 “dpkg is not like RPM” comments:

1. Software installation transactionality is an elementary requirement of contemporary systems, so get your damn act together and make dpkg like RPM with regards to this issue.
2. Even better: ask yourselves the following question: in what universe is this dpkg behavior (namely, enabling users to badly break their systems) a good thing?

And, please, let me rant some more, this time on the topic of package dependencies. Why does a simple 100 KB program from Hardy require me to upgrade my system C library (hence, my entire operating system) to Hardy? I thought glibc hadn’t changed in years! I hardly doubt that 100 KB program would have any trouble running if I unpacked the binary directly! So why does the package builder automatically require the latest version of the library — forcing me to download gigabytes of packages — when an old one would work just as well?

I’m a sysadmin, for fuck’s sake, I want my computer to be easily manageable, not harder to manage. What are you waiting for?

This will be a command-line tutorial-style article, and we’re gonna get down to basics, so no GUIs or automated installatinos from the Internet (automated dependency management) here. I will, however, make a recommendation as to which dependency management system you should use, because nowadays it would be rather insane to do the following work manually — except for very particular cases in which this tutorial will come in very handy.

Installing packages

You want new software? You have to install it. First you download it from the ‘Net (you’re gonna be looking for a file ending in .rpm), then you install it on your system. As simple as that… in theory.

The basic installation operation in RPM is rpm -i packagename-x.y.z.rpm. I tend to combine it with the -vh arguments which cause a progress bar to appear, and verbose messages to be printed onscreen for a variety of operations within the transaction.

Transaction. With RPM, to ensure the integrity of your computer, either the package installation happens, or it doesn’t happen at all. If you request to install 450 packages in a single command (which you could easily do), and one of them cannot be fit into the system, the transaction fails and your system remains untouched. Reasons for a failure may include that:

• the package you requested to install is already installed, whether the installed one is an older or a newer version
• the package would trample on files owned by another installed package
• the package is bad
• the package requires another package or set of packages installed beforehand — this is called a dependency

All of those reasons, and a couple more, could leave your system broken if RPM were actually to go through with installation. Therefore, RPM stops when it detects any of these conditions.

--force

Unless you augment the command with the --force argument. Trust me when I say this: if RPM is refusing to install a package, under no circumstances should you use --force even if you badly need to. RPM is probably much better informed than you are as to what the system actually needs to continue functioning properly, and you should trust it. --force is only handy when you’re absolutely sure of how the different related components of your system relate to each other.

Upgrades and downgrades

We see that rpm -i will fail if you attempt to install a newer or an older version:

• If you really need to downgrade, you can always append --oldpackage to the command line, and if the integrity checks are A-OK, RPM will downgrade a package for you.
• Conversely, if you want to upgrade, you use rpm -U instead of rpm -i. As a matter of fact, rpm -U will both install and upgrade packages happily, so you can just use rpm -U most of the time, and forget about -i.

RPM will also, in the spirit of preserving your system as pure and functional as possible, refuse to overwrite custom configuration files in your computer. It will, however, deposit them in the folder where they were meant to live, by appending a .rpmnew extension to the new file — so you can use the standard find command to find and inspect these config files, see if they bring anything new and useful to the table.

Controlled mass upgrades of what you already have

What if your best friend shared a CD-ROM with the latest updates — say, 850 RPM files — for your Linux distribution? You could go one by one and install each. That’s not gonna go well, because many of those updates may not be installed on your system.

What you could do instead is use rpm -F. Like issuing a command rpm -Fvh *.rpm while on the CD-ROM directory. -F means freshen — it will only upgrade packages for which older version are already installed on your system.

In theory, you could perform an entire operating system upgrade this way. In practice, it’s a little more complicated because as distros evolve, they start requiring more base packages which you don’t have installed, and -F won’t know which ones are these (although it will tell you which dependency it failed to find installed!). When I say in practice, I mean it — I have used this method to upgrade a machine that just didn’t have enough RAM to run an installer on — and I succeeded, but I had to go in smaller batches of rpm -F’s with some rpm -U’s in between.

Package removal

Want to remove a package? rpm -e is what you’re looking for. As discussed in the previous section, if a removal would break your system, RPM would not let you perform.

Config files are also dealt with gracefully. If you’ve changed any, RPM will save them with the same file name, and a .rpmsave extension. This is incredibly well-thought-out, because you will often need them sooner or later!

Listing and inspecting packages on your system

Do rpm -qa. The -q stands for query, and the -a stands for all. Most RPM commands accept -a as an argument to say all anyway, so it’s good to keep it in mind.

By default, it will present you a one-column list with all the package names and currently installed versions of them. If you want more specific information, rpm -qi packagename (note the use of a package name as opposed to a file name) is usually what you want. And rpm -qpi packagename-x.y.z.rpm (note the filename) is used to inspect uninstalled RPM packages on disk.

Want to know which package a file belongs to? rpm -qf /path/to/said/file will tell you. Very handy!

Integrity verification

Want to see if any files have been modified, and how? rpm -V packagename is the command. It will display a series of lines, one for each file that has been modified, or nothing at all if no files have been touched. Several aspects are displayed in a column-packed display: file times, permissions, integrity and a whole lotta more are included. Checkt he RPM manual page to see what each column means.

Perhaps you’d like a full audit of what’s on your system and what’s been changed? Combine two popluar arguments into rpm -Va. Verify all. It will probably be a long list so make sure to redirect the output to a file.

Getting packages for your distro

Easy. Go to Google, type the name of the package followed by a space and the letters “RPM”. Lots of results will follow. Find those results that appear from RPM repositories (those sites usually say they are RPM repositories, so you’ll have no problem). If one of your packages has a dependency problem, you can also type the dependency name into Google followed by “RPM” — and that will most likely help you solve dependency problems.

Get Smart!

But, honestly, that’s inefficient. Just get Smart, where you’ll be able to add said repositories to the Smart configuration (don’t panic, it’s usually an URL that needs to go into a file in /etc/smart/channels plus a command to import the security key of the repository — and repositories explain this procedure quite more eloquently than I do when I’m this sleepy). Then you will be able to invoke Smart via the command line or the nice GUI it has, request a particular package and install it. Smart will automatically download all the depended packages and install them for you.

Don’t forget to read the Smart manual page!

Conclusions

And that concludes our RPM crash course. If you have any doubts, please add them as comments below. And discuss!

Warning ahead: please note that I’m not gonna discuss Debian vs. Red Hat, but dpkg vs. RPM.

RPM, combined with a good package manager such as Smart, is vastly superior to the competition:

The strengths of RPM

• No indirections — 100% of the times, the manifest corresponds to the installed files. Have you seen dpkg detours/indirections (dunno their name in English)?
• Built-in integrity checking. rpm -V. Can’t get any easier than that.
• The dependency system is cleaner. It allows and actually by default generates dependencies on files, which rocks when several packages provide the same functionality.
• The RPM philosophy doesn’t allow interruptions during package installation or removal. No stupid questions about config files getting replaced or not, license agreements and configuration questions. This guarantees unattended transactional installations/upgrades (dpkg does not guarantee it without extra effort, because it defaults to interactive behavior) whereas with dpkg, if you answer one question wrong, your system may stay unconfigured. I know it because I lived it. I’d very much rather have the package come in a default useful configuration then configure it than have an assistant interrupt me upon installation to ask me how I want to configure it.
• RPM supports two versions of a package with the same name, whereas dpkg does not. I know this because I have a machine with kdelibs 3.5.7 and kdelibs 1.4 to support Katalog from KDE 1.0, and no broken dependencies. However, this is regrettably a theoretical advantage because as of late many RPM-based distros have picked up the bad habit of naming stuff after Debian package conventions (think libXXX2.2); the Debian practice is both unneeded in the RPM world and stupid because the version of the package does not conceptually belong in the package name, and doing that constitutes a hacky workaround around inferior technology which RPM does not need.

The disadvantages

The only things RPM has going against it? It’s a bit harder to prepare spec files for RPMs, and it’s hard to prepare distro-agnostic RPMs, which has the logical consequence of reducing availability of RPMs ready to install. RPM-based distros should take a clue from deb-based ones right now, or else they risk continuing to get kicked in the ass for the foreseeable future.

Discuss.

You’re free to use this diagram on your own site or documentation, as per the license that covers it (see the footer). However, I do ask you for credit and attribution if you use it. The image is also licensed under the GFDL, just in case you need it for free documentation.

The diagram is also available as a Dia diagram.

What you’re about to read is a short, true story that details how I went from idea to solution, using the “UNIX way” (divide a problem in little pieces, then jump each hurdle with the help of a domain-specific solution, and finally combine each response).

The story is — honest! — more about the underlying thought process than the technology I used — in fact, while I do know Python and I’ve grown white hairs using Linux, most of the stuff you’ll see here I taught myself in the course of this challenge.

TorrentFlux: what it is, how it works

TorrentFlux is a very interesting Web application you can install on your Web server. Once installed, it lets you in to a nice interface like any other BitTorrent program where you get to add .torrent files so the torrent gets downloaded directly in your Web server. Here it is:

Advantages of using TorrentFlux

The reasons to use such a program instead of a normal desktop BitTorrent client are varied:

• you would like to have a particular, big file (think Linux distro) right away on the server instead of uploading it through your ISP’s connection
• your ISP throttles or blocks BitTorrent (think Comcast)
• you would like to avoid downloading through 50 TCP connections at home, and just use one (think speedier Web browsing)
• you want to take advantage of the (usually) higher bandwidths available in your Web server during the day, and you’d like to download the file once it’s done, from beginning to end (think videos)
• you would like to queue up a download while away from home (think vacations)

Disadvantages

The disadvantages, compared to a regular BitTorrent client, are that:

• if you want the download in your PC, you need to queue it up separately in a download manager on your PC through FTP,
• you need to check TorrentFlux periodically to see if the torrent is done downloading, because you can’t download a torrent to your PC until the torrent is done,
• you can’t queue up downloads from TorrentFlux to your home PC if you’re away from your computer, can you?
• disk space might be limited in your Web server.

What we’ll do in this article is destroy these disadvantages.

The problem in a nutshell

From the aforementioned disadvantages we can clearly see what the problem is. Being the lazy bastards we are, we want the downloads in our home computer ASAP, without manual intervention. Our ultimate goal is to:

1. queue up a BitTorrent download in TorrentFlux, and
2. as soon as it’s finished, have our home PC automatically “check it out from the counter”, and then
3. erase the file from the TorrentFlux server.

Let’s go do it.

How I solved it

Remoting my server

How am I going to talk to my server? This question crossed my mind many times. In the end, I chose SSH, because it’s extremely easy to automate and have it work securely without constant interruptions — remember, we don’t want to be interrupted with password prompts or anything like it.

I wrote the code that more or less provided enough help for it to work. As programming languages go, Python is by far the best for the task at hand, so I used it. OK, I’ll admit it, I’m a Python junkie:

#!/usr/bin/env python

from subprocess import Popen,PIPE,STDOUT,call import fcntl import re import os import sys import signal

def getstdout(cmdline): p = Popen(cmdline,stdout=PIPE) output = p.communicate()[0] if p.returncode != 0: raise Exception, "Command %s return code %s"%(cmdline,p.returncode) return output def getstdoutstderr(cmdline,inp=None): # return stoud and stderr in a single string object p = Popen(cmdline,stdin=PIPE,stdout=PIPE,stderr=STDOUT) output = p.communicate(inp)[0] if p.returncode != 0: raise Exception, "Command %s return code %s"%(cmdline,p.returncode) return output def passthru(cmdline): return call(cmdline) # return status code, pass the outputs thru def getssh(cmd): return getstdout(["ssh","-o","BatchMode yes","-o","ForwardX11 no",torrentflux_server] + [cmd]) # return stdout of ssh. doesn't return stderr def sshpassthru(cmd): return call(["ssh","-o","BatchMode yes","-o","ForwardX11 no",torrentflux_server] + [cmd]) # return status code from a command executed using ssh

Interesting, isn’t it? Now I have two functions:

• one that gets me the text of a command executed by SSH, and
• another that discards the text but returns the return value of the executed command — which, if you know the UNIX convention, it’s zero for success, and anything else for errors.

Determining which torrents are done

The first hurdle I need to surpass: which torrents from the torrent list are already done? After a little poking around the Web, I discovered that with the help of a command called fluxcli that ships with TorrentFlux, I can get a list of active torrents in text form.

So that’s solved. All I needed is some code that parses the rows returned, selects only those rows that say Done or Seeding in the status column, and that should return the name of each torrent that’s ready to download.

The code in question is this:

def get_finished_torrents():
        stdout = getssh("%s transfers"%fluxcli)
        stdout = stdout.splitlines()[2:-5]
        stdout = [ re.match("^- (.+) - [0123456789.]+ MB - (Seeding|Done)",line) for line in stdout ]
        return [ match.group(1) for match in stdout if match ]

Stupidly simple, and it makes use of the SSH remoting functions (dare I say hacks?) I wrote before. And, may I say, only three lines, really. It returns a list with the names of the torrents.

Next up, getting the file names inside the torrents.