I’m using Kubuntu Hardy, and my computer has two 400 GB SATA hard disks. Yes, that’s all the storage I have at hand; as of three days ago, it was RAIDed using the multipath devices (md) kernel module, split in two LVM volumes: /and /home. Oh, and two same-size byte-aligned swap partitions, one on each disk, swapon‘ed pri=0.

I had been salivating over the thought of using ZFS in my workstation because of several killer features:

• The first one that comes to mind is end-to-end data integrity thanks to checksumming — I’ve already had many disks go bad on me, while others corrupted my data silently (which is, believe it or not, the most insidious thing ever, because after you’ve noticed it, backups won’t help you with that — you’ve probably already papered over your backups with new, bad data).
• The second one is compression. Together with tightly packed data, compression promises to increase performance and reduce disk utilization.
• The third one is the advanced transactional algorithm that yields an always-consistent disk structure. Unlike log-based filesystems, ZFS does copy-on-write and ripples the changes up through the filesystem tree; before the topmost node is updated, the changes don’t affect consistency; when the topmost node is updated, the disk is consistent as well. Never fsck again!

“Damn, gotta get me some of that, I thought”

Getting ZFS was actually a piece of cake: I went to the Mercurial repository for the project, selected the tip view, and downloaded a nice tarball. I then installed a couple of dependencies according to the README, and hit scons in a terminal window. Five commands were built:

• zfs-fuse, the daemon that serves FUSE requests. The FUSE module is an odd beast: applications futzing with a FUSE-mounted filesystem talk to the kernel VFS, which talks to FUSE, which talks to the daemon backing that particular mount. This userspace-kernelspace-userspace-kernelspace–userspace overhead, you will see, is a big deal.
• zfs and zpool, the main management commands that use IPC to talk to zfs-fuse.
• two others that you won’t care and I won’t care either.

A cursory inspection with such important system binaries was in order, so I ldd the daemon and the commands. zfs-fuse links to /usr/lib/libz*.so*. Not good, chicken and egg problem, linking to a library in a filesystem that will not be available before zfs-fuse is running? I rebuild it using a modified SConstruct file so it statically links that library in.

I had decided that my filesystem layout would be:

• 1 GB swap partition on each disk
• 1 GB / filesystem, composed of two RAID1 partitions (one on each disk), formatted with ext3 (in case of catastrophe, it’s nice to have something the kernel can boot without initial RAM disks)
• 398 GB ZFS volume, where I planned to drop /usr, /home and /var

But I didn’t have extra hard disks to make the switch. No problem, croupier, everything I have on red please — and spin that wheel! I installed ZFS directly on my running system. How did I do it? Well, if you must know:

• I offlined the second disk with mdadm.
• I swapoff‘ed its swap partition. At this point the disk is no longer busy.
• I repartitioned the disk (if the disk is non-busy, the kernel rereads the partition table just fine).
• Then I relied on the first all the time.

Yes, realtime no-boot filesystem switchover — or at least I thought it would be that easy (I was very wrong).

Then I mkfs.ext3ed the new 1 GB root filesystem, and mkswap‘ed the swap one. A couple of rsyncs later (which I scripted for consistency and repeatability), I had a new, working /. I mounted it and went in it, to remove mdadm.conf and lvm.conf lines that could prove problematic on next boot. At this point I was panicking because it was superstitiously conceivable that, after a reboot, md would want to rebuild the arrays and destroy the second disk.

I then copied the ZFS binaries in /sbin and ran it. A cursory lsof inspection told me that the ZFS socket was on /etc/zfs/zfs_socket. zpool create quickly gave me the 392 GB of disk space that were previously empty in the second disk, in which I created subvolumes, with adjusted mount points to end up under a temporary tree structure under /newfs. Curiously, after creating a subvolume, it’s not mounted, but a zfs mount -a works as you probably would expect.

I enabled compression in the root volume (subvolumes inherit attributes) and started rsyncing /var, /usr and /home into each subvolume. Cue the movie 32 hours later to have an idea of how slow it was. It was unbelievably slow — un-frigging-believable, with both CPUs nearly pegged and regularly hovering at 150% combined user+system. The worst part is, I was seeing disk throughput in the 2-3 MB/s range, using iostat 1 and zpool iostat 1. Keep in mind that performance (high write throughput, low responsiveness/latency during massive reads) is marketed as a ZFS selling point — and I don’t doubt the Sun guys… on Solaris, not Linux!

During that lengthy process I started finding out several things that would prove crucial later on:

• FUSE does not support mmap in the Linux kernel that my distribution uses. Many, many applications rely on that feature to work.
• There was no initscript for ZFS. I would have to write an initscript from scratch. On Kubuntu, where initscripts are being (1) phased out and (2) completely different to my beloved RPM distros.

At this point I was a bit nervous, if you’ll allow me to understate. But I wrote the initscript anyway:

#! /bin/sh

BEGIN INIT INFO

Provides: zfs

Required-Start: mountall

Required-Stop: sendsigs

Should-Start:

Should-Stop:

Default-Start:

Default-Stop:

Short-Description: Enable/disable the ZFS-FUSE subsystem

Description: Control ZFS-FUSE subsystem

END INIT INFO

PIDFILE=/var/run/zfs-fuse.pid LOCKFILE=/var/lock/zfs/zfs_lock

. /lib/init/vars.sh

. /lib/lsb/init-functions . /lib/init/mount-functions.sh

export PATH=/sbin:/bin unset LANG ulimit -v unlimited

do_start() { test -x /sbin/zfs-fuse || exit 0 PID=cat "$PIDFILE" 2> /dev/null if [ "$PID" != "" ] then if kill -0 $PID 2> /dev/null then echo "ZFS-FUSE is already running" exit 3 else # pid file is stale, we clean up shit log_action_begin_msg "Cleaning up stale ZFS-FUSE PID files" rm -f /var/run/sendsigs.omit.d/zfs-fuse "$PIDFILE" log_action_end_msg 0 fi fi

pre_mountall

log_action_begin_msg "Starting ZFS-FUSE process"
zfs-fuse -p "$PIDFILE"
ES_TO_REPORT=$?
if [ 0 = "$ES_TO_REPORT" ]
then
    true
else
    log_action_end_msg 1 "code $ES_TO_REPORT"
    post_mountall
    exit 3
fi

for a in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
do
    PID=`cat "$PIDFILE"`
     [ "$PID" != "" ] && break
    sleep 1
done

if [ "$PID" = "" ]
then
    log_action_end_msg 1 "ZFS-FUSE did not start or create $PIDFILE"
    post_mountall
    exit 3
else
    log_action_end_msg 0
fi

log_action_begin_msg "Immunizing ZFS-FUSE against OOM kills and sendsigs signals"
mkdir -p /var/run/sendsigs.omit.d
cp "$PIDFILE" /var/run/sendsigs.omit.d/zfs-fuse
echo -17 > "/proc/$PID/oom_adj"
ES_TO_REPORT=$?
if [ 0 = "$ES_TO_REPORT" ]
then
    log_action_end_msg 0
else
    log_action_end_msg 1 "code $ES_TO_REPORT"
    post_mountall
    exit 3
fi

log_action_begin_msg "Mounting ZFS filesystems"

zfs mount -a
ES_TO_REPORT=$?
if [ 0 = "$ES_TO_REPORT" ]
then
    log_action_end_msg 0
else
    log_action_end_msg 1 "code $ES_TO_REPORT"
    post_mountall
    exit 3
fi

if [ -x /usr/bin/renice ] ; then
    log_action_begin_msg "Increasing ZFS-FUSE priority"
    /usr/bin/renice -15 -g $PID > /dev/null
    ES_TO_REPORT=$?
    if [ 0 = "$ES_TO_REPORT" ]
    then
        log_action_end_msg 0
    else
        log_action_end_msg 1 "code $ES_TO_REPORT"
        post_mountall
        exit 3
    fi
    true
fi

post_mountall

}

do_stop () { test -x /sbin/zfs-fuse || exit 0 PID=cat "$PIDFILE" 2> /dev/null if [ "$PID" = "" ] ; then # no pid file, we exit exit 0 elif kill -0 $PID 2> /dev/null; then # pid file and killable, we continue true else # pid file is stale, we clean up shit log_action_begin_msg “Cleaning up stale ZFS-FUSE PID files” rm -f /var/run/sendsigs.omit.d/zfs-fuse “$PIDFILE” log_action_end_msg 0 exit 0 fi

pre_mountall

log_action_begin_msg "Syncing disks"
sync
log_action_end_msg 0

log_action_begin_msg "Unmounting ZFS filesystems"
zfs unmount -a
ES_TO_REPORT=$?
if [ 0 = "$ES_TO_REPORT" ]
then
    log_action_end_msg 0
else
    log_action_end_msg 1 "code $ES_TO_REPORT"
    post_mountall
    exit 3
fi

post_mountall # restore /var/lock and /var/run to their right places

log_action_begin_msg "Terminating ZFS-FUSE process gracefully"
kill -TERM $PID

for a in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
do
    kill -0 $PID 2> /dev/null
    [ "$?" != "0" ] && break
    sleep 1
done

if kill -0 $PID 2> /dev/null
then
    log_action_end_msg 1 "ZFS-FUSE refused to die after 15 seconds"
    post_mountall
    exit 3
else
    rm -f /var/run/sendsigs.omit.d/zfs-fuse "$PIDFILE"
    log_action_end_msg 0
fi

log_action_begin_msg "Syncing disks again"
sync
log_action_end_msg 0

}

case “$1″ in start) do_start ;; stop) do_stop ;; status) PID=cat "$PIDFILE" 2> /dev/null if [ "$PID" = "" ] ; then echo “ZFS-FUSE is not running” exit 3 else if kill -0 $PID then echo “ZFS-FUSE is running, pid $PID” zpool status exit 0 else echo “ZFS-FUSE died, PID files stale” exit 3 fi fi ;; restart|reload|force-reload) echo “Error: argument ‘$1′ not supported” >&2 exit 3 ;; *) echo “Usage: $0 start|stop|status” >&2 exit 3 ;; esac

:

The script should explain itself.

There were two problems, though. I derived my script from the NFS one and, in the process, I discovered that NFS was symlinked to be started at slot 31 in level 6 and 0. This means that the initscripts subsystem would call that script with a start argument when in reality, the action was in the stop block. Since I couldn’t figure out what kind of magic the Upstart initscripts compatibility subsystem does to get a stop block to run when a start block is requested by its configuration, I just created two glue scripts: one to start ZFS no matter what, and one to stop ZFS no matter what:

-rwxr-xr-x 1 root root 481 2008-06-18 04:09 /etc/init.d/mountzfs
-rwxr-xr-x 1 root root 488 2008-06-18 04:09 /etc/init.d/umountzfs

Then I studied the Kubuntu boot sequence very carefully, and used some elbow grease (update-rc.d) to symlink them to get the results I wanted:

lrwxrwxrwx 1 root root 19 2008-06-18 03:52 /etc/rc0.d/S35umountzfs -> ../init.d/umountzfs
lrwxrwxrwx 1 root root 19 2008-06-18 03:52 /etc/rc6.d/S35umountzfs -> ../init.d/umountzfs
lrwxrwxrwx 1 root root 18 2008-06-18 03:52 /etc/rcS.d/S36mountzfs -> ../init.d/mountzfs

Trust me, writing the script was the easy part — figuring out how it interacts with the rest of the system was much harder.

Finally, I rebooted to my new root filesystem on the second disk. If you thought that my system booted correctly, you would be very, very wrong indeed. Eighty percent of the boot sequence were red [ fail ]s and sh: command not found errors. At the end, the system dropped me into a recovery console, where I could finally switch the ZFS mount points to their final destinations. Then, just to try out: zfs mount -a.

/homemounted. /var couldn’t be mounted, because the boot process graciously created incredibly important missing directories in it. And then, deadlock.

Crap, what was wrong?

Alt+SysRq+R. Boot again. What’s wrong? No idea. Try strace. The friggin’ command is in /usr. Hypotheses ran through my head for two hours. With me in front of a very, very broken system. I tried everything under the Sun that I could get my hands on — which is not much when you don’t have a CD-ROM drive, mind you.

And (summarizing two hours) then, I tried this: zfs set mountpoint=/tmp/usr vault/usr ; mkdir -p /tmp/usr ; zfs mount vault/usr.

Miracle of miracles, it worked. I copied the entire cast of characters of Linux Debugging: The Movie into the very tightly packed /. I strace --ffed the hell out of zfs-fuse and I found the problem. The moronic mount.fuse subcommand, that actually connects the kernel and user endpoints, tries to read /usr/lib/locale/locale-archive right in the middle of mounting the filesystem! Instant deadlock that you can only get out of by using the SysRq OOM key (yes, zfs-fuse is actually a great OOM candidate — 1.5 GB VM size on this 1.0 GB RAM computer; yes, I discovered that on my own before I wrote the OOM immunization code in the initscript).

I then discovered two things: zfs-fuse didn’t deadlock when started from the recovery command, but it did lockup when starting it from the initscript. What you can’t see is that the version of the initscript that I initially wrote was sourcing the LANG variable from a configuration script in /etc. OK, so how do you solve locale problems? Instant fixup: unset LANG before running the command.

OK, so do I have a booting system now, or what? Wrong again. Some processes get started before the actual mounting of filesystems, and the ZFS subsystem can’t actually be started earlier in the boot process without creating an initramfs dependency or another, different, chicken-and-egg problem. So I moved what I could move from the ZFS volume’s /var into the /var directory of the / filesystem. I ended up with this structure backed up by ZFS (and the rest, you can safely assume, in a very tightly crammed ext3 filesystem):

zfs list
NAME              USED  AVAIL  REFER  MOUNTPOINT
vault             294G  69,8G    18K  none
vault/home        290G  69,8G   290G  /home
vault/usr        3,36G  69,8G  3,36G  /usr
vault/var         842M  69,8G    18K  none
vault/var/cache   515M  69,8G   515M  /var/cache
vault/var/lib     282M  69,8G   282M  /var/lib
vault/var/tmp    44,5M  69,8G  44,5M  /var/tmp

Boot again. Oh, yeah, I’m enjoying the 3-minute boot time on this formerly-a-screamer machine. D-Bus fails to start. D-Bus is actually very required for many things in Kubuntu, but I manage to start a GUI session up, if only to Google up what was wrong with it. That was probably not the best moment to find out that just starting the KDE 3.5 session took over ten minutes. All of this with less than 1 MB/s from the disk, according to iostat and 160% CPU usage, according to top.

Then I discovered the zfs-fuse Google group. It’s a fantastic place where everyone (including Ricardo Correia) received me very well and had lots of tips. Only there did I find out what was wrong with D-Bus — a bug that manifests itself only with FUSE filesystems, for which a patch exists and works.

At this point I’m extremely exhausted from this marathon session, so I basically just try to backport the patch into the dbus source package for my distribution. You’ve probably heard that Debian (and, by extension, Ubuntu) has a fantastic build system — it failed on me. Not only was apt not working (remember the mmap issue?), but dpkg-source also failed while trying to apply the patches for the source package. Oh, yes, I manage to solve this problem by learning, on-the-spot, how the apt build “system” actually works, and manually replicating the entire process that should be automated. Many thanks to the gents at #debian in Freenode for their kind responses to my questions.

Bam, built dbus (it’s yours if you want it). Installed it. Started it. And the chain of daemons that were depending on it just start up and take life. Neat trick, Upstart!

Back to performance questions and ZFS. Do you know what the real performance killer is? You’ll never guess it…

…icons! While GTK+ applications take marginally more time to start under a ZFS regime, KDE applications take an order of magnitude more. Before, on a warm working set, a KDE application took about 2 seconds to start. Today, Kmail takes in excess of five minutes to start. Why? Here’s why — multiply that by fifty thousand and you’ll get the idea. Each icon that the application requests results in thousands and thousands of access() and stat() calls. FUSE doesn’t use a kernel cache by default (there are several reasons for that), so the only cache that backs those requests up is the ARC cache, which is an impressive caching regime and technical achievement but, in this case, it’s very much like caching your car keys somewhere in Europe, because of the transatlantic userspace-kernelspace-userspace-kernelspace-userspace barrier. Per-call. When this is taking place, the CPUs remain pegged at 190%, eaten by ZFS alive, and the 12 case fans jump to 11.000 RPM.

The zfs-fuse Google groups guys came up with a couple of suggestions (all documented in the list, which I’m too lazy to link to again). These all are compile-time options, so a ZFS rebuild is in order for every one of them:

• scons debug=0. A very slight CPU usage decrease.
• Increasing the ARC cache. I doubled it from 128 to 256 MB. Turns out it’s not a caching problem and it doesn’t help at all.
• Mount option big_writes for FUSE filesystems. Here’s what I did about that:

Recompiled ZFS, this time enabling a FUSE mount option named big_writes that I’ve read about in the Google group. Yes, the daemon needs to be recompiled, and it’s not fast. No, I’m not actually jumping to the part where I actually compiled ZFS with big_writes first, then booted, only to find out that I needed a new kernel. Oh, wait, I just did. Fortunately, I did back zfs-fuse up.

Next up? Latest 2.6.26-rc6 kernel, because of:

• Hey, writable mmap is there for FUSE filesystems! Yeah! Now I can have apt-get back!
• big_writes.

When was the last time a kernel compile took four hours for you? Mine was yesterday. But it’s actually fun — the process hasn’t changed that much from 1998, and the distro already comes with a nice .config that you can reuse with make oldconfig. And, this time, you get to do out-of-tree kernel builds! Yay!

Well, I ticked the wrong option in make menuconfig anyway, because my kernel modules don’t fit my puny /, now at 400 MB free. Jeez, four hours. Google some more. Turns out I turned a debugging option on.

After this, FUSE userspace itself was due for a recompile. Another odyssey, whose fruits you can reap here (warning: CVS checkout).

OK, redo the initial RAM disk, adjust GRUB configuration, reboot with the latest kernel. It’s all good. More surprisingly, I’m actually getting some of my performance back. Some of it. As in “Kmail no longer takes five minutes to start — only three”.

And, most importantly, applications that depend on mmap now work correctly. My boot process isn’t an epic [ fail ] anymore — and that’s incredibly reassuring.

This is the point where my journey turns into smooth sailing. I zpool scrubed my new baby. After five hours, with the solid guarantee that my data was OK and nothing’d been lost or corrupted during the rsync, I nuked my first disk, replicated the new partition structure on it. A nice RAID1 array for the final /. A short rsync for the / filesystem. A quick mkswap for the new swap partition. A fast adjustment in /etc/fstab and another one in mdadm.conf for the new array. Reinstall and reconfigure GRUB on the first disk. And, finally, I leave the best for the latest:

zpool attach vault /dev/by-id/second-disk-huge-partition /dev/by-id/first-disk-huge-partition

Man, that rocked. It was unbelievably fast — like, disk-platter fast, around 40 to 50 MB per second, and the system didn’t get that much more slow when it was resilvering the first disk. Which kind of makes lots of sense, because zfs-fuse is now crossing the userspace-kernelspace barrier just once per operation. How do I know this? Well, strace: I know that what zfs-fuse does is, it opens the disk partition in direct I/O mode and then manages it for itself, responding to FUSE requests — but the resilvering process doesn’t involve FUSE at all, it’s just the two disks practically chatting with each other through zfs-fuse. Now I know for sure that ZFS will give me platter speeds. It’s just a matter of time (and maybe me pestering Ricardo Correia to collaborate with me on this same issue).

Questions that I haven’t solved yet? Sure, there are a lot. Two that haunt me:

• No root filesystem on ZFS. Others on the Google group have managed it. Me? I didn’t want to mess with /etc/zfs inside the initramfs, thank you very much.
• I know this for sure: the only active cache now is the userspace ARC cache from ZFS; I read the FUSE kernel code, and it clearly flushes files from the cache when programs open() them. Honestly, if I could wish for something to just become true overnight, I’d wish for the ARC to be moved into the kernel and to have it replace the page cache, but that won’t happen anytime soon. There’s a FUSE kernel_cache option, but I’m wary of enabling it. When I have been sufficiently reassured that the option won’t corrupt my precious data, I will enable it. That will be a couple of hours of reading someone else’s code, so I’m inclined to defer it for a few days. But, in theory, this should give me platter speeds instead of giving my 12 case fans ’speed’. At the hefty cost of RAM for two redundant caches.
• Do filesystem readahead and Linux disk scheduler algorithms mess up in some way with ZFS’ control of the platter? The data integrity question is closed, because the writes are submitted with barriers, but I’m worried that the Linux I/O scheduler is second-guessing the decisions of ZFS’ one.
• The /etc/init.d/sendsigs omit.d protocol I’m using on the initscript plain fails. I had to shunt the script with an exit 0 right before the killall5 in sendsigs because killall5 plain hung instead of ignoring ZFS as it should have done — and it needs to ignore ZFS because ZFS is unmounted later. This won’t be a problem once we get our own kernelspace ZFS implementation.

OK, that was my journey. I’m on ZFS now, my machine’s rock-solid (if a bit CPU-tired) and my data’s never been so safe. I also got compression, which saved me about 6 GB. Furthermore, I’ve given you the initscript, the steps and the software (except ZFS, but you can compile that yourself).

Go wild.

What you’re about to see is just a twenty-line snippet representing standard pathological behavior of a widely used KDE application that comes with Kubuntu. I’m at a loss of words, because what you’re about to see repeats itself (I kid you not) at least a twenty thousand times:

access("/usr/share/icons/crystalproject/24x24/devices/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/24x24/filesystems/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/24x24/filesystems/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/24x24/mimetypes/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/24x24/mimetypes/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/32x32/actions/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/32x32/actions/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/32x32/apps/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/32x32/apps/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/32x32/devices/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/32x32/devices/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/32x32/filesystems/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/32x32/filesystems/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/share/icons/crystalproject/32x32/mimetypes/mail_todo.xpm", R_OK) = -1 ENOENT (No such file or directory)

Now, let me ask you: what kind of engineering is this? Oh, I’ll just look for the icon named mail_todo of types XPM, SVG, SVGZ, PNG, JPG across thousands of directories and if I can’t find it, I just won’t draw it.

Modern desktop environments package all that shit up in a single icon cache and use mmap to quickly find it. Well, guess again, loser who invented that band-aid. I’m using a filesystem that doesn’t support mmap. Now my filesystem has to consume inordinate amounts of CPU to find each one of those two hundred icons your application uses, because you couldn’t find a better algorithm to find the motherfucking icon.

Oh, by the way, every time I delete an e-mail from KMail’s listing, the entire process is repeated again. I guess I can safely omit the part where Kicker (the panel) and Kopete all do the same things for each icon that they display. Woops, I didn’t omit it.

Just another tale of living in the bleeding edge of Linux computing.

In the interest of full disclosure: I’m using the CPU-heavy ZFS filesystem from Sun. I’m testing it because it offers unsurpassed reliability. Yes, it’s CPU-heavy. Yes, it runs through FUSE. But that’s not the point — the point is that this wouldn’t matter if KDE apps (and, I’m sure, many others as well) wouldn’t suck so much to find a motherfucking icon. After the latest torvalds kernel recompile and another ZFS recompile for performance (which failed), I have gained quite a lot of performance: now KMail doesn’t take 10 minutes to launch — it only takes four. On a dual-core Xeon, with 1 GB RAM and 7200 RPM dual platters. Four motherfucking minutes. I now fully agree with Cox when he said userspace does stupid things.

Proposed solution: make PackageKit do exactly what synaptic has been doing for the past three years: when a package install process blocks on file descriptor 0, unhide a hidden VTE widget. As it stands now, none of the publicly facing Ubuntu or Debian PacketKit pages have proposed any other solution to the problem, and it appears there’s no other proposed solution on the mailing list. (Take care to note that this has nothing to do with debconf.)

I absolutely disagree with you.  This goddamned practice of letting packages interrupt the upgrade process willy-nilly on the packager’s whim is idiotic and should have dealt with years ago (preferably with death by fire).  Don’t tell me that the Debian project had a breakthrough invention when they “invented” the “hey, let’s interrupt package installation to ask the user a dpkg-configure question or to run a postinstall script”. That is not a breakthrough; it’s a practice that should have long been abandoned.

The reason why dpkg has this brain damage is pretty clear: the installation/upgrade process and the post-install configuration process aren’t clearly separated. It boils down to a single question: it’s fine by me if you want to include a package that needs extra user input for configuration — so, why don’t you ask me for the configuration at first-run time, and ship a package with sane defaults?

The RPM take on this issue has a huge advantage: by limiting user interaction during upgrades/installations, it forces packagers to distribute sanely preconfigured packages. That’s the reason you can install Dovecot, Postfix, and SquirrelMail on Fedora, and have a working mail server right out of the box — save, perhaps, for changing the listening interface setting on the Postfix configuration file.

The Debian way, on the other hand, creates a moral incentive for the packager to punt configuration questions to the user. Honestly, you can’t expect end-users to interact with a shell. The minute I show my dad a Synaptic VTE question, is the minute my dad stops using Linux.

This is one of the things that suck big time in dpkg. RPM-based distros don’t have this problem. On Fedora, I can make an unattended upgrade and it will work 100% of the time, knowing that the configuration files won’t be replaced but I will have a way to diff those configuration files. Not so with Ubuntu. Not so with Debian. I’m being candid here: I’ve had to answer tens of questions after a Debian dist-upgrade. I’ve never had to answer a single question after a smart upgrade. Yes, I know I could tell dpkg-configure not to ask me questions, but I wonder when Debian will default to that behavior.

What’s more, the RPM way has a fantastic consequence: upgrades are fully transactional. You can’t make a transactional upgrade if you’re interrupting the upgrade process by opening stdin to ask bullshit questions or other hackery. Honest, the need to run apt-get -f install bullshit is something I’ve never, ever in my life experienced with Fedora’s or Red Hat’s RPM. And it’s a weekly fucking occurrence in Ubuntu!

Just now, I had temporarily turned Nagios2 off with /etc/init.d/nagios2 stop, and the goddamned apt-get -f install won’t finish because it insists on dpkg-configuring Nagios2, and the dpkg-configure process fails because it attempts to shut down Nagios2 and that fails, leaving my system in a circle-jerk of partial configuredness, that I wouldn’t know how to fix if I didn’t have ten years of Linux experience.

Oh, and for the record, the other reasons are:

1. You can’t have two versions of the package with the same name installed in the system (I have visited this argument before). Not very used in the RPM world either, but I have put that use into practice.
2. You can’t do an rpm -Va, and have dpkg check for permissions, dates, times, and file sizes/sums. What the hell were the dpkg developers thinking? This is practically the first thing I use to audit a server right before I take it under my wing, just to get a clear idea of what the people before me did with it.
3. When you remove a package with dpkg, the configuration files remain unless you purge it. Why can’t dpkg do like RPM — erasing configuration files unless they were modified by the user? Two different remove operations are one too many.
4. dpkg happily lets you destroy your system.

And, quoting you:

This has the added benefit of making it possible for any user on any distro. to unhide the VTE window when something goes wrong–and we all know that something goes wrong with package managers all the time.

I dunno about your experiences with Red Hat or Fedora, but when I use a GUI package manager on Fedora, and something goes wrong, what I’m interested in is the output of stderr. All GUI package managers neatly show these messages nowadays. And I rarely see any error dialog box with stderr contents. So don’t start with the “something goes wrong with package managers all the time” bullshit.

So, let’s amend your last paragraph: let’s make PackageKit redirect /dev/null into the dpkg subprocess. That should take care of things neatly, and screw all braindead packages out there. What’s more, if the package somehow refuses to continue installing, let’s send dpkg a SIGKILL after a while, then prompt the user to change to a non-braindead distribution. So PackageKit doesn’t support your favorite brain damaged behavior? Sucks to be you.

Jeez, how much more of this obsolescent crap and stupid policy do we have to swallow, just because it’s the policy of the most widely used distro? I thought open source and free software were about meritocracy, not the tyranny of the majority. And I wish that the portion of the brain of the developer that conceived this “installation is interruptible” brain damage would just necrotize thanks to an aneurysm.

These packages are built from the source code of the next release of Ubuntu, Hardy. They work flawlessly on my home machines (yes, plural). I built these packages myself for my usage:

• libpulse-browse0_0.9.8-1ubuntu3_i386.deb
• libpulse-mainloop-glib0_0.9.8-1ubuntu3_i386.deb
• libpulse0_0.9.8-1ubuntu3_i386.deb
• libpulsecore5_0.9.8-1ubuntu3_i386.deb
• pulseaudio-esound-compat_0.9.8-1ubuntu3_i386.deb
• pulseaudio-module-gconf_0.9.8-1ubuntu3_i386.deb
• pulseaudio-module-hal_0.9.8-1ubuntu3_i386.deb
• pulseaudio-module-lirc_0.9.8-1ubuntu3_i386.deb
• pulseaudio-module-x11_0.9.8-1ubuntu3_i386.deb
• pulseaudio-module-zeroconf_0.9.8-1ubuntu3_i386.deb
• pulseaudio-utils_0.9.8-1ubuntu3_i386.deb
• pulseaudio_0.9.8-1ubuntu3_i386.deb

Download them all in a folder, then just open a terminal window there and type:

sudo dpkg -i *deb

That should upgrade you to the latest PulseAudio. I was gonna write a tutorial, but then you would have had to download hundreds of megabytes of development packages.

Fast-forward to my contemporary Kubuntu system. I just installed (using dpkg -i) a series of packages, and dpkg happily unpacked them all on my filesystem, only to nonchalantly tell me — when the files were already replaced on my (now very broken) computer — that it could not configure these packages, because the version of my C library is too old.

With the new files are already on my system! A tad too late to blow up on my face, isn’t it? Now I have a badly broken system, and I can’t back out without resorting to manually downloading and installing old packages.

Question to the dpkg developers: If the version of my C library is too fucking old — and dpkg knows this fact — why the fuck does dpkg proceed with file installation, only to barf in the middle of the installation process, leaving me with a broken computer? If the dependencies don’t fucking fit, don’t fucking let me install the packages in the first place!

It’s not like transactional package installation is rocket science to get right, is it? Otherwise, how did the RPM guys figure this out so well?

In b4 “dpkg is not like RPM” comments:

1. Software installation transactionality is an elementary requirement of contemporary systems, so get your damn act together and make dpkg like RPM with regards to this issue.
2. Even better: ask yourselves the following question: in what universe is this dpkg behavior (namely, enabling users to badly break their systems) a good thing?

And, please, let me rant some more, this time on the topic of package dependencies. Why does a simple 100 KB program from Hardy require me to upgrade my system C library (hence, my entire operating system) to Hardy? I thought glibc hadn’t changed in years! I hardly doubt that 100 KB program would have any trouble running if I unpacked the binary directly! So why does the package builder automatically require the latest version of the library — forcing me to download gigabytes of packages — when an old one would work just as well?

I’m a sysadmin, for fuck’s sake, I want my computer to be easily manageable, not harder to manage. What are you waiting for?

sudo sed -i 's/feisty/gutsy/g' /etc/apt/sources.list && sudo apt-get update && sudo apt-get dist-upgrade

If at some point any of those processes fail, you will be returned back to the console. dist-upgrade will require your confirmation, so don’t just take off before the confirmation question.

Awesome awesomeness: start your downloads minutes after TorrentFlux finishes them

I’ve created a very, very complicated and interwoven (translation: awesome and straightforward) script that you may use and modify on your home computer. It works like this:

• Using SSH, it queries your TorrentFlux server’s transfer list, asking which torrents are completely downloaded.
• For each of the completed torrents, it grabs the file name of the downloaded file.
• Then, it uses rsync to incrementally transfer the downloaded file to your PC.
• Again using rsync, it removes the file from your TorrentFlux server, freeing disk space.

What you need to do before using the script

For you to use it, you will need to do the following:

• Enable SSH access to your (soon-to-be) TorrentFlux server.
• Set up public key (passwordless) authentication so that your home computer’s user account can freely log in to your server without asking for a password.
• Install TorrentFlux-b4rt and set it up so that your SSH user has read/write access to the torrents and downloads folders/files. Umask/permissions, you know the drill.
• Enable fluxcli usage on your TorrentFlux setup.
• Install rsync on both your server and your home computer.
• Install the BitTorrent package that contains the program named torrentinfo-console on your server.
• Create a destination directory on your home computer.
• Install a mail program such as mailx on your home computer.
• Install cron on your home computer.
• Configure the script altering the variables at the top.

Yeah, it’s a lot of work. Go bitch somewhere else, we’re here for the fun.

Installing the script

Put the script somewhere in your PATH. Make it executable. Set up a cron job for the purpose:

# m h  dom mon dow   command
0,10,20,30,40,50 * * * * /home/rudd-o/bin/torrentleecher -q

The script itself

It’s a stunning, hackish example of subprocessing, SSH remoting, pipeline integration, text processing, POSIX daemonizing and file locking, and samizdat logging. It gets almost all of them wrong, and yet it manages to stay standing. If you’re interested in reading a complete account of how this script grew out of nothing, well, click here.

It’s a miracle:

#!/usr/bin/env python

from subprocess import Popen,PIPE,STDOUT,call import fcntl import re import os import sys import signal

FIXME: this script doesn't deal with multifile torrents

vars!

wherever they are used, it's MOST LIKELY they need quoting

FIXME whatever calls SSH remoting needs to protect/quote the commands for spaces or else this might turn out to be a bitch

torrentflux_base_dir = "/var/www/html/torrents/torrents" torrentflux_download_dir = "/var/www/html/torrents/torrents/incoming" torrentflux_server = "yourserver.com" torrentleecher_destdir = "/home/rudd-o/download/" fluxcli = "fluxcli" torrentinfo = "torrentinfo-console" email_address = "rudd-o@awesome.com"

def getstdout(cmdline): p = Popen(cmdline,stdout=PIPE) output = p.communicate()[0] if p.returncode != 0: raise Exception, "Command %s return code %s"%(cmdline,p.returncode) return output def getstdoutstderr(cmdline,inp=None): # return stoud and stderr in a single string object p = Popen(cmdline,stdin=PIPE,stdout=PIPE,stderr=STDOUT) output = p.communicate(inp)[0] if p.returncode != 0: raise Exception, "Command %s return code %s"%(cmdline,p.returncode) return output def passthru(cmdline): return call(cmdline) # return status code, pass the outputs thru def getssh(cmd): return getstdout(["ssh","-o","BatchMode yes","-o","ForwardX11 no",torrentflux_server] + [cmd]) # return stdout of ssh. doesn't return stderr def sshpassthru(cmd): return call(["ssh","-o","BatchMode yes","-o","ForwardX11 no",torrentflux_server] + [cmd]) # return status code from a command executed using ssh def mail(subject,text): return getstdoutstderr(["mail","-s",subject,email_address],text)

def get_finished_torrents(): stdout = getssh("%s transfers"%fluxcli) stdout = stdout.splitlines()[2:-5] stdout = [ re.match("^- (.+) - [0123456789.]+ MB - (Seeding|Done)",line) for line in stdout ] return [ match.group(1) for match in stdout if match ]

def get_file_name(torrentname): cmd = "LANG=C %s %s/.transfers/%s"%(torrentinfo,torrentflux_base_dir,torrentname) stdout = getssh(cmd).splitlines() filenames = [ l[22:] for l in stdout if l.startswith("file name...........: ") ] assert len(filenames) is 1 return filenames[0]

def dorsync(filename): cmdline = ["rsync","-avzP","--remove-source-files", "%s:%s/%s"%(torrentflux_server,torrentflux_download_dir,filename),"."] return passthru(cmdline)

def exists_on_server(filename): cmd = "test -f %s/%s"%(torrentflux_download_dir,filename) returncode = sshpassthru(cmd) if returncode == 1: return False elif returncode == 0: return True else: assert False # Not reached

def get_files_to_download(): torrents = get_finished_torrents() for t in torrents: yield (t,get_file_name(t))

def lock(): global f try: fcntl.lockf(f.fileno(),fcntl.LOCK_UN) f.close() except: pass try: f=open(os.path.join(torrentleecher_destdir,".torrentleecher.lock"), 'w') fcntl.lockf(f.fileno(),fcntl.LOCK_EX | fcntl.LOCK_NB) except IOError,e: if e.errno == 11: return False else: raise return True

def daemonize(): """Detach a process from the controlling terminal and run it in the background as a daemon. """ try: pid = os.fork() except OSError, e: raise Exception, "%s [%d]" % (e.strerror, e.errno)

    if (pid == 0):           # The first child.
            os.setsid()
            try: pid = os.fork()              # Fork a second child.
            except OSError, e: raise Exception, "%s [%d]" % (e.strerror, e.errno)

            if (pid == 0):   # The second child.
                    os.chdir("/")
            else:
                    # exit() or _exit()?  See below.
                    os._exit(0)      # Exit parent (the first child) of the second child.
    else: os._exit(0)                # Exit parent of the first child.

    import resource                           # Resource usage information.
    maxfd = resource.getrlimit(resource.RLIMIT_NOFILE)[1]
    if (maxfd == resource.RLIM_INFINITY):
            maxfd = 1024

    # Iterate through and close all file descriptors.
    for f in [ sys.stderr, sys.stdout, sys.stdin ]:
            try: f.flush()
            except: pass

    for fd in range(0, 2):
            try: os.close(fd)
            except OSError: pass

    for f in [ sys.stderr, sys.stdout, sys.stdin ]:
            try: f.close()
            except: pass

    sys.stdin = file("/dev/null", "r")
    sys.stdout = file(os.path.join(torrentleecher_destdir,".torrentleecher.log"), "a",0)
    sys.stderr = file(os.path.join(torrentleecher_destdir,".torrentleecher.log"), "a",0)
    os.dup2(1, 2)

    return(0)

sighandled = False def sighandler(signum,frame): global sighandled if not sighandled: print “Received signal %s”%signum # temporarily immunize from signals oldhandler = signal.signal(signum,signal.SIG_IGN) os.killpg(0,signum) signal.signal(signum,oldhandler) sighandled = True

def report_file_failed(filename): try: os.symlink(os.path.join(torrentleecher_destdir,”.torrentleecher.log”),”%s.log”%filename) except OSError,e: if e.errno != 17: raise #file exists should be ignored of course sys.stdout.flush() sys.stderr.flush() errortext = “”"Please take a look at the log files in %s”"”%torrentleecher_destdir mail(”Leecher: error — %s”%filename,errortext)

def report_file_done(filename): try: file(”%s is done”%filename,”w”).write(”Done”) except OSError,e: if e.errno != 17: raise #file exists should be ignored of course mail(”Leecher: done — %s”%filename,”The file is at %s”%torrentleecher_destdir)

def main(): if not ( len(sys.argv) > 1 and “-D” in sys.argv[1:] ): daemonize() os.chdir(torrentleecher_destdir) if not lock(): # we need to lock the file after the daemonization if not ( len(sys.argv) > 1 and “-q” in sys.argv[1:] ): print “Other process is downloading the file — add -q argument to command line to squelch this message” sys.exit(0) signal.signal(signal.SIGTERM,sighandler) signal.signal(signal.SIGINT,sighandler)

    print "Starting download of finished torrents"

    try:

            for torrent,filename in get_files_to_download():
                    # no point in doing anything if the file was removed, right?
                    # so we continue if the file doesn't exist
                    if not exists_on_server(filename): continue
                    print "Downloading %s from torrent %s"%(filename,torrent)
                    retvalue = dorsync(filename)
                    if retvalue == 0:
                            report_file_done(filename)
                            print "Download of %s complete"%filename
                            # TODO: make use of fluxcli to kill the torrent out of TF
                    else:
                            if retvalue == 20:
                                    print "Download of %s stopped -- rsync process interrupted"%(filename)
                                    print "Finishing by user request"
                                    sys.exit(2)
                            elif retvalue < 0:
                                    report_file_failed(filename)
                                    print "Download of %s failed -- rsync process killed with signal %s"%(filename,-retvalue)
                                    print "Aborting"
                                    sys.exit(1)
                            else:
                                    report_file_failed(filename)
                                    print "Download of %s failed -- rsync process exited with return status %s"%(filename,retvalue)
                                    print "Aborting"
                                    sys.exit(1)

            print "Download of finished torrents complete"

    except Exception,e:
            report_file_failed("00 - General error")
            raise

if name == “main“: main()

If you want to learn the process of how this script grew out of nothing, here’s the story.

This is only the latest development in a story that’s been chock full with disappointment:

An anonymous reader writes “One week ago this community discussed the apparent price advantage of Ubuntu Dell over Vista. The article linked to a Dell IdeaStorm page with the status: ‘Implemented.’ Today the status has changed on that page to ‘Reneged: Ubuntu Dell is $225 More Than Windows Dell.’ The full price of a Ubuntu Inspiron 1420N is indeed $50 cheaper than the identical hardware configuration with Vista — except that a $275 free upgrade to 2GB memory and a 160-GB hard drive is available for Windows only.”

OK, let’s see the entire story in perspective (almost six months of events in six bullet points, so excuse me if the list’s a bit superficial):

1. Dell gets pummeled in its own very IdeaStorm: “we want Linux”, the customers say.
2. Dell balks at proposal. Users angry. Digg uproar.
   Dell recants. Users happy.
3. Dell “starts shipping” the machines, except there are no machines to buy. Users furious. Digg uproar.
   Dell corrects the stocking issue. Users happy.
4. Select Dell warranty offerings not available for Ubuntu-based machines. Users angry. Digg uproar.
   Dell warranty offerings equalized. Users not so happy this time, but, meh.
5. Linux Dell machines more expensive than Windows machines. Users angry.
   Dell corrects pricing mistakes, now Linux machines are $50 less. Users happy again.
6. A free hard disk / RAM upgrade (with a market value of $275) is now not available with Linux machines. Effectively, the Linux machines are more expensive than the Windows machines again.

Dell: do you really think we’re idiots? We can clearly see that a hard disk, a RAM memory module, and a hardware warranty plan have nothing to do with Windows — it’s obvious you’re doing this to steer people away from buying Linux machines.

Yes, we know Microsoft is inches away from dumping you as a partner. We’d rather you admit it to us than have you continually find new ways to cheat on us customers.

Come on, you morons, haven’t you figured out already that every single move you make is under the scrutiny of millions of Linux users with above-average intelligence?

This is the perfect example of how a company completely loses customer loyalty.

Update: since this topic has generated quite the stir lately (even though it’s old) you should know that whenever I’ve had people (mostly close ones) ask me to recommend a laptop, I’ve always ended up recommending Dell ones; they do make good hardware. Not only that, but (to all the sufferers out there), I actually have a Dell PowerEdge 1521 as my workstation, so shut up already with the insults already.

The catch? Right after the purchase, I personally install Kubuntu Linux (unlike Windows, Ubuntu’s the gift that keeps on booting) on them, and hand an original Kubuntu LiveCD to the laptop’s owner; I’m betting many people do the same, so if Dell sees low Linux sales figures, now you know why.

My answer: yes, categorically.

The open source advantage: why you should be using open source applications

Well, there are at least two reasons:

1. Open source applications tend to be higher-quality. I don’t want to generalize, because I’ve test-driven tons of immature open source applications, but a great portion of them are very high-quality, and tend to improve with time, since everyone can “pitch in” and help develop applications further.
2. They’re (usually) free. Why pay money to “the man”, when you can save money and do better?
3. They respect standards better. You can easily move to Linux or other operating systems, because they respect standards and save their information in easy-to-use, easy-to-backup, easy-to-move file formats.

So, which applications?

Open source applications for Microsoft Windows number in the thousands. The most widely used have gotten great amounts of press coverage, are mature, stable and well-known:

• OpenOffice.org: this Microsoft Office killer has nearly everything to be the next king in office-suite land.
• Mozilla Firefox: unless you’ve lived under a rock for the past 5 years, it’s hard to believe you haven’t heard about it. Plainly said, it’s the best Web browser, ever.
• The VideoLAN client: it’s a straightforward, no-frills, fast media player. The amazing thing about VLC is that it supports lots and lots of video and audio formats — you can even play Flash videos with it(for example, downloaded from YouTube with the VideoDownloader Firefox plugin).
• Azureus: downloading torrents in your computer? Azureus is the best there is for Windows.

Chances are, you’re already using open source applications in your computer. Note I said applications — some portions of Microsoft Windows are direct descendants of other open source operating systems (such as FreeBSD). But the short list doesn’t stop there.

Where do I get them?

Three distinct projects aim to bring lots of originally-Linux open source applications to Windows:

1. Open Source Windows. Billed as This is the best Windows software that we know of. No adware, no spyware, just good software., it’s absolutely on the spot.
2. TheOpenCD. Instead of having to download each application, you can simply download the CD and start test driving them. Obviously, you can copy the CD as many times as you like, and give it to acquaintances, friends and family.
3. PortableApps. If you have an USB thumb drive, you’ll find immense value in this project; if you’re on the road, you no longer need to install and configure each application on the computer you’re about to use — just install them once in your thumb druve, then plug it on any Windows computer.

And the march to Windows continues!

KDE on Windows! Replace your Windows Explorer with a much more powerful desktop environment!

Actually, what gets me even more excited is the fact that Amarok, the best music player/manager ever to see the light, is going to be available on Windows as well. Amarok cleanly blows iTunes, Windows Media Player and Winamp out of the water.

I’ve open sourced my applications… what next?

Run them for a couple of days, maybe two weeks, and avoid running the applications they replaced (Internet Explorer, Outlook Express, and so forth).

Get comfortable with them. Most likely, you’ll be using them in Linux too. Take this opportunity to learn, effortlessly, more about computing. No one ever complained when they saw a résumé and noticed extra computing experience, right?

Finally, install Linux. I recommend KUbuntu, because it’s more similar to Windows than regular Ubuntu. Download the live CD, burn it to a CD, then boot from the CD. Run it for a while, see if you like it, install it — once you’ve installed it to your computer, you can use Adept (included in KUbuntu) to install your beloved open source applications.

Did I mention installing applications on KUbuntu is much easier than on Windows?

More Linux questions?

Have a Linux question? Why don’t you drop by our Month of Linux Answers and leave your question as a comment? I’ll get to it right away :-).