Dag-Erling Smørgrav thinks I’m a kook. He’s so convinced of it, he went the extra mile and published it all on the Web.

What’s this about? It’s about a rather unpleasant exchange of e-mails between the developers of Varnish and yours truly. A long tirade of e-mails showing the open source process gone bad. A discussion prompted by technical deficiencies in a product, that quickly turned into the personal and away from the professional.

Warning: technicalities ahead

Now, I apologize in advance because I’m going to dive a bit into technical matters. I must, because if I don’t do this, this entire post would amount to nothing more than unfounded accusatory drivel against the Varnish developers. If I’m going to bash them, at least I’m gonna do it soundly grounded on evidence and proof.

Why it all started

It all started a few days ago, when I was working for a customer who specifically requested an HTTP acceleration strategy for his server. Being that his site gets an average of 1 visit per second (translating into a 15-hits per second cascade on his rather powerful Web server), he was looking into this strategy as one of the alternatives to “make his site run faster”.

Varnish is marketed as an HTTP accelerator, much like Squid’s HTTP accelerator mode. What an HTTP accelerator is supposed to do is (on the surface) rather simple: it sits between the client and the Web server, inspecting HTTP requests and storing “cacheable” requests in a cache. The next time someone requests a file that is on the cache, the HTTP accelerator just serves the file from its cache, avoiding a costly trip to the Web server.

HTTP acceleration is brutally effective when done right. I myself have experienced a cascade of traffic on a rather puny server, which was weathered more or less successfully after a 15-minute outage thanks to the wonderful Squid HTTP accelerator mode. Then someone suggested I try Varnish. And so I did, because it was a reasonable alternative with a couple of unique features useful in their own right.

But Varnish is a useless piece of crap…

Until, of course, I discovered that Varnish, in its default configuration, is useless. My site wasn’t running any faster. My customer’s site wasn’t either. This was a completely out-of-whack, unexpected effect. Whereas I had experienced a tenfold increase in performance with Squid, Varnish was actually slowing our sites down.

It was time to whip out test gear.

[…after a couple hours of mind-numbing testing, reconfigurations and retries…]

And so, I discovered what the problem was. Whenever I tried hitting the Web server with my trusty wget companion, Varnish would work just fine. Whenever I used my Web browser… let’s just say Varnish was just making things slower than before.

In other words: Varnish would outperform the competition on synthetic benchmarks, but actually make the site run slower in real-life usage scenarios.

…because its default caching policy is braindead and useless…

The issue? Cookies. Once our Web sites had set cookies on my browser, Varnish would no longer cache them. In fact, every time a request was made when cookies were sent along, Varnish would just not cache the results.

For those of you who don’t know how cookies work: as soon as any resource on a Web site sets a cookie for the domain, the subsequent elements are requested using that cookie. It doesn’t matter if it’s /index.php or /something.js or /image.png that sets a cookie — all subsequently downloaded elements from the same domain are fed that cookie as well. Since Google Analytics sets the _utma and _utmz cookies on my site, no pages get cached, no images get cached, and no JavaScript scripts get cached.

Let me clear that up again: once a single resource on your site sets a cookie, all subsequent requests for that resource receive that cookie from your browser, until the cookie expires or gets cleared. This is not a crazy idea of mine — test it yourself using Firefox and Firebug — but rather an unfortunate consequence of the cookie system design that almost no one understands.

And, yep, that’s right; the default Varnish configuration does not cache requests with cookies. As you probably are already aware of, all decent medium-to-large Web sites set one or more cookies on their visitors’ browsers (think Google Analytics subscribers, for example). As a result, Varnish caches nothing on those sites. Zilch. Nada.

Picture this: an entire project devoted to providing a new, revolutionary (and admittedly sound) caching architecture… and its end product doesn’t cache shit! Talk about false advertising!

What use could I possibly find, then, for a non-caching cache?

But wait, there’s more: simply configuring Varnish to disregard cookies as caching differentiators doesn’t work, because then it eagerly caches everything, without regard to standard caching policies! So logged-in users experience all sorts of Web site malfunctions, stale pages, and related issues.

Of course, being the evil, malignant hacker that I am, always eager to find misuses of technology, I actually managed to configure Varnish (using complicated logic, pounded out of practically the thin air that Varnish documentation is) to ignore all cookies except for the login cookie. Now visitors on both my site and my customer’s site get cached pages, and logged-in users don’t experience malfunctions. And that was the absolute best compromise I could achieve, because their (utterly limited procedural) configuration language won’t allow for smart caching of static objects vs. serving fresh dynamic ones.

This strut wasn’t without cost: diagnosing and working around the problem took me three hours. Compare that to my Squid experience: fifteen minutes from installation to working cache (and that included wading through the extensive documentation!).

But I got to collect the proceeds from the customer. And my site is sort of functional, again. Who said evil hackery didn’t pay?

…and their developers act like spoiled rock star brats…

I reported this as a bug. In a matter of hours, the bug was nixed, claiming it’s a configuration problem, and diverting me to their mailing list. No engineering effort went into investigating this. And I had to subscribe to one of their lists (yes, yet another list to follow).

I reported this issue to their mailing list. They responded by saying that, basically, I was an abrasive person (they got that right), that I was wrong and deluded, that I had insulted them from the outset (not true until today) and that their software is perfect, since “they have developed it in close relation to real-world experience”.

They spent several e-mails instructing me on how wrong I was, yet never explaining why. That shouldn’t surprise anyone — they can’t possibly expect to explain why I’m wrong, because I’m not.

…and they have the audacity to call me a kook…

Adding further insult to injury, today, Dag-Erling had the audacity to insult me and call me a kook:

We’ve had our share of kooks on the Varnish mailing list. There was one who suggested that Varnish should in fact be a client-side proxy, not a server-side cache (we get the client-side proxy thing a lot, but the bit about throwing out the server-side cache was new), and another who tried to convince us that the features our users are asking for (and paying for) in 2.0 are “of no real use”. A recurring question is the one about how Varnish compares to <insert name of project with which Varnish has aboslutely zero feature overlap>. Recently, we had an unrepentent narcissist who kindly informed us that Varnish is “braindead and stupid”, and that I am a “rude thick-headed and thin-skinned individual” for trying to point out his errors and complaining about his language.

And he finishes his tirade with a particularly ridiculous omen:

In the meantime, all of these exchanges are lovingly archived for posterity by Mailman, Gmane, The Mail Archive and others. Perhaps, as the kooks grow older and wiser, they will come to regret their past indiscretions…

Oh, mighty Dag-Erling, thanks for generously pointing out the errors of my ways. I have been converted by your sacred river of knowledge.

Not.

Please note that the “unrepentent narcissist” link points to my own page. The “rude thick-headed and thin-skinned individual” phrase is mine — I’ll openly admit to it — why wouldn’t I if it weren’t true? According to Dag-Erling, I’m the equivalent of an evil, destructive person; therefore, factual statements I make have no value, because of their source. Notice how he frames the discussion in terms of “me insulting him” — when the truth is, I refrained from anything even close to insulting him up until my last e-mail, where I called him thick-headed and thin-skinned.

Hey, if I wanted to insult him, I could have told him he was a moron. I didn’t. Although in Dag-Erling’s parallel fictional universe, this paragraph could be construed as me insulting him.

The facts: why Varnish sucks (for potential implementors)

Well, Dag-Erling, guess what: I also have a blog, and with a little bit of clout, may I narcissistically remark. And here’s my very biased (yet fully grounded in fact) statement:

1. Varnish is a remarkable feat of engineering, with the potential to be a much faster replacement for Squid in HTTP accelerator mode.
2. However, it is muddied by a braindead and stupid configuration default policy and mechanism.
3. Varnish provides a default configuration that results in a slower site for 99% of the sites out there. Does your site set even a single cookie? You can forget about having it accelerated unless you’re willing to dig deep into the Varnish configuration language.
4. Varnish’s internal caching mechanism doesn’t obey even the minimum requisite client-side HTTP caching pragmas. It fails to obey other established caching headers, and support for them cannot even be implemented by end users through configuration, because there’s no mechanism to control cache behavior based on Web server HTTP headers — only on client headers. Want to prevent caching of files without an ETag response header? You’re screwed.
5. The documentation surrounding your project sucks. The manual page for VCL is full of ambiguities, and you haven’t even bothered to provide a state flow diagram for the configuration system.
6. Varnish refuses to start if your /tmp is mounted noexec. /es, that’s right, Varnish attempts to compile a “shared lib” and load it from /tmp (totally unexpected!). Discovering this was an excruciating experience because the startup script doesn’t give an indication, and the log files don’t either.
7. The final nail in the coffin is Varnish’s support system. At least Dag-Erling and Poul-Henning Kamp are abrasive thin-skinned individuals who can’t bear others pointing out the deficiencies in his project. Dag-Erling sucks at public relations, and Poul-Henning isn’t that far away either. You see, they’re too busy to actually take a minute to listen to his end-users because they’re always right, so fuck the users.

As it currently stands, I strongly recommend my 1000+-a-day readership against implementing Varnish. You are much better off going with Squid:

• Its default caching policy is sensible. It automatically caches static objects, whether cookies are sent with requests or not. It detects whether an object is cacheable by the presence of Last-Modified or Expires: headers, and uses an adaptive TTL algorithm much more suited for real-life workloads.
• Dynamic pages aren’t eagerly cached by Squid. This provides the best mix of cached static content for maximum performance, and freshest content for maximum visitor satisfaction.
• It’s tested, it’s reliable and it’s been used in multi-million-dollar serve farms, unlike Varnish.

Dag-Erling would like you to buy his propaganda that the Squid caching model is actually a client-side caching model, and tons of other yadda-yadda. And, you know, he might technically even be a bit right. But the bottom line is simple: if you have a dynamic site, Squid makes it go faster out-of-the-box, while Varnish doesn’t… unless you have a resident HTTP expert or many $$$ for hiring someone to set it up for you (hey, by the way, I’m for hire — after all, I did unbreak my Varnish setup sort-of-successfully). The rest of his argument is just a set of irrelevant technicalities.

You can talk architecture all you want; yes, Squid is slower, Varnish is faster. But Squid works, and Varnish is broken.

And that, in my book, is the definition of sucking. Hard.

But, hey, it’s their project. And if they stay on this path, it’ll be irrelevant.

Yes, it’s his software. He has the absolute right to steer the project where he wants. What he doesn’t understand is that contributors (kooks, as he likes to call them) have a right to voice their concerns, and that their input is valuable, no matter how narcissistic (I am) and abrasive (I’m not) their voices may be.

Dag-Erling and Poul-Henning: you’re the newcomers, the new kids on the block, and you have an obligation to excellence if you want to ever be top dogs. I’m sorry to tell you this, but a hard-to-configure HTTP accelerator that doesn’t accelerate by default… well, isn’t even wrong, and it doesn’t earn much respect from me.

Don’t expect to earn your users’ respect with that attitude, punk. Otherwise, tomorrow, when Varnish stops sucking so much, you may not have the clout to put it in the limelight.

Update: straight from the mailing lists — someone else is having the exact same problem I have, with major slowdowns as his penalty. I certainly hope his hacky fix (which is much like my own hacky fix) doesn’t cause his site to malfunction like mine did. That e-mail just confirms everything I said on this post.

Update 2: now Poul-Henning is drumming his own drum, framing me as if I had said that Varnish sucks because it doesn’t do what I want, and he’s the best thing since sliced FreeBSD, and I’m an idiot. Newsflash, Poul-Henning: Varnish doesn’t suck because it doesn’t do what I want — it sucks because it doesn’t do what it should.