Tales from responsivenessland: why Linux feels slow, and how to fix that
Desktop performance on Linux computers has been a hot-button issue of late, and a source of longstanding fights among the Linux developers. Today, I want to show you how I boosted (and you can boost) desktop performance dramatically.
The problem
What’s the number one cause of desktop performance problems?
Let me answer that question with another set of questions:
- Have you noticed how slow your computer is after after unzipping a big file?
- Perhaps you can also notice this effect overnight as well — you return to your computer, and every open application takes a while before they behave snappily again.
- Do you use Amarok? Then you’ll notice this effect after Amarok has finished scanning your music collection.
This problem happens all the time. In fact, I would be willing to bet this is the cause most people find Linux to be slow.
The source of the problem
Here’s my chief finding: desktop performance has very little to do with schedulers and real performance. It has everything to do with disk caching and perceived performance.
Now, bear with me for a second, because what I’m about to explain will let you understand the following. Based on my limited understanding of the Linux kernel (please correct me if I’m wrong, especially about the terminology), there are two distinct cache types:
- a block cache: caches block device contents
- a inode/dentry cache: one layer above the block cache, caches directory entries and other filesystem-related things that cost more to look up than just block device contents
Problem number 1 is that the block cache competes with applications for RAM. Problem number two is that the block cache competes with the inode/dentry cache.
On a normal desktop Linux computer, when you’re short of RAM, Linux will aggressively try to free memory to enlarge the caches. It usually does this by paging out unused portions of applications to disk. This is not good.
Why? Since disks are slow, the following consequences follow:
Let’s get technical (skip this heading if you’re uncomfortable with technicalities)
For desktop performance, where workloads are unpredictable, responsiveness needs dictate that applications’ working sets should be in core as much as possible, because the probability of major page faults in desktop loads is much larger than on server loads.
Let me rephrase that: Screw data. Prioritize code.
In other words: it’s OK if your application stalls a bit while reading non-cached data from disk, because that isn’t going to make the application seem frozen for a couple of seconds. It’s not OK if the application freezes because it needs to execute code that was paged out to disk.
All in all, how Linux uses your RAM matters a lot for desktop performance
You can see where this is going: if you have 1 GB of RAM (or less) you unzip a 300 MB archive, and the Linux kernel will swap out portions of your applications to disk in an effort to accomodate both the 300 MB file and the blocks of the uncompressed files about to be written on disk. It will also shrink the inode/dentry cache.
Linux, of course, thinks it’s doing a good thing — the archive uncompresses, in effect, rather quickly because of this. You, of course, want to kick Linux in the nuts because while (and after) the uncompress took place, starting an application or opening a new browser window suddenly takes ages, and your open applications “blank out” for a couple of seconds before reacting to your clicks and keystrokes.
Think about it: Linux is catering to the needs of the uncompression program, and you are actively telling your computer you want to do other stuff while it happens. Linux starts to scramble between reading the archive, writing data to disk, and bringing the portions of the applications you’re using “back from the dead” (from swap). Guess who loses? If you guessed “the applications” or “you”, you’d have been right.
Even worse: suppose you want to browse your home directory while all of this happens. In addition to the fact that your file manager is probably half-on-disk by now, you’ve got the problem that Linux needs to re-read your home directory, since the inode/dentry cache has been squashed.
Not good. I posit that the default settings may be very good for workloads with many repeating tasks, but definitely ungood for desktop situations, when applications (and code paths) are diverse and respond only to the whims of the user.
What we want is perceived performance
Let’s take a file manager for example. Let’s focus on Konqueror (it’s a nice case study, and it’s a nice file manager). Suppose I hit the home button on my panel, which shows one of the (usually hidden and prelaunched) Konqueror instances and prompts it to browse my home directory.
If you have lots of RAM, it won’t be a problem — both Konqueror and the contents of your home directory will be in memory, so it’ll be blindingly fast. But if you’re rather short of memory, it’s a different matter — what happens next determines whether you feel your computer slow or fast.
If Konqueror has been paged out, it will appear to be frozen (or take longer to “start up”) for a couple of seconds, until Linux has paged necessary code paths in. If, on the contrary, my home directory has been evicted from the RAM cache, Konqueror will show up instantly and be responsive, while the home directory loads.
I’d much rather wait for the directory display than have to wait for Konq to unfreeze because it was paged out. The difference is that in the first scenario, I can close the window, use the menus, navigate among the window controls, change the URL, abort the operation; in the second case, I’m screwed until Linux decides to fully pagein whatever Konq needs.
That’s what I’m arguing for — perceived performance, not throughput. It matters to me that I can manipulate my file manager half-a-second after I’ve hit the home button. It doesn’t matter to me that, because of this preference, the home directory actually takes one second longer to finish displaying.
Variations of this pattern can be found everywhere: in file open dialogs, in multimedia applications with collection managers, basically everywhere an operation requires some sort of progress report.
The solution
There are two distinct and complementary measures we’ll take to solve this problem. Keep reading to find out about them.
Pages: 1 2
October 3rd, 2007 at 7:23
[...] a quick plug for a massively useful article I found on the Rud-o site. It works, my laptop is actually showing the letters as I type them, while I’m [...]
October 3rd, 2007 at 7:45
I wanted to ask in what distribution you ask… It’ll be interesting to compare how different dists tune those parameter. Maybe ubuntu is different on server/desktop ? may debian changes it through task select ? and how about comparing ubuntu to fedora/suse ?
October 3rd, 2007 at 8:11
Hey there Oz. Distros usually do not tune these parameters at all. At least that has been my experience in debian, fedora, red hat, centos and ubuntu. Suse I don’t use because I hate it with my guts.
October 3rd, 2007 at 13:31
I use Fedora 7, kernel 2.6.22, on a laptop with 1 gig of RAM.
At no point did it start paging anything to disk. Although swapping between applications during the disk writing is slow (more due to high CPU usage). My swappiness is the default 60, and I even tried with upping the dd size to 1500M.
Are you sure your kernel has been built with the preemptive patch applied? Could it be that you’ve got an older kernel, 512megs of RAM, etc?
October 3rd, 2007 at 22:26
Paul_one:
1 GB of RAM over here. Swapping was heavy. It’s not the dd that causes the swapping but the copying AFTER the dd instead. But I’m not using your kernel version so your results obviously may have varied.
October 4th, 2007 at 3:25
Although I do feel the slowness effect mentioned in my home machine (Mandriva 2007, old AMD Duron 800 with 384 MB RAM) I cannot notice it in my work system (Kubuntu 7.04, VMWare VM with 256 MB RAM, running on a P4 2.8 GHz). I didn’t notice any difference after changing swappiness to 1, because the system was responsive enough to start. I think I have to try it home.
October 4th, 2007 at 4:16
none: That’s because your 256 meg machine is always under a lot of memory pressure, so cache usage rarely goes high. 256 meg is just too little to run several applications effectively. However, I wish it was enough!
Of course, 3 GB machines won’t feel the effect either. The entire working set of tens of applications plus almost everything needed to run them fits in cache snugly, and it’s almost impossible to cause a cache eviction in machines so highly endowed.
October 4th, 2007 at 6:25
[...] have noticed this. Linux is tuned more for the server out of the box. I found this article at linuxtoday.com. This not only tunes vm.swappiness(which to me should ALWAYS be at zero) with [...]
October 4th, 2007 at 9:47
I’ve been running my P4-2.4 with 1 GB of RAM and no swap at all, for about two months now. Normal desktop use.
I have not run into any problems yet. Is there anything I should worry about, though?
October 4th, 2007 at 12:45
I thought the point about perception sometimes being more important than reality was especially interesting. I saw this in action when I solved my own responsiveness problem lately with the good old fashioned ‘nice’ command.
I am currently enrolled in an online masters degree program. While I am watching the downloaded lectures, I often want to do other things on the computer at the same time, usually related to the lecture I’m watching, sometimes not. The problem was whenever I would open a web page, the video and audio playback would get choppy for a couple seconds.
Well, I started increasing the priority of mplayer using the nice command, and the choppiness stopped. The weird thing is the web browser feels faster now too, even though I know with the analytical side of my brain that it actually slowed down. It’s just that when the choppiness was present, I paid a lot more attention to how long it took to download and render a web page. Now, my subconscious doesn’t see any difference to the computer between when a web page is loading and when one isn’t, leading it to conclude that the web browser is being more efficient now.
October 4th, 2007 at 14:18
Rudd-O, good work. You actually measured something and drew conclusions.
What you have found is the old ‘batch/interactive’ tension that has existed since the first TTY was hooked up back in the 50s. Good interactive performance is wasteful, in terms of context switches, and messes up the throughput numbers of batch processes.
Which is why we have client server! And why SMP is the sticking point (got a Core 2 with a single compromise kernel tuning?).
If you ever look at the kernel tuning differences between XP and Windows Server 200Y you will see the difference (look I need a day job, OK?) The server is not a good desktop and XP is not the best place for that .ASP server.
Which brings us to what the much vaunted distros are doing to earn their value added. Why, when you install, does the distro not ask if you want Workstation/Compromise/Server tuning settings? Perhaps that sort of question you only get asked in RHEL or SLES?
October 4th, 2007 at 14:58
Peter:
Interesting. But I haven’t found the batch/interactive dichotomy. If you look closely, what I have found is the swap/cache dichotomy.
For desktop performance, where workloads are unpredictable, responsiveness needs dictate that applications’ working sets should be in core as much as possible, because the probability of major page faults in desktop loads is much larger than on server loads.
The rest is just scheduler tuning. And this article in no way touches that — Linux’s scheduler is actually pretty good at letting batch jobs perform at their best, while giving good responsiveness if and only if there aren’t any page faults involved.
October 4th, 2007 at 19:45
It works fine on my 256M memory machine after changing vm.swappiness to 1. BTW: Fedora is much slower comparing to debian before tuning.
October 4th, 2007 at 23:17
[...] Tales from responsivenessland: why Linux feels slow, and how to fix that (tags: linux performance) [...]
October 4th, 2007 at 23:38
Thanks for writing this up. I’m going to give it a try over the next week and see how it goes.
I can tell you that Debian Etch uses vm.swappiness=60 and vm.vfs_cache_pressure=100 by default. Ubuntu Dapper Drake Server Edition does, too. So does RHEL/CentOS, and you can feel it, too, especially when you make an LTSP server out of it.
October 5th, 2007 at 2:35
Tips para aumentar el rendimiento en un Linux de escritorio [EN]…
Uso de un par de configuraciones para dar prioridad a las aplicaciones sobre la escritura en disco. La mejora para trabajar con el ordenador es muy significativa (lo he probado). Además, explica tecnicamente el porqué hay que hacerlo….
October 5th, 2007 at 12:05
Very nice article, I thought it was going to be some bashing of one system or the other, but you managed to explain a difficult concept in a simple way. This kind of tuning is usually done by making some rules like “twice as much ram you have for this, three times for that” because in the enviroment Linux has cultural heritage, multiuser and multimachines places, that even seasoned admins forget they can use the twinking for making their desktop a nicer monouser machine.
One interesting thing is we are used to expect a much higher perfomance of Linux than Windows in the same machine, and some ppl are dissapointed to make an artificial test and find that Linux isnt faster in every way. I have been around with OS/2 fans, and Windows NT fans back in the times, and everyone knew a trick that his favourite OS could do faster than the others to show the others and brag a bit. Sadly the flamewars usually dont help to discuss properly the issues like you managed to do about this particular one.
Maybe is time for some tweakUI like opensource project for tuning the kernel in a visual way, to show off some real Linux muscles.
October 5th, 2007 at 14:31
Thanks! Various other tools and parameters may help.
October 6th, 2007 at 8:13
[...] source This entry was posted on Saturday, October 6th, 2007 at 9:13 pm and is filed under Tutorial. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site. Tags: Tutorial. or this post. 1 views. Related Taking Puppy Linux for a WalkHow to relock your iPhone before the firmware updateSpeed Up Your Torrents With OnoWays to Increase Laptop Battery LifeThe definitive dual-booting guide: Linux, Vista and XP step-by-stepInstall Linux apps on your MacMac Tip: Zoom into any area on the screenSeven steps to a green data centerSeven Ways to Improve VistaHacking the Apple TVHow to Back up Your iPod — Without iTunesTen dangerous claims about smart phone securityVista on a stick: how to flash your OSThe Right Operating System for YouSeven ways to keep your search history privateBest way to Hide the Recycle Bin from the DesktopHow dangerous is Skype?Understanding the Major File Formats: WAV, MP3, and MIDIHow to: Windows and Ubuntu on One PCHow to Choose a Domain NameWhich type of digital camera is right for you?Cool stuff: Here’s how to build your own Yahoo PipeHow to run a legal copy of Vista for 120 days without activationHow to Make Windows Start Up FasterFinding love on the cell phoneFirewalls: Defending Your Home Network from AttackHow to Buy an MP3 PlayerAll About Windows Vista Sidebar GadgetsThe Top 100 Alternative Search EnginesShould I Shut Down My PC At Night?Get Better Portraits by Turning Your Camera Upside Down?Improve PC Performance: Trashing Useless SoftwareSharing Folders Over a Home NetworkMake Ubuntu look like OSXHow To Create A Favicon In Photoshop [...]
October 6th, 2007 at 8:26
Tales from responsivenessland: why Linux feels slow, and how to fix that…
[...][...]…
October 6th, 2007 at 10:52
We’ve been discussing just these sorts of tuning issues on LXer.com, although without specifics like this article goes into. Especially about how tunable mainframe OSs were in the past.
With IBM working on Linux, I wonder if their experience tuning will make it into the kernel…
A tuning tool would be a great idea. With all the attention that the scheduler has received recently, maybe such a tool will get the needed “itch scratching”.
October 6th, 2007 at 12:19
[...] vm.vfs_cache_pressure=50 Aby uzyskać więcej nformacji na ten teamt wejdź na strone pewnego bloggera.Według niego wydajność na desktopie można zwiększyć przez odpowiednie zagospodarowanie [...]
October 6th, 2007 at 19:26
[...] Tales from responsivenessland: why Linux feels slow, and how to fix that Desktop performance on Linux computers has been a hot-button issue of late, and a source of longstanding fights among the Linux developers. Today, I want to show you how I boosted (and you can boost) desktop performance dramatically. (tags: linux performance tuning desktop cache memory) [...]
October 7th, 2007 at 1:52
Interesting article.
I tried on my system (Fedora 7, 2.6.22.9, 2G RAM, X86_64), and I didn’t notice any slow down during the swapiness test. There was only a small speed improvement on the second test.
Perhaps these tweaks aren’t going to be that helpful for a system with so much RAM?
October 7th, 2007 at 6:35
[...] abhi_chavan wrote an interesting post today onHere’s a quick excerpt [...]
October 7th, 2007 at 7:19
[...] Tales from responsivenessland: why Linux feels slow, and how to fix that Desktop performance on Linux computers has been a hot-button issue of late, and a source of longstanding fights among the Linux developers. Today, I want to show you how I boosted (and you can boost) desktop performance dramatically. (tags: linux howto guide tutorial performance tweak desktop) [...]
October 7th, 2007 at 9:54
Tales from responsivenessland: why Linux feels slow, and how to fix that…
Is your system a little slow due to swapfile management? Try this solution….
October 7th, 2007 at 17:05
[...] synesthete wrote an interesting post today onHere’s a quick excerpt [...]
October 8th, 2007 at 12:36
[...] Zwei Zeilen Code für ein sich schneller anfühlendes Linux. Mal ausprobieren … [...]
October 9th, 2007 at 5:07
[...] Tales from responsivenessland: why Linux feels slow, and how to fix that [...]
October 15th, 2007 at 11:31
[...] I’m thrilled to have found an article describing a simple setting change that will make desktop Linux systems feel much more responsive as we use them. Keep in mind, I said desktop systems, and not servers. If you do try the advice [...]
October 19th, 2007 at 13:00
[...] Change following system parameters based on this explanation of cache issues. [...]
October 20th, 2007 at 20:10
[...] Why Linux feels slow, and how to fix that [...]
October 29th, 2007 at 1:01
I have my swappiness set to the default of 60, and I did the tests you said, but no application code ever got swapped to disk, so there was absolutely no difference between swappiness of 60 or 1.
Debian sid, 1gb ram, 512mb swap. Perhaps it is because my swap is so small (never felt the need for more).
October 30th, 2007 at 14:46
[...] 4.- Tune up your Linux and make it faster [...]
November 5th, 2007 at 1:48
Slowskies.
November 6th, 2007 at 6:31
Thanks for these geek tricks. I had a performance boost on my pclinuxos 2007 machine.
November 20th, 2007 at 18:36
I tried your 5 command test
sync
echo 3 > /proc/sys/vm/drop_caches
dd if=/dev/zero of=/tmp/testfile count=1 bs=900M
find / > /dev/null
cp /tmp/testfile /tmp/testfile2
on Gentoo GNU/Linux on a ASUS A6F notebook with 1 GB of RAM (graphics is shared) and patiently waited until commands finished running. I expected some drop in responsiveness, but unminimizing aplications, openning new tabs in Firefox, running new applications, listing directories etc, and everything responded instantly.
December 4th, 2007 at 21:23
[...] Tales from responsivenessland: why Linux feels slow, and how to fix that (tags: howto cache sysadmin desktop tuning performance linux) [...]
February 24th, 2008 at 11:28
Omg it worked. I have Debian Etch r 3 on a Pentium III 650 Mhz PC with 256 MB SDRAM. I was discouraged by the first test and after the I put in “sysctl -w vm.swappiness=1″. But after I did that I tried to run multiple things on my PC and it worked. OMG. The only stupid thing it did not work on was expectedly OOO.org. Hah! Now if you can speed that up…
March 30th, 2008 at 18:39
[...] a great article on speeding up the Linux Desktop here. Did the two things it suggested and the system is running faster than it ever did under [...]
March 31st, 2008 at 17:53
I am eager to try this! I use Ubuntu Gutsy and have only 736MB RAM, and have noticed that sometimes Ubuntu seems slower than it used to be… and the scenarios described seem to tell it exactly as I have seen it.
Thanks! Joel A. Burdick
July 18th, 2008 at 15:52
Good tips
You can speed up a slow dog
Or get a breed bred for speed http://puppylinux.org/home/overview