Tales from responsivenessland: why Linux feels slow, and how to fix that
Tuning swappiness to prevent impromptu RAM hijacking
Swappiness is the name Linux kernel developers gave to the preference between paging applications out to disk and (in practice) shrinking caches. If it’s close to 0, Linux will prefer to keep applications in RAM and not grow the caches. If it’s close to 100, Linux will prefer to swap applications out, and enlarge the caches as much as possible. The default is a healthy 60.
The irony of this preference is that, in fact, paging an unused application out generally produces a net performance increment, since the cache really helps a lot when it’s needed — but this net performance increment translates to a net drop in perceived performance, since you usually don’t care whether a file uncompresses a few seconds later, but you do care (a lot) when your applications don’t respond instantaneously.
On a desktop computer, you want swappiness to be as close to zero as possible. The reason you want to do this (even though it might hurt actual performance) is because this will immunize your computer to memory shortages caused by temporary big file manipulations (think copying a big video file to another disk). The cache will still be as big as possible, but it won’t displace running applications.
Don’t believe me? If you have two gigabytes to spare on your / partition, then run the following commands in order:
sync echo 3 > /proc/sys/vm/drop_caches dd if=/dev/zero of=/tmp/testfile count=1 bs=900M find / > /dev/null cp /tmp/testfile /tmp/testfile2
Don’t do anything else while they run. Yes, they will take a while. Be patient. What this does is rather simple: creates a big 900 MB file, attempts to find all files on your file system, then copies the file one more time.
Finished? Now do something with your computer. Un-minimize your minimized applications, do something relating to your everyday work…
What happened? Excruciatingly slow, right? Did you notice how applications were hung for a while, while the hard disk light blinked frantically?
OK, let’s tune swappiness. As root, run:
sysctl -w vm.swappiness=1
and retry the commands above. Big difference, right?
That’s because, with swappiness turned down, the Linux kernel no longer attempts to enlarge the cache by paging applications out. Not unless you’re experiencing an extremely high memory shortage.
To make the change permanent, write vm.swappiness=1 on your /etc/sysctl.conf file.
Filesystem caches are more important than other caches
We’ve already established that the filesystem cache is important because, without it, file browsing goes extremely slowly as well. Now we’ll learn how to tell Linux that we want it to prefer inode/dentry cache to other caches.
Try this:
sync echo 3 > /proc/sys/vm/drop_caches dd if=/dev/zero of=/tmp/testfile count=1 bs=900M
sysctl -w vm.vfs_cache_pressure=100 find / > /dev/null cp /tmp/testfile /tmp/testfile2 time find / > /dev/null
sysctl -w vm.vfs_cache_pressure=50 find / > /dev/null cp /tmp/testfile2 /tmp/testfile3 time find / > /dev/null
rm -f /tmp/testfile /tmp/testfile2 /tmp/testfile3
This series of commands does basically what the former did, only this time it attempts to measure the time it takes to find all files on your disk — after creating big files. It does so twice — once with the default settings for cache preference, and once with settings heavily tilted to favor inode/dentry cache.
Check the output. Did the second run take less time? It definitely cut times in more than half on my desktop computer. When this knob is closer to 1, the kernel will prefer to protect the inode/dentry cache by freeing block cache first.
To make the change permanent, put vm.vfs_cache_pressure=50 on your /etc/sysctl.conf file.
Please experiment with this value. Values close to 100 provide no gain. Values close to zero can cause huge swap activity during big filesystem scans.
That’s it
Desktop Linux is quite good. However, there are certain situations in which it could improve. Fortunately, Linux comes right out of the box with the things you need to tune it for desktop performance.
Don’t miss out on the incredible performance gains you’ll feel. And feel free to subscribe to this site, if you want to continue learning.
A shameless plug
If you want to find out how your applications are behaving with regard to actual disk usage, let me recommend iogrind — one of Michael Meeks’ latest tools in the fight against the slowskys, a sort of Filelight for disk accesses. And if you have extensive ReiserFS knowledge, please help!
Pages: 1 2
October 3rd, 2007 at 7:23
[...] a quick plug for a massively useful article I found on the Rud-o site. It works, my laptop is actually showing the letters as I type them, while I’m [...]
October 3rd, 2007 at 7:45
I wanted to ask in what distribution you ask… It’ll be interesting to compare how different dists tune those parameter. Maybe ubuntu is different on server/desktop ? may debian changes it through task select ? and how about comparing ubuntu to fedora/suse ?
October 3rd, 2007 at 8:11
Hey there Oz. Distros usually do not tune these parameters at all. At least that has been my experience in debian, fedora, red hat, centos and ubuntu. Suse I don’t use because I hate it with my guts.
October 3rd, 2007 at 13:31
I use Fedora 7, kernel 2.6.22, on a laptop with 1 gig of RAM.
At no point did it start paging anything to disk. Although swapping between applications during the disk writing is slow (more due to high CPU usage). My swappiness is the default 60, and I even tried with upping the dd size to 1500M.
Are you sure your kernel has been built with the preemptive patch applied? Could it be that you’ve got an older kernel, 512megs of RAM, etc?
October 3rd, 2007 at 22:26
Paul_one:
1 GB of RAM over here. Swapping was heavy. It’s not the dd that causes the swapping but the copying AFTER the dd instead. But I’m not using your kernel version so your results obviously may have varied.
October 4th, 2007 at 3:25
Although I do feel the slowness effect mentioned in my home machine (Mandriva 2007, old AMD Duron 800 with 384 MB RAM) I cannot notice it in my work system (Kubuntu 7.04, VMWare VM with 256 MB RAM, running on a P4 2.8 GHz). I didn’t notice any difference after changing swappiness to 1, because the system was responsive enough to start. I think I have to try it home.
October 4th, 2007 at 4:16
none: That’s because your 256 meg machine is always under a lot of memory pressure, so cache usage rarely goes high. 256 meg is just too little to run several applications effectively. However, I wish it was enough!
Of course, 3 GB machines won’t feel the effect either. The entire working set of tens of applications plus almost everything needed to run them fits in cache snugly, and it’s almost impossible to cause a cache eviction in machines so highly endowed.
October 4th, 2007 at 6:25
[...] have noticed this. Linux is tuned more for the server out of the box. I found this article at linuxtoday.com. This not only tunes vm.swappiness(which to me should ALWAYS be at zero) with [...]
October 4th, 2007 at 9:47
I’ve been running my P4-2.4 with 1 GB of RAM and no swap at all, for about two months now. Normal desktop use.
I have not run into any problems yet. Is there anything I should worry about, though?
October 4th, 2007 at 12:45
I thought the point about perception sometimes being more important than reality was especially interesting. I saw this in action when I solved my own responsiveness problem lately with the good old fashioned ‘nice’ command.
I am currently enrolled in an online masters degree program. While I am watching the downloaded lectures, I often want to do other things on the computer at the same time, usually related to the lecture I’m watching, sometimes not. The problem was whenever I would open a web page, the video and audio playback would get choppy for a couple seconds.
Well, I started increasing the priority of mplayer using the nice command, and the choppiness stopped. The weird thing is the web browser feels faster now too, even though I know with the analytical side of my brain that it actually slowed down. It’s just that when the choppiness was present, I paid a lot more attention to how long it took to download and render a web page. Now, my subconscious doesn’t see any difference to the computer between when a web page is loading and when one isn’t, leading it to conclude that the web browser is being more efficient now.
October 4th, 2007 at 14:18
Rudd-O, good work. You actually measured something and drew conclusions.
What you have found is the old ‘batch/interactive’ tension that has existed since the first TTY was hooked up back in the 50s. Good interactive performance is wasteful, in terms of context switches, and messes up the throughput numbers of batch processes.
Which is why we have client server! And why SMP is the sticking point (got a Core 2 with a single compromise kernel tuning?).
If you ever look at the kernel tuning differences between XP and Windows Server 200Y you will see the difference (look I need a day job, OK?) The server is not a good desktop and XP is not the best place for that .ASP server.
Which brings us to what the much vaunted distros are doing to earn their value added. Why, when you install, does the distro not ask if you want Workstation/Compromise/Server tuning settings? Perhaps that sort of question you only get asked in RHEL or SLES?
October 4th, 2007 at 14:58
Peter:
Interesting. But I haven’t found the batch/interactive dichotomy. If you look closely, what I have found is the swap/cache dichotomy.
For desktop performance, where workloads are unpredictable, responsiveness needs dictate that applications’ working sets should be in core as much as possible, because the probability of major page faults in desktop loads is much larger than on server loads.
The rest is just scheduler tuning. And this article in no way touches that — Linux’s scheduler is actually pretty good at letting batch jobs perform at their best, while giving good responsiveness if and only if there aren’t any page faults involved.
October 4th, 2007 at 19:45
It works fine on my 256M memory machine after changing vm.swappiness to 1. BTW: Fedora is much slower comparing to debian before tuning.
October 4th, 2007 at 23:17
[...] Tales from responsivenessland: why Linux feels slow, and how to fix that (tags: linux performance) [...]
October 4th, 2007 at 23:38
Thanks for writing this up. I’m going to give it a try over the next week and see how it goes.
I can tell you that Debian Etch uses vm.swappiness=60 and vm.vfs_cache_pressure=100 by default. Ubuntu Dapper Drake Server Edition does, too. So does RHEL/CentOS, and you can feel it, too, especially when you make an LTSP server out of it.
October 5th, 2007 at 2:35
Tips para aumentar el rendimiento en un Linux de escritorio [EN]…
Uso de un par de configuraciones para dar prioridad a las aplicaciones sobre la escritura en disco. La mejora para trabajar con el ordenador es muy significativa (lo he probado). Además, explica tecnicamente el porqué hay que hacerlo….
October 5th, 2007 at 12:05
Very nice article, I thought it was going to be some bashing of one system or the other, but you managed to explain a difficult concept in a simple way. This kind of tuning is usually done by making some rules like “twice as much ram you have for this, three times for that” because in the enviroment Linux has cultural heritage, multiuser and multimachines places, that even seasoned admins forget they can use the twinking for making their desktop a nicer monouser machine.
One interesting thing is we are used to expect a much higher perfomance of Linux than Windows in the same machine, and some ppl are dissapointed to make an artificial test and find that Linux isnt faster in every way. I have been around with OS/2 fans, and Windows NT fans back in the times, and everyone knew a trick that his favourite OS could do faster than the others to show the others and brag a bit. Sadly the flamewars usually dont help to discuss properly the issues like you managed to do about this particular one.
Maybe is time for some tweakUI like opensource project for tuning the kernel in a visual way, to show off some real Linux muscles.
October 5th, 2007 at 14:31
Thanks! Various other tools and parameters may help.
October 6th, 2007 at 8:13
[...] source This entry was posted on Saturday, October 6th, 2007 at 9:13 pm and is filed under Tutorial. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site. Tags: Tutorial. or this post. 1 views. Related Taking Puppy Linux for a WalkHow to relock your iPhone before the firmware updateSpeed Up Your Torrents With OnoWays to Increase Laptop Battery LifeThe definitive dual-booting guide: Linux, Vista and XP step-by-stepInstall Linux apps on your MacMac Tip: Zoom into any area on the screenSeven steps to a green data centerSeven Ways to Improve VistaHacking the Apple TVHow to Back up Your iPod — Without iTunesTen dangerous claims about smart phone securityVista on a stick: how to flash your OSThe Right Operating System for YouSeven ways to keep your search history privateBest way to Hide the Recycle Bin from the DesktopHow dangerous is Skype?Understanding the Major File Formats: WAV, MP3, and MIDIHow to: Windows and Ubuntu on One PCHow to Choose a Domain NameWhich type of digital camera is right for you?Cool stuff: Here’s how to build your own Yahoo PipeHow to run a legal copy of Vista for 120 days without activationHow to Make Windows Start Up FasterFinding love on the cell phoneFirewalls: Defending Your Home Network from AttackHow to Buy an MP3 PlayerAll About Windows Vista Sidebar GadgetsThe Top 100 Alternative Search EnginesShould I Shut Down My PC At Night?Get Better Portraits by Turning Your Camera Upside Down?Improve PC Performance: Trashing Useless SoftwareSharing Folders Over a Home NetworkMake Ubuntu look like OSXHow To Create A Favicon In Photoshop [...]
October 6th, 2007 at 8:26
Tales from responsivenessland: why Linux feels slow, and how to fix that…
[...][...]…
October 6th, 2007 at 10:52
We’ve been discussing just these sorts of tuning issues on LXer.com, although without specifics like this article goes into. Especially about how tunable mainframe OSs were in the past.
With IBM working on Linux, I wonder if their experience tuning will make it into the kernel…
A tuning tool would be a great idea. With all the attention that the scheduler has received recently, maybe such a tool will get the needed “itch scratching”.
October 6th, 2007 at 12:19
[...] vm.vfs_cache_pressure=50 Aby uzyskać więcej nformacji na ten teamt wejdź na strone pewnego bloggera.Według niego wydajność na desktopie można zwiększyć przez odpowiednie zagospodarowanie [...]
October 6th, 2007 at 19:26
[...] Tales from responsivenessland: why Linux feels slow, and how to fix that Desktop performance on Linux computers has been a hot-button issue of late, and a source of longstanding fights among the Linux developers. Today, I want to show you how I boosted (and you can boost) desktop performance dramatically. (tags: linux performance tuning desktop cache memory) [...]
October 7th, 2007 at 1:52
Interesting article.
I tried on my system (Fedora 7, 2.6.22.9, 2G RAM, X86_64), and I didn’t notice any slow down during the swapiness test. There was only a small speed improvement on the second test.
Perhaps these tweaks aren’t going to be that helpful for a system with so much RAM?
October 7th, 2007 at 6:35
[...] abhi_chavan wrote an interesting post today onHere’s a quick excerpt [...]
October 7th, 2007 at 7:19
[...] Tales from responsivenessland: why Linux feels slow, and how to fix that Desktop performance on Linux computers has been a hot-button issue of late, and a source of longstanding fights among the Linux developers. Today, I want to show you how I boosted (and you can boost) desktop performance dramatically. (tags: linux howto guide tutorial performance tweak desktop) [...]
October 7th, 2007 at 9:54
Tales from responsivenessland: why Linux feels slow, and how to fix that…
Is your system a little slow due to swapfile management? Try this solution….
October 7th, 2007 at 17:05
[...] synesthete wrote an interesting post today onHere’s a quick excerpt [...]
October 8th, 2007 at 12:36
[...] Zwei Zeilen Code für ein sich schneller anfühlendes Linux. Mal ausprobieren … [...]
October 9th, 2007 at 5:07
[...] Tales from responsivenessland: why Linux feels slow, and how to fix that [...]
October 15th, 2007 at 11:31
[...] I’m thrilled to have found an article describing a simple setting change that will make desktop Linux systems feel much more responsive as we use them. Keep in mind, I said desktop systems, and not servers. If you do try the advice [...]
October 19th, 2007 at 13:00
[...] Change following system parameters based on this explanation of cache issues. [...]
October 20th, 2007 at 20:10
[...] Why Linux feels slow, and how to fix that [...]
October 29th, 2007 at 1:01
I have my swappiness set to the default of 60, and I did the tests you said, but no application code ever got swapped to disk, so there was absolutely no difference between swappiness of 60 or 1.
Debian sid, 1gb ram, 512mb swap. Perhaps it is because my swap is so small (never felt the need for more).
October 30th, 2007 at 14:46
[...] 4.- Tune up your Linux and make it faster [...]
November 5th, 2007 at 1:48
Slowskies.
November 6th, 2007 at 6:31
Thanks for these geek tricks. I had a performance boost on my pclinuxos 2007 machine.
November 20th, 2007 at 18:36
I tried your 5 command test
sync
echo 3 > /proc/sys/vm/drop_caches
dd if=/dev/zero of=/tmp/testfile count=1 bs=900M
find / > /dev/null
cp /tmp/testfile /tmp/testfile2
on Gentoo GNU/Linux on a ASUS A6F notebook with 1 GB of RAM (graphics is shared) and patiently waited until commands finished running. I expected some drop in responsiveness, but unminimizing aplications, openning new tabs in Firefox, running new applications, listing directories etc, and everything responded instantly.
December 4th, 2007 at 21:23
[...] Tales from responsivenessland: why Linux feels slow, and how to fix that (tags: howto cache sysadmin desktop tuning performance linux) [...]
February 24th, 2008 at 11:28
Omg it worked. I have Debian Etch r 3 on a Pentium III 650 Mhz PC with 256 MB SDRAM. I was discouraged by the first test and after the I put in “sysctl -w vm.swappiness=1″. But after I did that I tried to run multiple things on my PC and it worked. OMG. The only stupid thing it did not work on was expectedly OOO.org. Hah! Now if you can speed that up…
March 30th, 2008 at 18:39
[...] a great article on speeding up the Linux Desktop here. Did the two things it suggested and the system is running faster than it ever did under [...]
March 31st, 2008 at 17:53
I am eager to try this! I use Ubuntu Gutsy and have only 736MB RAM, and have noticed that sometimes Ubuntu seems slower than it used to be… and the scenarios described seem to tell it exactly as I have seen it.
Thanks! Joel A. Burdick
July 18th, 2008 at 15:52
Good tips
You can speed up a slow dog
Or get a breed bred for speed http://puppylinux.org/home/overview