Is your Net connection slow? Ever wonder what the hell's going down the wire? Here are five free and effective tools to diagnose network traffic issues.
If you've been using Linux for a while, no doubt you're already amazed with the amount of software there is for you to use, for free. My own last count revealed in excess of three thousand applications ready to install and use. But how do we find the gems?
Look no further, because here they are. These are tools I use on a daily basis to both diagnose my own Internet connections and network performance in remote hosts. You can't claim to be a network management expert if you haven't used at least 5 of these tools, but you don't need to be an expert to use them either, because they're brutally easy to use.
The net essentials in our toolbox
Let's explore the first three tools in our toolset. They're powerful and easy enough for you to get an overview of what's going on with your Net connection.
KSysGuard: view aggregate network traffic
My favorite, in all aspects, is KSysGuard. With it, you can plot network usage for all your network interfaces (even though in 99% of all cases, there's only one network interface, and it's named
eth0). Take a look at this screenshot:
It's not black magic. It's a sheet divided in rectangles, where you can drag and drop the sensors listed on the left to any of the empty rectangles on the right. When you place any of the sensors on the left, KSysGuard starts plotting.
Bonus points for KSysGuard: you can keep tabs on a remote machine via SSH. All you need to do is:
- create an account on the remote machine,
- install a small program named
ksysguardd(which you can find after installing KSysGuard in the path
/usr/bin/ksysguardd) on the remote machine,
- set up passwordless SSH authentication for that account,
- connect with KSysGuard to the remote machine using File -> Connect to machine...
ksysguardd program is self-contained so you can be confident you won't be introducing security issues.
Here's an overview sheet plotted from a remote machine (in this case, this Web host):
Here's another, showing my MythTV PVR and backup machine:
Oh, did I mention KSysGuard can plot much, much more than just network traffic? I guess not, but the screenshots speak for themselves. Oh, and the screens are fully customizable, down to the colors used to plot the graphs. You can save the worksheets for later usage as well.
To install this tool, use your distribution's favorite package manager, and look for KSysGuard or the
In the next page, we'll explore the different uses of
(May I respectfully request, if you like this article, that you use one of the links right below to submit it to your favorite news site?)
netstat: dumb but it does the job
What if you want to find out which programs are communicating with hosts on the Internet? That's
netstat is a terminal program, I'm gonna open a terminal program to get the following screenshots. Do not worry -- you can follow the examples alright, because (despite what Windows zealots think) the terminal does not bite. Trust me: I was bitten by a Rottweiler once (quite the scare!), but so far not ever by the terminal. Swear to God!
(No, the terminal in Linux is not as stupid as MS-DOS. In fact, MS-DOS got nothin' on the Linux terminal. The Linux terminal can show you random fortune cookies -- try pulling fortunes out of MS-DOS, bitch! And unlike MS-DOS, you don't need to be a rocket scientist to type on the Linux console.)
rudd-o@karen:~$ netstat -tnp Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 22.214.171.124:57830 126.96.36.199:1863 ESTABLISHED20312/kopete tcp 0 0 188.8.131.52:35859 184.108.40.206:1863 ESTABLISHED20312/kopete tcp 0 0 220.127.116.11:33625 18.104.22.168:5190 ESTABLISHED20312/kopete tcp 0 0 22.214.171.124:55528 126.96.36.199:1863 ESTABLISHED20312/kopete tcp 0 0 188.8.131.52:32943 184.108.40.206:143 ESTABLISHED6965/evolution tcp 0 0 220.127.116.11:45010 18.104.22.168:22 ESTABLISHED6291/ssh tcp 0 0 22.214.171.124:41794 126.96.36.199:1863 ESTABLISHED20312/kopete tcp 0 0 188.8.131.52:39049 184.108.40.206:1863 ESTABLISHED20312/kopete tcp 0 0 220.127.116.11:55288 18.104.22.168:1863 ESTABLISHED20312/kopete tcp 0 0 22.214.171.124:58186 126.96.36.199:1863 ESTABLISHED20312/kopete tcp 0 0 188.8.131.52:44143 184.108.40.206:22 ESTABLISHED4436/ssh tcp 0 0 220.127.116.11:50222 18.104.22.168:1863 ESTABLISHED20312/kopete tcp 0 0 22.214.171.124:40477 126.96.36.199:1863 ESTABLISHED20312/kopete tcp 0 0 188.8.131.52:50458 184.108.40.206:1863 ESTABLISHED20312/kopete tcp 0 0 127.0.0.1:37979 127.0.0.1:22 ESTABLISHED26792/nxssh tcp 0 0 220.127.116.11:42312 18.104.22.168:1863 ESTABLISHED20312/kopete tcp 0 0 22.214.171.124:38203 126.96.36.199:80 ESTABLISHED12728/ktorrentQ0Chw tcp 0 0 188.8.131.52:38026 184.108.40.206:80 ESTABLISHED6482/ktorrentO6i7Ib tcp 0 0 220.127.116.11:36559 18.104.22.168:80 ESTABLISHED8756/ktorrentscWrEa tcp 0 0 22.214.171.124:33910 126.96.36.199:80 ESTABLISHED8745/ktorrentUEYsvc tcp 0 0 188.8.131.52:33679 184.108.40.206:80 ESTABLISHED11637/ktorrentEVzOe tcp 0 0 220.127.116.11:41446 18.104.22.168:22 ESTABLISHED30037/ssh tcp 0 0 22.214.171.124:34096 126.96.36.199:80 ESTABLISHED25019/ktorrentQwA3q tcp 0 0 188.8.131.52:34395 184.108.40.206:80 ESTABLISHED25495/ktorrentlqrhg tcp 0 0 220.127.116.11:33537 18.104.22.168:1863 ESTABLISHED20312/kopete tcp 0 0 192.168.3.1:48198 192.168.3.2:22 ESTABLISHED5922/ssh tcp 0 0 22.214.171.124:60241 126.96.36.199:5223 ESTABLISHED20312/kopete tcp 0 0 188.8.131.52:35334 184.108.40.206:1863 ESTABLISHED20312/kopete tcp 0 0 220.127.116.11:56878 18.104.22.168:80 ESTABLISHED4060/ktorrentMT0VEb tcp 0 0 22.214.171.124:56819 126.96.36.199:80 ESTABLISHED4059/ktorrentuhNgxa tcp 0 0 188.8.131.52:56575 184.108.40.206:80 ESTABLISHED4051/ktorrentvWDA2b tcp 0 0 192.168.3.1:42163 192.168.3.2:4713 ESTABLISHED5733/amarokapp tcp 0 0 220.127.116.11:57884 18.104.22.168:22 ESTABLISHED8500/ssh tcp 0 0 22.214.171.124:51115 126.96.36.199:80 ESTABLISHED11635/ktorrent3bmrc tcp 0 0 188.8.131.52:34656 184.108.40.206:1863 ESTABLISHED20312/kopete tcp 0 0 220.127.116.11:49335 18.104.22.168:1863 ESTABLISHED20312/kopete tcp 0 0 22.214.171.124:34686 126.96.36.199:80 TIME_WAIT - tcp 0 0 192.168.3.1:60960 192.168.3.2:22 ESTABLISHED29903/ssh tcp6 0 0 ::ffff:127.0.0.1:22 ::ffff:127.0.0.1:37979 ESTABLISHED26793/sshd: rudd-o
You can see netstat (invoked with the tnp arguments -- TCP, show IP addresses, show programs) on that screenie, which comes up instantly after you've hit ENTER.
Sometimes it's useful to see host names instead of just IP addresses -- for that, you can use
netstat -tp, removing the
n argument. That will take a bit longer, but in the end will show something like this:
rudd-o@karen:~$ netstat -tp Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 30.cpe-190-10-133:35859 by1msg3145616.phx.:msnp ESTABLISHED20312/kopete tcp 0 0 30.cpe-190-10-133:33625 188.8.131.52:aol ESTABLISHED20312/kopete tcp 0 0 30.cpe-190-10-133:55528 by2msg2132816.phx.:msnp ESTABLISHED20312/kopete tcp 0 0 30.cpe-190-10-133:32943 tobey.rudd-o.com:imap2 ESTABLISHED6965/evolution tcp 0 0 30.cpe-190-10-133:45010 lujor.com:ssh ESTABLISHED6291/ssh tcp 0 0 30.cpe-190-10-133:41794 by2msg1242016.phx.:msnp ESTABLISHED20312/kopete tcp 0 0 30.cpe-190-10-133:39049 by1msg3245804.phx.:msnp ESTABLISHED20312/kopete tcp 0 0 30.cpe-190-10-133:55288 by1msg4082108.phx.:msnp ESTABLISHED20312/kopete tcp 0 0 30.cpe-190-10-133:58186 by2msg1161905.phx.:msnp ESTABLISHED20312/kopete tcp 0 0 30.cpe-190-10-133:44143 node6501.gplhost.co:ssh ESTABLISHED4436/ssh tcp 0 0 30.cpe-190-10-133:50222 by2msg1262105.phx.:msnp ESTABLISHED20312/kopete tcp 0 0 30.cpe-190-10-133:40477 by2msg2233102.phx.:msnp ESTABLISHED20312/kopete tcp 0 0 30.cpe-190-10-133:50458 by2msg1141816.phx.:msnp ESTABLISHED20312/kopete tcp 0 0 localhost:37979 localhost:ssh ESTABLISHED26792/nxssh tcp 0 0 30.cpe-190-10-133:42312 by1msg3082308.phx.:msnp ESTABLISHED20312/kopete tcp 0 0 30.cpe-190-10-133:38203 184.108.40.206:www ESTABLISHED12728/ktorrentQ0Chw tcp 0 0 30.cpe-190-10-133:38026 220.127.116.11:www ESTABLISHED6482/ktorrentO6i7Ib tcp 0 0 30.cpe-190-10-133:36559 18.104.22.168:www ESTABLISHED8756/ktorrentscWrEa tcp 0 0 30.cpe-190-10-133:33910 22.214.171.124:www ESTABLISHED8745/ktorrentUEYsvc tcp 0 0 30.cpe-190-10-133:33679 126.96.36.199:www ESTABLISHED11637/ktorrentEVzOe tcp 0 0 30.cpe-190-10-133:41446 tobey.rudd-o.com:ssh ESTABLISHED30037/ssh tcp 0 0 30.cpe-190-10-133:34096 188.8.131.52:www ESTABLISHED25019/ktorrentQwA3q tcp 0 0 30.cpe-190-10-133:34395 184.108.40.206:www ESTABLISHED25495/ktorrentlqrhg tcp 0 0 30.cpe-190-10-133:33537 by2msg1104414.phx.:msnp ESTABLISHED20312/kopete tcp 0 0 30.cpe-190-10-133:33537 by2msg1104414.phx.:msnp ESTABLISHED20312/kopete tcp 0 0 karen:48198 gabriela:ssh ESTABLISHED5922/ssh tcp 0 0 30.cpe-190-10-133:60241 kc-in-f125.google.:5223 ESTABLISHED20312/kopete tcp 0 0 30.cpe-190-10-133:35334 by2msg1282513.phx.:msnp ESTABLISHED20312/kopete tcp 0 0 30.cpe-190-10-133:56878 220.127.116.11:www ESTABLISHED4060/ktorrentMT0VEb tcp 0 0 30.cpe-190-10-133:56819 18.104.22.168:www ESTABLISHED4059/ktorrentuhNgxa tcp 0 0 30.cpe-190-10-133:56575 22.214.171.124:www ESTABLISHED4051/ktorrentvWDA2b tcp 0 232 karen:42163 gabriela:4713 ESTABLISHED5733/amarokapp tcp 0 0 30.cpe-190-10-133:57884 tobey.rudd-o.com:ssh ESTABLISHED8500/ssh tcp 0 0 30.cpe-190-10-133:51115 126.96.36.199:www ESTABLISHED11635/ktorrent3bmrc tcp 0 0 30.cpe-190-10-133:34656 by2msg1104406.phx.:msnp ESTABLISHED20312/kopete tcp 0 0 30.cpe-190-10-133:49335 by1msg3082119.phx.:msnp ESTABLISHED20312/kopete tcp 0 0 karen:60960 gabriela:ssh ESTABLISHED29903/ssh tcp6 0 0 localhost:ssh localhost:37979 ESTABLISHED26793/sshd: rudd-o
What if you want to know which programs are listening on network events on your computer? Use
netstat -ltnpu to see that:
rudd-o@karen:~$ netstat -ltnpu Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:7013 0.0.0.0:* LISTEN 26831/nxagent tcp 0 0 192.168.3.1:873 0.0.0.0:* LISTEN 2347/rsync tcp 0 0 0.0.0.0:8010 0.0.0.0:* LISTEN 20312/kopete tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN 23655/smbd tcp 0 0 0.0.0.0:81 0.0.0.0:* LISTEN 23876/apache2 tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 5549/vsftpd tcp 0 0 192.168.3.1:25 0.0.0.0:* LISTEN 3520/master tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 3520/master tcp 0 0 0.0.0.0:445 0.0.0.0:* LISTEN 23655/smbd tcp6 0 0 :::7013 :::* LISTEN 26831/nxagent tcp6 0 0 :::22 :::* LISTEN 655/sshd udp 0 0 0.0.0.0:32768 0.0.0.0:* 5306/avahi-daemon: udp 0 0 0.0.0.0:514 0.0.0.0:* 5147/syslogd udp 0 0 192.168.3.1:137 0.0.0.0:* 23653/nmbd udp 0 0 192.168.2.4:137 0.0.0.0:* 23653/nmbd udp 0 0 0.0.0.0:137 0.0.0.0:* 23653/nmbd udp 0 0 192.168.3.1:138 0.0.0.0:* 23653/nmbd udp 0 0 192.168.2.4:138 0.0.0.0:* 23653/nmbd udp 0 0 0.0.0.0:138 0.0.0.0:* 23653/nmbd udp 0 0 0.0.0.0:67 0.0.0.0:* 9964/dhcpd udp 0 0 0.0.0.0:68 0.0.0.0:* 20406/dhclient3 udp 0 0 0.0.0.0:8010 0.0.0.0:* 20312/kopete udp 0 0 0.0.0.0:5353 0.0.0.0:* 5306/avahi-daemon:
You can see I run a pretty tight machine -- by the way, hackers, don't bother because those services listening on all network interfaces are firewalled to the outside world.
The next page discusses mtr -- the rich man's
mtr: the rich man's
I'm gonna stop for a while and thank my fabulous boss and friend Thomas Goirand for showing me this tool. Hi, Thomas! Up to the day he showed me this tool, I was stuck using the old
...I see I've abandoned my audience for a while there. OK, let's get back to business.
traceroute is an old-school tool that attempts to find out the route Internet packets take from one machine to another. It sort of "pings" each hop in the way, displaying the time it took for each "pong" to turn around and "return to the mothership".
As you can see, it's very useful if you want to find out why a particular Internet host (that's technicalese for "computer") is not responding. Say I want to
traceroute this host from home:
rudd-o@karen:~$ traceroute -n rudd-o.com traceroute to rudd-o.com (188.8.131.52), 30 hops max, 40 byte packets 1 184.108.40.206 24.185 ms 9.690 ms 8.959 ms 2 220.127.116.11 7.929 ms 8.949 ms 48.975 ms 3 18.104.22.168 10.743 ms 38.930 ms 8.950 ms 4 22.214.171.124 129.928 ms 131.279 ms 119.625 ms 5 126.96.36.199 122.800 ms 131.904 ms 144.903 ms 6 188.8.131.52 120.918 ms 116.912 ms 184.108.40.206 154.041 ms 7 220.127.116.11 130.707 ms 18.104.22.168 119.989 ms 22.214.171.124 129.829 ms 8 126.96.36.199 117.842 ms 188.8.131.52 131.958 ms 184.108.40.206 119.965 ms 9 220.127.116.11 300.844 ms 219.831 ms 18.104.22.168 131.913 ms 10 22.214.171.124 127.874 ms 126.879 ms 126.96.36.199 134.978 ms 11 188.8.131.52 136.897 ms 184.108.40.206 154.934 ms 153.903 ms 12 220.127.116.11 168.898 ms 18.104.22.168 128.928 ms 22.214.171.124 140.870 ms 13 126.96.36.199 138.928 ms * 188.8.131.52 281.783 ms 14 184.108.40.206 159.037 ms 220.127.116.11 196.766 ms 18.104.22.168 149.823 ms 15 22.214.171.124 146.927 ms 126.96.36.199 167.857 ms 188.8.131.52 168.168 ms 16 184.108.40.206 162.990 ms 161.532 ms 162.902 ms 17 220.127.116.11 155.931 ms 158.558 ms 18.104.22.168 171.051 ms 18 22.214.171.124 158.107 ms 126.96.36.199 168.646 ms 188.8.131.52 158.128 ms
At three pings per hop (on each row) you can see my information takes eighteen hops from home to, well, this different home :-). Of course, if any of those hops is having a problem,
traceroute will tell you -- and therein lies the value of
traceroute, because it can tell you exactly where is the problem that makes a host inaccessible.
traceroute is nice and good, but slow. This trace took in excess of ten seconds. Also, I specified the -n argument which causes traceroute to show IP addresses instead of names -- otherwise it would have been really slow.
mtr beats the crap out of traceroute. It does the same thing, only in parallel (yes, even converting the IP addresses to meaningful names. I'm going to show you a screen capture, but a video (I'm too lazy for that) would do it more justice -- with mtr you see, in real time, the speed of each one of the hops in the way:
Moreover, you can keep it running and it will keep on running, updating its display every few times a second. Very useful to pinpoint the faulty spot on sporadic/temporary network outages!
Try it out. Install
mtr using your distribution's package management utilities, then run
mtr yahoo.com. Then try it on another host. Tip: if roundtrip times are more than two hundred milliseconds, you can forget about playing Unreal Tournament with that computer.
Oh, thanks again, Thomas!
Now, let's discuss flow analysis, and how to identify applications that are hogging your network.
Flow analysis: what's going on, and where
The tools we have seen so far are quite useful, but sometimes you want to point your finger at a network hog. And none of them are useful for that. What we'll do this time is inspect two (quite popular, I must confess) programs used to do flow analysis -- figuring out how traffic is apportioned, which program or network service is responsible, and where it goes.
iftop: network usage in real-time
I'm not gonna explain why the program
iftop is named
iftop -- I'm just gonna say
iftop. Damn, that's a lot of
iftops on one sentence!
iftop shows you how network traffic is being apportioned, breaking it down by host. This, for example, is a screenshot taken from my computer:
You can see three distinct flows in the three rows there, and the black bars depict the relative network usage for each flow (two bars per flow -- one for incoming and one for outgoing traffic). The last rows in the shot display the total network usage.
The two flows after the first are irrelevant -- we'll ignore them -- because they are broadcast traffic from either Avahi or Windows network browsing. As they say in the Wizard of Oz, do not pay attention to the man behind the curtain.
However, the first row is a lot more interesting. It shows a steady flow of data, in the vicinity of 1.6 megabits per second, from my "desktop" computer to my PVR computer. You see, my "desktop" computer is a Dell PowerEdge blade server, too loud to use as a desktop, and too cheap to have a sound card.
Since I'm a cheap and poor bastard, instead of bothering with an USB sound card for the PowerEdge, I just reuse one of the several sound cards in my PVR computer through the network and the magic of PulseAudio. Yes, I'm listening to music on my "desktop" computer, and the audio travels through my local network into my other computer, which relays the audio to my stereo, through my speakers, to my ears. You get the point.
Now here's a more meaningful screenshot, displaying network usage of my Internet connection (in this case, I typed the command
iftop -i eth2 on the terminal window):
Now you can see a lot more flows -- why are the Messenger servers so intent on focusing in my computer? Well, I guess I now have homework to do, start investigating if anything's wrong.
iftop out. Use your Linux distribution's package manager to install
iftop, then just run it on a terminal window (you may have to type
sudo iftop to run it, because it requires administrative privileges). When you're bored, close it down. However, don't get surprised if you see a lot of traffic going in and out while downloading BitTorrent torrents.
Don't forget to read the
iftop manual page as well (type
man iftop on a terminal window). It's got lots of information for you to learn more, and a couple of other usage modes -- I like the one that shows me each protocol independently, and the other one that shows traffic in Bytes per second instead of bits per second. Very useful.
But there's a prettier tool.
Etherape: network usage in real-time, with graphical goodness
iftop? Well, Etherape is like
iftop, only cooler because it's a graphical program:
This is the very same network activity shown in the first iftop screenshot. Notice how the flow is now colored and has a distinct size based on which host is sending the traffic? Now here's a screenshot of Etherape capturing and displaying traffic on my Internet connection:
Note how it's much easier to discern network activity? Etherape colorizes traffic according to protocol, and each protocol gets a distinct color. You can use this neat tool to find out which kind of traffic is bogging your network down. Admittedly not very useful if you're using it on your own computer, but an absolute must if you are diagnosing traffic issues on an Internet gateway. You know, these pesky users downloading movies and music? They stand right out on Etherape.
The reason I prefer
iftop to Etherape is simple, and related to my job: I can use
iftop to diagnose remote hosts much more easily. When I run Etherape on another host, it runs slowly because my Internet connection isn't exactly great, and Etherape needs to send the pictures over the Internet.
iftop's much quicker because it only sends characters. But you can't deny Etherape is prettier.
Now let's find out what exactly all these programs are talking about on the wire. You didn't know that was possible, did you?
Traffic inspection with Wireshark
As always, we're leaving the best for the last. Wireshark is different from all the tools you've seen so far. It's much harder to use, basically because it requires intimate knowledge of TCP/IP and many network concepts. It's also harder to use because there's no simple "overview" that you can derive from using Wireshark: it's down-to-the-wire details all the way.
Which makes it brutally useful to diagnose problems where identifying network chatter is essential to figuring out what is going wrong. Check this screenshot out to see what I mean:
Network chatter, as I was saying, is what Wireshark's all about:
1) Each row in the first pane displays an overview of each packet that Wireshark has captured. 2) The second pane provides a dissection of whatever packet is selected on the first pane, down to the bits and what each mean. 3) The third pane shows the actual packet contents in hexadecimal and textual views.
There's another view that warrants further attention. For most packets, you can right-click on any packet listed on the first pane, and choose an option titled Follow TCP stream:
Sorry about the blur you see on this screenshot, but I had to black out certain areas lest I be hacked like a steak. The point of the screenshot is to show you one of the essential features of Wireshark: network chat in real-time. In this shot, text colored in red was sent by the initiating host, while text colored in blue was sent by the responding host.
In short, Wireshark will capture network traffic from any host and to any host that crosses your Ethernet (or wireless) connections. It's one of those tools that "remove the magic factor" from computers, because it lets you see, in real-time, what the hell your computer (and others) is doing with your network connection.
As of now, I'm not sure if you perceive the looming danger with Wireshark. It's practically the reason all secure network protocols use encryption (OK, technically it's malicious Wireshark users sitting on network routers). Encrypted protocols appear mostly as gibberish to Wireshark. Unencrypted protocols, like HTTP and Telnet, do not. That's why I had to blur my screenshot.
However bad you think tools like this are widely available, you cannot deny their usefulness. Without it, I'd be out of a job -- keep in mind that my job does not include network penetration testing, yet I still find myriad uses for Wireshark.
I'm getting too political now. Just grab it and use it. If you don't, you'll never know what you were missing. If you do, you might learn a thing or two about network security you didn't know were there.
OK, that's about everything you must know if you want to diagnose a network problem. I'm not going to show you the evil tools now (hey, Wireshark is borderline evil and outright illegal in Germany), but rest assured at some point I will show you how to use them as well.
In the meantime, back to my evil deed... I mean, day job.