The script

Stash it in /usr/lib/nagios/plugins of the Dirvish backup machine, naming it check_dirvish. This script assumes that your Dirvish vaults are in /mnt/backup, so tune it if that isn’t true in your case:

#!/bin/bash

for a in /mnt/backup/* ; do if [ -f ls -d "$a/"* 2> /dev/null | grep -v /dirvish | sort -g | tail -1/rsync_error ] ; then echo "CRITICAL: latest backup in vault $a failed" exit 2 else /bin/true fi done echo "OK: All backups OK"

The security setup

Create a nagios user on your Dirvish backup machine, and set up SSH passwordless authentication.

Now, if your Dirvish vaults are accessible only to root, set up sudo to allow Nagios to run this script as root:

nagios ALL = NOPASSWD: /usr/lib/nagios/plugins/check_dirvish

The Nagios setup

Finally, set Nagios up:

define command{
        command_name    ssh_dirvish_sudo
        command_line    /usr/lib/nagios/plugins/check_by_ssh -t 29 -H $HOSTADDRESS$ -C 'sudo /usr/lib/nagios/plugins/check_dirvish'
        }
define service{
        use                             generic-service
        host_name                       gabriela
        service_description             Backups
        check_command                   ssh_dirvish_sudo
        }

Of course, the sudo call is only needed if the Dirvish vaults are restricted for the nagios user.

Awesome awesomeness: start your downloads minutes after TorrentFlux finishes them

I’ve created a very, very complicated and interwoven (translation: awesome and straightforward) script that you may use and modify on your home computer. It works like this:

• Using SSH, it queries your TorrentFlux server’s transfer list, asking which torrents are completely downloaded.
• For each of the completed torrents, it grabs the file name of the downloaded file.
• Then, it uses rsync to incrementally transfer the downloaded file to your PC.
• Again using rsync, it removes the file from your TorrentFlux server, freeing disk space.

What you need to do before using the script

For you to use it, you will need to do the following:

• Enable SSH access to your (soon-to-be) TorrentFlux server.
• Set up public key (passwordless) authentication so that your home computer’s user account can freely log in to your server without asking for a password.
• Install TorrentFlux-b4rt and set it up so that your SSH user has read/write access to the torrents and downloads folders/files. Umask/permissions, you know the drill.
• Enable fluxcli usage on your TorrentFlux setup.
• Install rsync on both your server and your home computer.
• Install the BitTorrent package that contains the program named torrentinfo-console on your server.
• Create a destination directory on your home computer.
• Install a mail program such as mailx on your home computer.
• Install cron on your home computer.
• Configure the script altering the variables at the top.

Yeah, it’s a lot of work. Go bitch somewhere else, we’re here for the fun.

Installing the script

Put the script somewhere in your PATH. Make it executable. Set up a cron job for the purpose:

# m h  dom mon dow   command
0,10,20,30,40,50 * * * * /home/rudd-o/bin/torrentleecher -q

The script itself

It’s a stunning, hackish example of subprocessing, SSH remoting, pipeline integration, text processing, POSIX daemonizing and file locking, and samizdat logging. It gets almost all of them wrong, and yet it manages to stay standing. If you’re interested in reading a complete account of how this script grew out of nothing, well, click here.

It’s a miracle:

#!/usr/bin/env python

from subprocess import Popen,PIPE,STDOUT,call import fcntl import re import os import sys import signal

FIXME: this script doesn't deal with multifile torrents

vars!

wherever they are used, it's MOST LIKELY they need quoting

FIXME whatever calls SSH remoting needs to protect/quote the commands for spaces or else this might turn out to be a bitch

torrentflux_base_dir = "/var/www/html/torrents/torrents" torrentflux_download_dir = "/var/www/html/torrents/torrents/incoming" torrentflux_server = "yourserver.com" torrentleecher_destdir = "/home/rudd-o/download/" fluxcli = "fluxcli" torrentinfo = "torrentinfo-console" email_address = "rudd-o@awesome.com"

def getstdout(cmdline): p = Popen(cmdline,stdout=PIPE) output = p.communicate()[0] if p.returncode != 0: raise Exception, "Command %s return code %s"%(cmdline,p.returncode) return output def getstdoutstderr(cmdline,inp=None): # return stoud and stderr in a single string object p = Popen(cmdline,stdin=PIPE,stdout=PIPE,stderr=STDOUT) output = p.communicate(inp)[0] if p.returncode != 0: raise Exception, "Command %s return code %s"%(cmdline,p.returncode) return output def passthru(cmdline): return call(cmdline) # return status code, pass the outputs thru def getssh(cmd): return getstdout(["ssh","-o","BatchMode yes","-o","ForwardX11 no",torrentflux_server] + [cmd]) # return stdout of ssh. doesn't return stderr def sshpassthru(cmd): return call(["ssh","-o","BatchMode yes","-o","ForwardX11 no",torrentflux_server] + [cmd]) # return status code from a command executed using ssh def mail(subject,text): return getstdoutstderr(["mail","-s",subject,email_address],text)

def get_finished_torrents(): stdout = getssh("%s transfers"%fluxcli) stdout = stdout.splitlines()[2:-5] stdout = [ re.match("^- (.+) - [0123456789.]+ MB - (Seeding|Done)",line) for line in stdout ] return [ match.group(1) for match in stdout if match ]

def get_file_name(torrentname): cmd = "LANG=C %s %s/.transfers/%s"%(torrentinfo,torrentflux_base_dir,torrentname) stdout = getssh(cmd).splitlines() filenames = [ l[22:] for l in stdout if l.startswith("file name...........: ") ] assert len(filenames) is 1 return filenames[0]

def dorsync(filename): cmdline = ["rsync","-avzP","--remove-source-files", "%s:%s/%s"%(torrentflux_server,torrentflux_download_dir,filename),"."] return passthru(cmdline)

def exists_on_server(filename): cmd = "test -f %s/%s"%(torrentflux_download_dir,filename) returncode = sshpassthru(cmd) if returncode == 1: return False elif returncode == 0: return True else: assert False # Not reached

def get_files_to_download(): torrents = get_finished_torrents() for t in torrents: yield (t,get_file_name(t))

def lock(): global f try: fcntl.lockf(f.fileno(),fcntl.LOCK_UN) f.close() except: pass try: f=open(os.path.join(torrentleecher_destdir,".torrentleecher.lock"), 'w') fcntl.lockf(f.fileno(),fcntl.LOCK_EX | fcntl.LOCK_NB) except IOError,e: if e.errno == 11: return False else: raise return True

def daemonize(): """Detach a process from the controlling terminal and run it in the background as a daemon. """ try: pid = os.fork() except OSError, e: raise Exception, "%s [%d]" % (e.strerror, e.errno)

    if (pid == 0):           # The first child.
            os.setsid()
            try: pid = os.fork()              # Fork a second child.
            except OSError, e: raise Exception, "%s [%d]" % (e.strerror, e.errno)

            if (pid == 0):   # The second child.
                    os.chdir("/")
            else:
                    # exit() or _exit()?  See below.
                    os._exit(0)      # Exit parent (the first child) of the second child.
    else: os._exit(0)                # Exit parent of the first child.

    import resource                           # Resource usage information.
    maxfd = resource.getrlimit(resource.RLIMIT_NOFILE)[1]
    if (maxfd == resource.RLIM_INFINITY):
            maxfd = 1024

    # Iterate through and close all file descriptors.
    for f in [ sys.stderr, sys.stdout, sys.stdin ]:
            try: f.flush()
            except: pass

    for fd in range(0, 2):
            try: os.close(fd)
            except OSError: pass

    for f in [ sys.stderr, sys.stdout, sys.stdin ]:
            try: f.close()
            except: pass

    sys.stdin = file("/dev/null", "r")
    sys.stdout = file(os.path.join(torrentleecher_destdir,".torrentleecher.log"), "a",0)
    sys.stderr = file(os.path.join(torrentleecher_destdir,".torrentleecher.log"), "a",0)
    os.dup2(1, 2)

    return(0)

sighandled = False def sighandler(signum,frame): global sighandled if not sighandled: print “Received signal %s”%signum # temporarily immunize from signals oldhandler = signal.signal(signum,signal.SIG_IGN) os.killpg(0,signum) signal.signal(signum,oldhandler) sighandled = True

def report_file_failed(filename): try: os.symlink(os.path.join(torrentleecher_destdir,”.torrentleecher.log”),”%s.log”%filename) except OSError,e: if e.errno != 17: raise #file exists should be ignored of course sys.stdout.flush() sys.stderr.flush() errortext = “”"Please take a look at the log files in %s”"”%torrentleecher_destdir mail(”Leecher: error — %s”%filename,errortext)

def report_file_done(filename): try: file(”%s is done”%filename,”w”).write(”Done”) except OSError,e: if e.errno != 17: raise #file exists should be ignored of course mail(”Leecher: done — %s”%filename,”The file is at %s”%torrentleecher_destdir)

def main(): if not ( len(sys.argv) > 1 and “-D” in sys.argv[1:] ): daemonize() os.chdir(torrentleecher_destdir) if not lock(): # we need to lock the file after the daemonization if not ( len(sys.argv) > 1 and “-q” in sys.argv[1:] ): print “Other process is downloading the file — add -q argument to command line to squelch this message” sys.exit(0) signal.signal(signal.SIGTERM,sighandler) signal.signal(signal.SIGINT,sighandler)

    print "Starting download of finished torrents"

    try:

            for torrent,filename in get_files_to_download():
                    # no point in doing anything if the file was removed, right?
                    # so we continue if the file doesn't exist
                    if not exists_on_server(filename): continue
                    print "Downloading %s from torrent %s"%(filename,torrent)
                    retvalue = dorsync(filename)
                    if retvalue == 0:
                            report_file_done(filename)
                            print "Download of %s complete"%filename
                            # TODO: make use of fluxcli to kill the torrent out of TF
                    else:
                            if retvalue == 20:
                                    print "Download of %s stopped -- rsync process interrupted"%(filename)
                                    print "Finishing by user request"
                                    sys.exit(2)
                            elif retvalue < 0:
                                    report_file_failed(filename)
                                    print "Download of %s failed -- rsync process killed with signal %s"%(filename,-retvalue)
                                    print "Aborting"
                                    sys.exit(1)
                            else:
                                    report_file_failed(filename)
                                    print "Download of %s failed -- rsync process exited with return status %s"%(filename,retvalue)
                                    print "Aborting"
                                    sys.exit(1)

            print "Download of finished torrents complete"

    except Exception,e:
            report_file_failed("00 - General error")
            raise

if name == “main“: main()

If you want to learn the process of how this script grew out of nothing, here’s the story.

Put this on a script, and run it with the offending process ID as the first argument, and the destination directory as the second one:

#!/bin/bash

pid="$1" # get the PID cwd="$2" # first argument is the CWD, there is a quoting bug below but I really couldn't care less

now let's command the GNU debugger

gdb -q >> EOF attach $pid call (int) chdir("$cwd") detach quit EOF

Inspired by Behdad. Took me one minute to write it, two to look up how it’s done. Thank Google.

Things to notice:

1. Yes, there’s a quoting bug in the here-document fed to GDB. I couldn’t care less; post the fix as a comment if you want to.
2. Caveat luser: when you sweep the rug off of bash’s feet using this trick, the bash prompt will continue to have the wrong idea of the current directory, but every other command (not using the $PWD variable, of course) will operate correctly.
3. Processes holding files open on a volume to be mounted could conceivably get their files closed using GDB as well. Trivial and left as an exercise for the reader.
4. bash is my bitch, I rule, then you die.

The regular monitoring application suite is just not enough. top only analyzes CPU usage. vmstat and iostat only present summaries, but can’t find culprits. strace is too low-level and doesn’t allow to visualize proportions of disk accesses among processes. But pinpointing the exact application or daemon that’s taxing computer resources, whether it’s for short or long times, that’s currently impossible.

Therefore, I propose someone very clever creates these two applications, which should close the gap.

Application 1: psnettop

We need a combo between top and iftop. I should be able to fire up an application that shows me network usage, not by port or by interface, but by process. That way, I can quickly diagnose whether it’s BitTorrent or Firefox. Network admins could remotely log onto users’ computers and inspect which applications are sucking their Internet links.

It should also let me collapse statistics per-application or per-PID.

On Windows land, there’s the fantastic NetLimiter which, apart from setting network bandwidth limits per-process, automatically shows you the instantanteous bandwidth per-process. We need this.

Application 2: disktop

We also need an application, top-style, that shows us disk transactions per application. Not just disk accesses, but real major page faults (swap-ins) and disk reads/writes, bypassing the disk cache. One column for page faults. Another for blocks read. A third for blocks written. They should be reported as they happen, on hardware, so the caches should be ignored. Thus, system admins can identify applications that are:

• causing memory pressure to increase
• sucking up disk bandwidth with excessive swapins/outs
• exercising the disk by loading files not in the cache

strace does not cut it, because it doesn’t show which accesses caused actual disk hardware operations, and because it only monitors a single process.

It should also let me collapse statistics per-application or per-PID.

Extra wishes

If these applications or monitoreables could be taken advantage of in the following ways, it’d be even greater.

KSysGuard integration

As much of a fan I am of the command line, I’ve grown very accustomed to the great KSysGuard. If these monitoreables could be integrated in a graphical console, then it would be very easy to spot problems at a glance.

Logging à là sar

Those of us who have used sar know it’s a valuable way to make assessments on a timeline. If periodic logging of these monitoreables were incorporated into the system, we could quickly diagnose problems that have occurred in the past, and share log files with debuggers and developers everywhere. Thus we wouldn’t rely on anecdotic evidence and hearsay when it comes to discussing naughty applications.

Please help!

Sadly, I don’t have the skillset required to do this. I’m hoping someone in the Free Software community can help us in this sense. I understand that clever work can use SystemTap to derive these values. So there’s a start.

Have a great day!

We’ll cobble a lot of technologies together in order to achieve this amazing feat. While no experience is required with most of the technologies we’ll be using, you’ll need to be proficient in configuring Nagios. I’m also assuming you already have a Nagios instance up, running and operational.

Beware that this guide will require you to save your SSH password for the remote server in a file — not awfully secure. This file will be unreadable to anyone except the Nagios service checker — not even the Web interface has access to it. I consider this an acceptable compromise under the circumstances, but you may not.

Without further ado, here’s the guide.

Why this guide?

Now, what prompts me to do this guide? Of course, it’s an itch to be scratched. My home computer runs Nagios, and I’ve set it up to periodically check for the health of the Apache service that runs this site, because it tends to go haywire and very slow during traffic storms.

The check_http Nagios plugin does work. So why don’t I use it? Because it doesn’t work as I intended. This is better explained by live example.

Most Nagios plugins are simple command-line programs which accept two timeout threshold arguments: the warning and the critical threshold. They report back the success, failure, warning or critical status to the Nagios server in a line of text.

This makes for powerful, modular server supervision and testing. Remote testing of service health is a piece of cake: once you’ve set it up, it’s fire and forget (until any problem with the server arises). In this particular case, Nagios periodically runs the check_http plugin, which times the response time of the Apache server; if the service takes too long to respond, Nagios issues a WARNING or a CRITICAL message to me.

Of course, in a perfect world, air poses no resistance, cows are perfectly round, and network congestion is a non-issue. But my computer is at 18 high-latency hops from the server I’m checking. To boot, I usually have a BitTorrent client open and, no matter now much tuning you do, that kind of traffic breeds congestion. While the service itself may be serving one page in 0.5 seconds, Nagios usually reports times above 10 seconds (in my book, that’s CRITICAL).

So check_http, as shipped, is useless to me — I cannot reliably determine if my remote host is truly slow, or if it’s just me.

The solution: time server response times directly at the source

With that in mind, I set out to find a solution. And the first one that sprang to mind is the one that I implemented. Basically, running check_http directly on this server would yield good timing information, and drastically cut back on warnings. And you can run check_http via SSH, which is an even better proposition.

Keep reading to find out how I did it.

Yes, it does happen. Sometimes the file system of your boot volume gets corrupted… oh, okay, that probably sounded like gibberish, but it’s true. In other words, your disk can get so messed up, that you can’t even get to the recovery console because Windows XP will boot, show the Windows XP logo, wait a few seconds, and then simply restart.

How on Earth do you fix this?

Easy! You just use Linux to solve this problem. Now, pay attention, because this solution does require some elbow grease, but if you need your files right away, you’d better be prepared.

First step: download INSERT

INSERT is the Inside Security Linux distribution. It comes with a lot of utilities designed to perform forensics and fixing systems (yes, even Windows XP!). Additionally, you have full read and write capabilities for Windows XP’s NTFS file system (which isn’t included in most Linux distributions. As if it weren’t enough, INSERT is a very small download (in relation to other Linux distributions). And, finally, it will autodetect your computer’s hardware, without the need for installation.

So, just in case, download INSERT and burn it on a disk. It fits on a business card-sized CD-R (normal CD-Rs are also okay). Place it in your wallet, or on your car’s glove compartment — you’ll end up needing it eventually.

Step two: boot into INSERT

Insert your newly burnt INSERT disk on your computer, and tell your computer to boot from the CD-ROM (instructions for this, of course, vary, since this is a task to be done through the BIOS or the boot menu of your computer).

Here’s how your BIOS boot menu may look like. Take it with a grain of salt.

After INSERT boots, you’ll see a prompt with a nice logo.

Type insert 2 and hit ENTER. You’ll see INSERT booting and autodetecting your hardware, and then you’ll be dumped into a command interpreter. That’s Linux.

Here’s how INSERT looks like when it’s inspecting your computer

And here’s the INSERT prompt. This is a bash shell.

Do not be afraid. Everything is just fine.

Step three: fix your NTFS volume

If you’re like me, and you have an El Cheapo computer (which is, like, 99% of the world population) you’ll have a single IDE disk. If you’re not so lucky (or, perhaps, even luckier, and you have several disks) you’ll have SATA or SCSI disks, one or more. View the contents of the /etc/fstab file:

cat /etc/fstab

should show you the contents of that file. You’ll see several lines, and each line will most probably begin with a /dev/hdXX or /dev/sdXX. Usually, your first NTFS volume should be /dev/hda1 (for IDE disks) or /dev/sda1 (for SATA or SCSI disks).

The contents of my hypothetical /etc/fstab file

Run ntfsfix on it.

ntfsfix /dev/sda1

That should chug along for a few seconds, perform some checks, and return to the prompt. Voilà! Your Windows XP system should be bootable again. Type sync then hit ENTER to ensure data to the disks is written, wait for two seconds, and then hit the reset button of your computer (or power it off and then back on). Don’t forget to remove the CD-ROM as soon as the computer starts.

Once Windows XP boots, you’ll see it claiming that it needs to check the disks. Let it check your disks.

That’s it. Your computer should now boot normally

Going beyond

Of course, while using INSERT, there’s a lot of things just waiting to be discovered. You’ll be able to back up your data onto other disks without touching anything from the source disks. You’ll also be able to log on to the Internet and download software. You’ll also be able to check your disks for Windows viruses. But that’s material for another article. In the meantime, I hope this small tutorial has convinced you to give Linux a shot. Perhaps you won’t need to rescue a Windows XP system ever again

Hope everything’s working just fine for you now. Have a great day!

For the record, I’m assuming that you’ll be using your favorite distribution’s packages for the installation. I’m also assuming that you have root access to your server.

Oh, by the way, I’ve getting several comments about users having issues with highlighted keywords on my articles. If you’re one of those users, please get a Web browser that displays transparent PNGs properly. Microsoft Internet Explorer is not one of them. Any comments about highlighted keywords from IE users will go straight to /dev/null.

Postfix: the reliable mail package

Postfix is my favorite mail server package. Why? Postfix is unique in that:

• it’s surprisingly easy to configure (contrast that to Sendmail, and you’ll understand this point)
• it’s very secure, featuring a security model with separate user contexts for each task
• it’s also very popular
• it’s fast

Configuring Postfix

Remember that I’m assuming you’re installing Postfix from your favorite distribution’s install CDs or download servers. With that in mind, I’d really like to assume that you’ve already installed Postfix, and skip to the configuration part.

To set Postfix up so it runs as a service, use the following commands:

/sbin/chkconfig --add postfix
/sbin/chkconfig --level 35 postfix on

And that’s it. /sbin/service postfix start and see if it’s running with a quick ps ax command. You should see several new processes.

Now, on to configuration. The Postfix configuration file is main.cf. You can usually find it in /etc/postfix (although your distribution’s file layout may vary). The three most commonly changed configuration settings are the following:

• myorigin
• mydestination
• mynetworks_style and mynetworks
• inet_interfaces

Stick to this tutorial. We’ll understand each one of these, in order.

Mail origin: myorigin

Your mail server needs to know how it should identify itself to other mail servers. And here’s how.

The myorigin configuration key specifies the origin domain for all mail sent through your server. While, by default, myorigin is configured to use your server hostname, you may want to change it to your actual domain name. You can accomplish this by setting the configuration key as follows:

myorigin = $mydomain

$mydomain will automatically be filled in with the domain name your server is using.

Destinations: mydestination

Your mail server also needs to know which domains it serves. mydestination fills that need.

mydestination lets Postfix know which domains your server will accept mail from. In other words, mail sent to any of the domains listed in mydestination will be delivered to local mail boxes. The default value:

mydestination = $myhostname localhost.$mydomain

is not exactly useful, but it’s very secure, though most of the time you’d like it to receive mail for your entire domain. If that’s what you want, you should configure Postfix as follows:

mydestination = $myhostname localhost.$mydomain $mydomain

That, of course, assumes that an MX DNS record exists and points to your server as the final destination for e-mail. Don’t forget the DNS step: it’s very important (but, sadly, outside the scope of this 5-minute tutorial).

Of course, there’s nothing stopping you from accepting mail for multiple destinations. An example:

mydestination = $myhostname localhost.$mydomain usm.edu.ec company.com

Relay control

Postfix has a relay control that’s very simple to use. In short, relaying means using your mail server to transfer mail from a client to a destination which is not your mail server.

As you’ve probably noticed, most consumer-level ISPs have an outbound mail server set up for their clients. That’s a good example of a relaying mail server: that mail server is configured to accept relay requests from customers’ e-mail clients. Of course, all mail servers should have a properly tuned relay control configuration — overly zealous servers won’t let their users send mail to other users, while overly permissive servers will let spam and all its consequences through.

You’ll no doubt be in the same position in your organization: you’ll need workstations in your organization to be able to relay mail through your mail server. Again, that’s easy to accomplish. By default, Postfix is configured to accept e-mail from local networks. To change this, we’ll explore two configuration options.

First of all, there’s the mynetworks_style configuration parameter. This parameter can take the following values:

• subnet (default): Postfix will let e-mail clients relay mail, if they are in the same IP subnetworks Postfix is connected to.
• class: Postfix will let e-mail clients relay mail, if they are in the same class A/B/C networks Postfix is connected to. Trust SMTP clients in the IP subnetworks that Postfix is connected to.
• host: Postfix will let e-mail clients relay mail, only if they run in the same server

For most usage scenarios, you’ll want to leave this configuration as subnet.

If you want more fine-grained control, there’s the mynetworks parameter. Through mynetworks, you let Postfix know exactly which networks are allowed to relay mail. Keep in mind that the mynetworks_style parameter will be ignored if mynetworks is set.

mynetworks = 168.100.189.0/24, 127.0.0.0/8

is a good example. That will let e-mail clients relay mail, but only if they’re originating either from the mail server, or any of the computers in the IP subnetwork 168.100.189..

Network configuration

The final, and perhaps most important, configuration parameter for Postfix, is the inet_interfaces parameter. Through inet_interfaces, you let Postfix know which network interfaces should use.

The default is usually:

inet_interfaces = all

though your distribution may have changed this to make Postfix only listen through the localhost network interface (Fedora Core is a good example of this). You can set this to one or more network addresses, domain names, or leave it as all. I recommend you leave it as all and do any securing through iptables.

Letting Postfix know you’ve changed its configuration

Once you’ve configured Postfix, restart it using the /sbin/service postfix restart command (though this may vary from distribution to distribution). You should have a Postfix server up and running. To test it, I recommend the following testing procedure:

• sending an e-mail from root in that server to root, and seeing if it arrives
• sending an e-mail from root in that server to another user in the machine, and seeing if it arrives
• sending an e-mail to a user in the machine, using a mail client within your organization (that is, one that falls in the mynetworks group), and seeing if it arrives
• sending an e-mail to a user in the machine, using Hotmail, Gmail or any other free e-mail provider, and seeing if it arrives
• sending an e-mail to a Hotmail or Gmail account, using a mail client within your organization, and seeing if it arrives
• sending an e-mail to a Hotmail or Gmail account, using a mail client outside of your organization’s mynetworks, and configured using your newly installed mail server as the outbound mail server — and verifying that this test fails, which is a telltale sign that your server won’t act as a spam server

Conclusions

I honestly don’t want to start a religious flamewar but, in my own very humble opinion, Postfix is the best mail server out there. If you’re not using Postfix, I seriously recommend it to you. Give it a spin — it’ll probably stay in your organization for good.

There’s more documentation about Postfix in the project’s Web site. Be sure to peek into it as soon as you have more than 5 minutes to spare.

That’s it for today. Happy hacking!

Most servers are configured once, then left to run for as long as they can (which, in practical terms, means ‘until hardware or software fails’). More often than not, the cause for server failure is rooted in configuration mistakes. That’s why it’s wise to keep a log of all configuration changes, and take detailed notes every time a setting is changed. But, as the collection of servers you need to manage grows, bookkeeping tasks turn into a humongous monster.

No more. Enter Subversion.

What Subversion is

Subversion is, primarily, a source code control system. You feed your files into a Subversion repository, and from that point on, Subversion oversees them. As you or other developers change files, you periodically commit your changes to the Subversion repository. Subversion tracks those modifications, and allows you to replicate them, take snapshots of an entire source tree, and other nifty things. Even file moves, copies and deletes are (nearly) transparently tracked by Subversion — and it has the memory of an elephant: you can delete files and, later on, recall them by revision number.

Subversion also excels with text files; fortunately for us, UNIX configuration files are, mostly, text.

Subversion is, as you probably already know, a better CVS than CVS. Indeed, it was designed as a replacement for CVS, so if you’re familiar with CVS, you’ll probably feel right at home with Subversion.

Let’s understand three fundamental concepts in Subversion.

• There’s a central repository. Only the repository administrator should touch that. You’ll be playing the role of repository administrator only once.
• Users don’t touch the central repository. They check out private “working copies” and work on those copies. Once they’ve hacked their working copy into a desired state, they commit those changes into the repository. You’ll be playing the role of user most of the time.
• On any given working copy, there are two ‘axes’ users can travel (whether to check out a few files, or to make a diff between two states):
  • in time: once you’re using Subversion, you can go back and forth “in time”, by switching to older and newer revisions (the newest of which is customarily called HEAD)
  • between branches and tags: at any point in time, you can instruct Subversion to make a copy of a folder into another folder. You can use this functionality to make periodic snapshots of a certain state, regardless of revision number.

How we’ll use Subversion

First, let’s review the tasks we’ll perform:

• We’ll create a Subversion repository to store the server configuration history
• We’ll prepare the configuration directory
• We’ll check the configuration files in
• We’ll investigate how to perform followup tasks

I’m assuming that you have Subversion installed; in other words, you should have the svn and svnadmin commands and they should work properly. I’m also assuming that you’ll be performing the following tasks as root.

The ideal situation to begin applying this tutorial is right after your server has been freshly installed. However, for practical purposes, any server that’s configured and running will do.

Okay. That’s enough of the lists and introductions. Time for some action.

Creating the Subversion repository

If you’re familiar with UNIX, you’ll know /var is the customary directory for files that pertain to the whole system and are changed. So, following tradition, we’ll create a /var/preserve/config repository. Type the following command at your console:

[rudd-o@amauta2 ~]# svnadmin create \
/var/preserve/config

(note the backslashes are being used to add whitespace)

That should create a /var/preserve/config directory, with a couple of files in it. Those files are not meant to be editing, and they’ll be opaque to us for the rest of the tutorial. As usual, I’d advise you to secure that directory so only root can read and write files to it.

Now, you’ll create two directories directly into the repository. You’ll use these directories to travel back and forth between known configuration states.

• trunk/ will contain the current configuration
• tags/ will contain snapshots (we’ll learn more about them later)

To perform this task, just type:

[rudd-o@amauta2 ~]# svn mkdir \
file:///var/preserve/config/trunk/ \
file:///var/preserve/config/tags/ \
-m 'Creating trunk and tags directories'

The -m argument specifies a message to attach to the operation. You can consult these messages afterwards through the svn log command.

Preparing the configuration directory

In true UNIX tradition, /etc is the place to go for system-wide configuration. For the rest of the tutorial, I’ll assume those are the files you want to keep in check.

To track files in /etc, you need to both:

• place its contents into the Subversion repository
• make it into a working copy

That’s easily accomplished via the following command:

[rudd-o@amauta2 ~]# svn checkout \
file:///var/preserve/config/trunk/ /etc

Once you’ve done that, /etc will be a working copy. Time to add existing files into Subversion.

Checking existing configuration files into the repository

[rudd-o@amauta2 ~]# cd /etc
[rudd-o@amauta2 /etc]# svn status

You should see a long listing of files, like this:

?      4Suite
?      acpi
?      adjtime

The question marks at the beginning of each line mean that Subversion has no idea what those files are doing there. So, you’ll add them to the repository:

[rudd-o@amauta2 /etc]# svn add *

You’ll see svn working intensely to add those files. Note that the files are not being added to the repository yet — they’re only being queued for addition. To commit these files into the repository:

[rudd-o@amauta2 /etc]# svn commit \
-m 'Initial addition of files'

And svn should start doing its magic. Once it’s done, it’ll tell you the revision number.

Followup maintenance

Okay, let’s review a few things you need to keep in mind from now on.

When configuration files are added to /etc

Check for added files with svn status /etc. You should see them listed with a question mark.

You should use svn add to add them to the working copy, and then svn commit the added files into the repository. Many people make the mistake of configuring freshly installed files. Do not do that. Instead, commit new files first, then edit. That way, you’ll have a way to track modifications right back to the pristine configuration files.

When configuration files are deleted from /etc

Check for deleted files with svn status /etc. You should see them listed with an exclamation sign.

After doing the check, svn delete them. Don’t forget to commit at the end.

Before moving files or directories within /etc

Disregard the standard UNIX coreutils for file management, because they bypass Subversion tracking. Use svn add, svn copy, svn move and so on. Don’t forget to commit at the end.

What to write on log messages

Use your imagination! At the very least, you should be typing who did the change, the reasons behind the change and any other related ideas that might be forgotten but needed later on.

How to view the log and audit changes

Use svn log /etc. Adding a --verbose option will show you the paths changed.

You can also obtain a patch (diff output) of changes between revisions. Use svn diff -r first:last /etc, replacing first and last with the first and last revision numbers you want to use as first and last reference points, respectively. Remember that HEAD works great when used in place of the last revision. A plain svn diff /etc will show you the differences in files that haven’t been committed yet, in diff format.

How to snapshot the configuration and travel between snapshots

To snapshot the configuration, first ensure no files have changes pending to be committed in /etc (using svn diff /etc and svn commit /etc), then use the following command:

[rudd-o@amauta2 ~]# svn copy \
file:///var/preserve/config/trunk \
file:///var/preserve/config/tags/a-snapshot

replacing a-snapshot with the name of the snapshot you want to take. Subversion will take a snapshot of the current configuration state.

Seeing what’s changed between two tags or the trunk and a tag is easy:

[rudd-o@amauta2 ~]# svn diff \
file:///var/preserve/config/tags/a-snapshot \
file:///var/preserve/config/tags/a-newer-snapshot

[rudd-o@amauta2 ~]# svn diff \
file:///var/preserve/config/tags/old-snap \
file:///var/preserve/config/tags/trunk

And traveling to a tag, making the whole /etc and its contents go to a known state, is easy as well.

[rudd-o@amauta2 ~]# cd /etc; \
svn switch file:///var/preserve/config/tags/known-state

I don’t recommend making any changes to configuration files and then committing those changes if your /etc directory has been switched to a tag, because you’d overwrite the snapshot with those changes. Better to create a top-level branches/ directory in your repository, copy the tag to branches/a-branch-name, switch to the branch a-branch-name and then make changes there.

Conclusions

Indeed, adding Subversion to the server management mix introduces extra complexity. But the gains fairly outweigh the drawbacks. Clever readers should have noticed, just by skimming this piece, the potential that Subversion has. Subversion makes auditing, known configuration state restoration, and configuration backups much more deterministic and predictable.

Moreover, common snapshotting tasks can be automated via Vixie cron jobs. If you know how to serve Subversion repositories through the network, you’ll see that designing, staging and distributing configuration sets and managing them remotely from a central location is incredibly easy. And there’s so much more Subversion can do to stretch your limited available time. If you want to know more, there’s always the Subversion book (aptly named Version control with Subversion).

And that’s my cue to continue my day job. Happy hacking!

Since all you have is 5 minutes (which is one of the tenets of this Server management series, and quite possibly the only simple truth in your case), in this installment, we’ll unlock the secret of server log foraging.

All you need is grep.

I don’t know what the Beatles would say about this heading, but, indeed, grep is all you need.

This guide is laid out in two distinct sections:

• Configuring PHP to report errors
• Actually mulling over the server logs

I’m assuming you’re using Apache on Linux, of course. I will also assume you have access to your server logs and write permission to the PHP configuration file (usually /etc/php.ini).

Configuring PHP to report errors

Okay, fire up your trusty editor of choice (I use nano, and that’s only because I’m accustomed to pico — no longer shipped with any Free Software distro — and please let’s not get into an editor flamefest here) while being root. Open /etc/php.ini and look for the following configuration keys:

display_errors = On

Set it to Off. Production PHP applications should not reveal any bugs to the outside world. For the true hacker, the log files will say all that’s needed. And, for that, change the following configuration key:

error_reporting = E_ALL

Set it to E_ALL, because you’re interested in warnings, notices and errors. Finally, look for the following configuration key:

log_errors = Off

and set it to On. Save the file, and restart Apache (here’s how).

Let the application run for a couple of minutes (or hours, depending on your visitor volume).

grepping the logs

Now it’s time to start working on those logs. Locate your Apache error log file (usually at /var/log/httpd/error_log. Then use the following command to find all PHP-related error messages:

grep PHP /var/log/httpd/error_log

That may spit out a hefty amount of messages, so you’ll probably want to redirect it into a file:

grep PHP /var/log/httpd/error_log > /root/phperrors

The greater-than symbol directs the output of grep into the file /root/phperrors.

Inspect the newly created file with your favorite editor and look for messages. Does any of them say PHP Fatal Error or PHP Error? If so, you’ve got a real problem between hands. Warnings are also a cause for concern, because they usually mean an error occurred in your application, but the application just kept going like nothing happened. Notices aren’t so much a concern, but they should be dealt with individually, because they usually point to common programming mistakes that must be fixed.

The PHP log messages provide you with another useful tidbit of information: they say the file and line number of the error, notice or warning:

PHP Warning: strtotime() [client 157.100.84.91] PHP Warning: strtotime(): Called with an empty time parameter. in /home/rudd-o/public_html/links.php on line 53, referer: http://rudd-o.com/index.php

That information is very useful to find the source of the problem.

Conclusions

Whether you have to fix errors yourself, or there’s a dedicated team behind the Web applications you’re overseeing, any PHP-related messages usually pinpoint to issues that need to be paid attention to.

It’s also a good idea to automate this procedure into a shell or CGI script for the developers of the applications you oversee. That way, they’ll have a way to get early notice of any issues, and you won’t need to be involved in the process. Here’s an example shell script that would send those messages to someone at your company, to be run through cron:

#!/bin/bash

grep PHP /var/log/httpd/error_log | \ mail -s 'PHP messages' someone@yourcompany.com

Finally, this procedure should also be standard on testing and staging servers. Finding a problem early on the development stage is invaluable. You really don’t know how many customers you may lose to bugs that slip into production.

And that’s it for today! Remember, keep your Web applications bug-free, and they’ll give lots of value in return. Happy hacking!

Print these instructions out and post them on a wall or a bulletin board in your office. If you have a sizable park of Windows computers to manage, learn them by heart. They may be very useful to you in the near future.

These instructions work if you’re running a SAMBA domain server on Linux or any other UNIX, and your domain server is not using LDAP services to store SAM information, but the standard SAMBA TDB files.

Three steps are all that’s required, if you have a properly configured SAMBA server (regrettably, out of the scope of this 5-minute topic).

Add the machine account on the server

Okay, time to do this. As root, on the console, add a UNIX user account, with the following command:

[root@amauta2 ~]# /usr/sbin/useradd 'machinename$'

That should create a UNIX user account that, by default, has a disabled password. So it won’t be useable as an interactive shell or graphical login account. But, anyways, remember to replace machinename with the machine name you intend to set on the XP computer. Do note that the useradd command may be on a different directory than /usr/sbin on your computer.

Please note that the single quotes are relevant. Otherwise, they would be unprotected by the shell’s variable replacing tendency.

Now run the following command:

[root@amauta2 ~]# smbpasswd -ma 'machinename'

This command actually creates the machine account on the SAMBA server.

Disable RequireSignOrSeal

According to a contributor, you can skip this step if you’re using SAMBA 3 or higher. But if you aren’t, then it’s time to disable a setting that makes Windows XP complain when attempting to join a SAMBA domain. The famed RequireSignOrSeal.

Physically go to the Windows XP computer. Log on using an administrative account (Administrator comes to mind) on the local machine. Open the Registry editor (regedit.exe). Now open the key named:

My Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters

You’ll see, on the right pane, a key named requiresignorseal. Double-click it and set the value data to 0. If it doesn’t exist, create a key named requiresignorseal of type REG_DWORD and set it to 0.

Now, that machine is ready to join the SAMBA domain.

Configure the machine to join the domain

Open the Properties tab of My Computer. Click the Computer Name tab, and click the Change button.

The computer name should be the same as the name of the machine account you created in the first step. On the Member of: group, click Domain, and type the domain name you’ve configured in the SAMBA server.

Click OK. A password prompt will surprise you. Enter the root user name and the root password of the SAMBA server, and hit OK. In the few moments after you’ve hit OK, Windows XP and SAMBA will be negotiating the process of joining the domain.

f everything went OK, you’ll see a Welcome to the XYZ domain popup. If something went wrong, you’ll have a hard time figuring out what went wrong; the first place to go is the SAMBA log file.

Conclusions

Okay, that’s it. You’ve successfully integrated one of the most pervasive unfree software packages to your trusty SAMBA server.

These instructions are bound to change in the future, as the SAMBA team continues to move aggresively towards LDAP. But, in the meantime, for small- to medium-sized businesses, the ideal SAMBA setup won’t be needing LDAP magic anytime.

In case you want more information on the subject, there’s always the Windows XP section of the SAMBA Client configuration HOWTO. That’s all for today. Happy hacking!

If you recall the article titled Tuning an Apache server in 5 minutes, you’ll also know that there’s a tunable for Apache which lets you set the maximum number of Apache processes that run on your server. Once you’ve tuned Apache, it only makes sense to tune MySQL to handle that many connections simultaneously.

Before you go on, “Tuning a MySQL server in 5 minutes” is indeed an exaggeration. I concede you that. Database tuning is so much more than what this article says. I don’t mean to disrespect DBAs: they usually perform large amounts of magic in order for databases to get from abysmal to top-dog performance. But the first step to having a site that doesn’t break with traffic surges is usually what I’m about to discuss.

Tips for very high loads

Okay, on to our business. First off, if you’re handling a very large number of simultaneous connections to your Apache server (in the order of 250 or higher), it would make sense to offload the database processing to a different server. That way, you’ll have more control over loads exerted by Apache and by MySQL, separately.

If you’re short on money, that’s of course not an option. Keep reading to find out an acceptable compromise then.

The (important) differences between static and dynamic page loads

For the purpose of this article, we’ll name two distinct types of connections to your Web server:

Dynamic requests

Any request to your Apache server that causes a MySQL connection to be opened and database queries to be emitted. A good example is a PHP page which requests a list of products from your database.

Static requests

Any request to your Apache server which doesn’t incur the cost of a MySQL connection. Examples of these are static HTML pages or file downloads.

And, of course, you’ll need to discriminate between those two.

Figuring out the right maximum number of MySQL connections

Usually, a good starting estimate is one dynamic request for each 5 requests. That’s because most pages load CSS style sheets, and images, although those files do not get loaded on subsequent requests from the same visitors (partly due to browser caching). To get an exact number for this ratio, however, you’ll need to analyze your Apache access_log log file (manually, or via the known Analog or Webalizer log analysis packages).

Once you’ve arrived to an accurate estimate for your scenario, multiply that ratio by the maximum number of connections you’ve configured on your Apache server. For example, if your Apache server is serving a maximum of 256 clients (which is a lot), and your ratio of dynamic requests vs. all requests is 1/8, you’d have an expected maximum of 32 database connections. Just to be on the safe side, multiply that by two, and you’ll have a foolproof figure. But if you want to be really, really certain, you should always expect a maximum of 256 database connections.

Setting the maximum connections on your MySQL server

Using your favorite text editor, as root, open up the /etc/my.cnf file (the location of the file may vary according to your distribution). You should see something like this:

[ecuagol@216-55-181-30 ~]$ cat /etc/my.cnf
[mysqld]
safe-show-database
innodb_data_file_path=ibdata1:10M:autoextend
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

[mysql.server] user=mysql basedir=/var/lib

[safe_mysqld] err-log=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid

We’ll be dealing with the [mysqld] section. Under that section, add two new parameters (or modify them, if they are already there and they aren’t commented):

• set-variable = max_connections = 60
• set-variable = max_user_connections = 60

MySQL defaults to 1 max connection, with 1 max connection per user. Evidently, you’ll be replacing 60 with your expected maximum number of connections. With these settings, you’ll be on the safe side.

The reason you’re setting both max_connections and max_user_connections is because, generally, Apache appears to your MySQL database server as one single user. So, you need to raise them both.

You may also want to increase other parameters, if you’ll be expecting heavy loads or unusual queries:

• set-variable = max_allowed_packet=1M (sanity check to stop runaway queries)
• set-variable = max_connect_errors=999999
• set-variable = table_cache=1200

Where 1200 should be max_user_connections multiplied by the maximum number of JOINs your heaviest SQL query contains.

After tuning your server, restart MySQL (/sbin/service mysqld restart usually does the trick on Fedora Core).

Conclusions and final words

That’s it! Hope I’ll see you around for the next installment. By the way, if you spot any inaccuracies or errors, feel free to comment on it using the comment form right below this article. Happy hacking!

Before we go on, you should know something: this is not an article about securing Apache. This is an article about making Apache behave under heavy load conditions.

Okay, now that we’re here, let’s discuss scalability.

Scalability

Scalability is simply the ability of a server to withstand heavy loads. If you tried to read the last article, Hardening a Linux server in 10 minutes, you probably noticed that this server was down.

That’s a scalability fault.

Let’s put it in another light. This server has 512 MB of RAM. The surge of traffic (thanks to LinuxToday links pointing to this site) caused the server to fail (more accurately, the MySQL server appeared to hang). Brag all you want about Linux’s ability to survive these events, nothing will help you against a misconfigured server.

It all boils down to configuration

In this particular case, the misconfiguration was Apache’s. Weighing 13 MB per httpd process (though some of it is shared with other processes), it’s pretty simple to understand that a runaway Apache server can bring your server down completely. When your Apache server starts serving a lot of requests, all those processes quickly fill the available memory (physical and virtual). When your Linux server runs out of RAM, it will start killing processes it deems ‘memory hogs’. Usually the first ones to go down are the MySQL processes. If you’re serving dynamic pages, that’s a disaster.

On to Apache configuration

By default, Apache comes preconfigured to serve a maximum of 256 clients simultaneously. This particular configuration setting can be found in the file /etc/httpd/conf/httpd.conf (though the location of the file may vary, depending on the Linux distribution you use).

Whip your favorite text editor out and open that file (remember that you should be doing this as root — the administrative account on the majority of Linux servers out there).

Look for MaxClients. It will probably look like this:

# prefork MPM

StartServers: number of server processes to start

MinSpareServers: minimum number of server processes which are kept spare

MaxSpareServers: maximum number of server processes which are kept spare

ServerLimit: maximum value for MaxClients for the lifetime of the server

MaxClients: maximum number of server processes allowed to start

MaxRequestsPerChild: maximum number of requests a server process serves

<IfModule prefork.c> StartServers 4 MinSpareServers 3 MaxSpareServers 10 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 10000 </IfModule>

That’s the configuration section for the prefork module. 99% of the Apache servers out there use the prefork module to serve requests, so unless you have an exotic configuration, you’ll be changing these settings.

Time to calculate a good value for the MaxClients directive. Find out how much memory your Apache processes use. Using top, check the RES column. That’s the resident memory size. It should say the size in megabytes that your Apache processes are taking. In my example, it’s 22m.

Figure out a good value. If your server has 512 MB of RAM (in my case, this is true), and you’re sharing your server with MySQL and Sendmail (true in my case, as well), you’ll want to reserve about half of it for Apache (256 MB). Divide that by the resident memory each process takes up, and you’ll have a number of processes (say, 11). That’s the maximum amount of processes you can run without resorting to virtual memory. Resorting to virtual memory (swap) will make your server thrash and become extremely slow.

It’s, of course, all about balance. If you have one gigabyte of swap, you may want to raise the number of Apache processes. Raising it too much will cause heavy traffic to spawn lots of Apache processes, bringing your server down.

Setting the MaxClients and StartServers directive

You now have your start value (in our example, it was 11). Change the MaxClients and the ServerLimit directives to it. Save the file and restart Apache (/sbin/service httpd restart does that trick in Fedora Core).

Now it’s time to start testing. Keep a root login open to that server. Using your favorite testing tool (ab and wget are good at this), start a storm of connections (more than 1024 simultaneous requests) directed to a page served by your Apache server (ideally, one that exercises the server, like dynamic pages with lots of queries). Issuing the uptime command in your root login should not yield a load average above 1, and the server should respond to commands quickly.

[rudd-o@amauta2 conf]$ uptime
 15:54:18 up  1:41,  3 users,  load average: 0.86, 0.70, 1.50

Tuning the configuration

That’s great. Once the test is finished, duplicate MaxClients and StartServers, and try your storm test again. The load average should be low.

Keep tuning until you hit your maximum desired load average. For servers used interactively often, having a load above 3 is way too much to use the server comfortably. For servers used mostly as real servers, a maximum load average of 10 should be acceptable. More than that, and you’ll find yourself needing to reboot the server when experiencing heavy traffic conditions, because no terminal or remote console will respond quickly to commands, and managing the server will be impossible.

Conclusions

That’s it! With practice, you’ll be able to skip the memory math and learn the ideal setting for any server. Other tuning options you may try (in order of diminishing returns):

• Eliminating unnecessary Apache modules from the configuration (perhaps uninstalling them altogether, by use of RPM or your favorite distribution’s packaging tool)
• Recompiling Apache, optimizing for memory consumption (the -Os option of gcc)
• Recompiling Apache, building modules in instead of having them run as modules

Remember: if you have any questions or suggestions, please leave them as comments below. Happy hacking!

Print these instructions out, and keep them posted on a wall in your office or home. Before plugging a freshly installed network server, simply remember to follow these instructions. Make these instructions second nature to you. Just in case you were wondering, a printout of this page will be “printer-friendly” because of the stylesheet used in this page.

You’ll need a bit of experience with the Linux command-line environment, as the following commands are usually issued in a terminal. You will need root access on your server as well. By the way, the following instructions apply to any LSB-compliant Linux distribution, but I’ll use Fedora Core as an example.

Step 1: turn all unneeded services off

There are two kinds of network services:

• those that get started as init.d services
• those that get started by xinetd

This distinction is important, as xinetd can start services on demand, while services started through init.d run all the time.

Okay, time to start securing your server. On a terminal, as root (and, for the purposes of this tutorial, assume this from now on) run netstat -ltunp. You should see a listing like this one:

[root@andrea rudd-o]# netstat -ltunp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 0.0.0.0:3493                0.0.0.0:*                   LISTEN      30562/upsd
tcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      12461/mysqld
tcp        0      0 0.0.0.0:6543                0.0.0.0:*                   LISTEN      12490/mythbackend
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      1771/portmap
tcp        0      0 0.0.0.0:6544                0.0.0.0:*                   LISTEN      12490/mythbackend
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      31537/cupsd
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      2143/sendmail: acce
tcp        0      0 :::80                       :::*                        LISTEN      5024/httpd
tcp        0      0 :::22                       :::*                        LISTEN      2009/sshd
tcp        0      0 0.0.0.0:19                  0.0.0.0:*                   LISTEN      2019/xinetd

Those are all processes listening to specific ports. As you can see, the PID (process ID) and the program name are displayed as well.

Make two lists: - one for the services you absolutely need (which you should already know by heart), and - one for the services that are expendable or you can start manually when they’re needed (tip: each program name usually ships with a man page).

Shutdown each service on the second list (except for xinetd) That’s a pretty straightforward task. Each one of those services are started by init.d. To find out the name of the service control script, just hop to /etc/rc.d/init.d and look for a file with a name similar to the program name.

Example: suppose I don’t need mythbackend. To stop it: /etc/rc.d/init.d/mythbackend stop (some distributions provide the service mythbackend stop command, which is easier on your fingers). Now, to disable it: chkconfig --del mythbackend. After doing this, you should check to see if the offending service went away, with the same netstat -ltunp command.

That pesky xinetd

Great. So you got rid of the unneeded services. But there’s more. As we saw earlier, xinetd has its own ways. In practice, this means that some services will be started on demand — thus, you won’t see them under your netstat -ltunp listing.

To find out which services xinetd manages, hop to /etc/xinetd.d and do a directory listing. You should see some service configuration files. Identify the ones you won’t be using, and edit each one of them, adding a line that says disable = yes between the curly braces.

Note that some services already ship with disable = yes, but some ship with disable = no. If one of the configuration files says disable = no, just change it to disable = yes. Now reload xinetd with the famous /etc/rc.d/init.d/xinetd reload, and run netstat -ltunp again, just to be sure.

That’s step 1. With a bit of practice, you should be doing this in five minutes or less.

Step 2: limit access to running services using iptables

Great, our server now runs the absolutely required services, and no more. But some of those services aren’t meant to be accessed from everywhere, right? For example: I may have a MySQL database server running, but that doesn’t mean MySQL should be accessible from any random IP address on the Internet, right?

So, we’ll use the firewall to stop evil at the door. Again, make a list of services. For each item on the list, identify which IP addresses should be able to reach the service. For each service on your list, write down the TCP/UDP port(s) they use.

In my example, MySQL uses TCP port 3306, and should only be accessible by localhost (127.0.0.1).

Time to compose and activate the iptables rules. Doing a quick check with iptables -L, I can see that my INPUT chain (the one I’ll be working with, since I want to disallow INPUTs to my server) is empty:

[root@andrea xinetd.d]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT) target prot opt source destination

Chain OUTPUT (policy ACCEPT) target prot opt source destination

Your mileage may vary, because your distribution may already have set up some basic iptables rules; to make these instructions foolproof, I will be inserting rules at the beginning of the INPUT chain.

In this case, I want to allow access to 127.0.0.1:3306, and deny access to everyone else on port 3306, in that order. So two rules are needed. I’ll add the “allow” rule into position 1 (the very first):

[root@andrea xinetd.d]# iptables -I INPUT 1 --protocol tcp --destination-port 3306 -s 127.0.0.1 -j ACCEPT
[root@andrea xinetd.d]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     tcp  --  localhost.localdomain  anywhere            tcp dpt:mysql

Great. I’m telling the firewall to -j ACCEPT all --protocol tcp connections to --destination-port 3306 from the address -s 127.0.0.1. Now, I’ll insert the “deny” rule into position 2:

[root@andrea xinetd.d]# iptables -I INPUT 2 --protocol tcp --destination-port 3306 -j REJECT
[root@andrea xinetd.d]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     tcp  --  localhost.localdomain  anywhere            tcp dpt:mysql
REJECT     tcp  --  anywhere             anywhere            tcp dpt:mysql reject-with icmp-port-unreachable

See how easy it is? Let me explain: rule 2 tells the firewall to -j REJECT all --protocol tcp connections to --destination-port 3306 from any address (since I omitted the address). Since rules are processed “top-down” (from 1 to n), the first one that matches an incoming connection is applied. If no rules match, then the default policy (which is normally ACCEPT) kicks in.

Lather. Rinse. Repeat for every service that you want to secure.

Finally, save the rules. For this, you’ll need to use your distribution’s tools. For Fedora Core, that’s as easy as issuing the command service iptables save and ensuring that the iptables service runs at boot time: chkconfig --add iptables.

It’s worth noting that some people prefer to -j DROP instead of DENYing. DROP means that your server will ignore connection attempts (neither denying connections nor accepting them). I prefer DENY, because it’s easier to pinpoint a problem with iptables rules that way, and (most importantly) DROP rules make those ports appear as filtered to a hostile port scanner (which hints to the attacker that a service is running).

So that’s it, from insecure to secure in 10 minutes! If you have any suggestions or questions, please leave them as comments below. Happy hacking!

I ended up keeping swap active. And here’s why.

But… I do have enough RAM for my programs… so I disabled swap. What’s the dillio?

But, if you had enough RAM to keep the entire working set in memory (in other words, to keep both data and code for all your running programs in main memory, at all times), why would you want swap? More concretely, if you had 512 MB of RAM, and your open programs occupied 490 MB of RAM, why would you need swap?

We all know using the disk as swap is slow. So, many people eliminate swap from their systems as soon as they install lots of RAM. And one of the biggest benefits of having lots of RAM is caching. Linux will attempt to cache disk files as aggressively as possible. Taking a page from the example I just concocted, if all I had left is (512-490) = 22 MB of free RAM, Linux would only be able to cache 22 MB of data.

Caching and program startup speed

Why is that figure important? Because, by now, you’ve probably noticed that the slowest operation on Linux is loading an application (especially big ones, like OpenOffice.org). But what you probably don’t know is that most of the time used to load an application is spent on disk accesses: firing up OpenOffice.org causes at least 130 MB of mostly small files to be read from the disk, and those disk accesses add up dramatically, mainly because of disk seek time.

You’ve also probably noticed that loading the same application a little while later takes a much shorter time. Why is that? It’s the cache who’s responsible of this big speedup: if Linux avoids reading files from disk, it runs much, much faster.

It’s also clear that having 22 MB available for a disk cache is just not enough to have OpenOffice.org start in less than 10 seconds, whether it’s the first time you’re starting it, or you just ran it a few moments ago and you’re running it again.

Most other everyday operations are affected. Opening folders in Nautilus or Konqueror are good examples.

So, how does swap actually enter the RAM equation here? What’s Linux not telling me?

It’s unbelievably simple, although not apparent at first.

Most running programs have code paths and data in memory that they rarely, if ever, touch. I would be bold and say most program-allocated memory is very rarely touched, but I might be wrong. So, the RAM taken up by unused code and data would actually be better utilized as cache for frequently-used files (or, even more beneficial, inode and dentry cache).

Having a swap partition gives Linux a space to put those rarely-used code paths and data segments on disk, freeing memory for the disk cache and new tasks. (Actually, shared libraries, unlike program data, can be evicted from RAM and re-read again from the shared library files when needed, but this incurs more disk seeks and computation effort than reading them back from swap, so if Linux has swap available for use, it’ll always prefer to put the shared libraries’ memory into swap).

The more memory available for the cache, the faster the response for frequently-performed tasks (like opening your huge MP3 folder… or my own huge MP3 folder, with Nautilus).

And that’s exactly why having swap is beneficial: unused code and data can be evicted from RAM, and that RAM can be put to better use. Even if you had over a gigabyte of physical RAM, that RAM would be better utilized as disk cache instead of being taken up by unused code. I’m not kidding: swap can make the difference between zero disk reads and three hundred disk reads.

This becomes more critical when low on RAM: imagine if the system were able to cache those 22 MB plus a 150-something MB of RAM, gotten by evicting unused stuff? Seen under that light, starting OpenOffice.org up is beginning to sound even doable, right?

There’s another reason, but it’s not performance-related. Some applications do allocate and consume a lot of memory. If you were to load several of those applications, the possibility of hitting zero free memory without swap is greater. And hitting zero memory is not good, because the system will kill processes it deems “memory hogs”. Best-case scenario: the application you just started dies. Worst-case scenario: a system daemon providing vital services (in my computer, it would be MythTV recording an episode of Nip/Tuck) dies. No one wants that!

In short: don’t skimp on swap. I do not believe in the 2.5X formula for swap (RAM times two point five). But I do keep twp half-a-gigabyte swap partitions (on different disks, in separate controllers, with equal priorities for added performance, and not the disk I use for MythTV) mounted at all times.

And how much RAM should you buy? As a general rule, if having all your commonly-used applications open causes the system to become slow (especially when opening the GNOME or KDE panel menu, or switching between applications), with applications appearing to “hang” for a couple of moments, that’s your cue to get more RAM: double it.

A faster processor won’t help at all, seeing as most of the time is spent on reading and writing to the disk swap partition. More swap won’t save you from this occurrence (indeed, it’s the accesses into swap which is making your applications “hang”), but not having it would be even worse for system performance (seeing as your system would still be short on memory, and it wouldn’t have a space to put unused data away).

What might help, if you’re short on money and long on free disk space, is setting up two swap partitions, each on a different disk connected to a different controller, and mounted with the same priority.

But remember that deciding to not be a cheap bastard and getting more RAM is the best course of action.