How to automate torrent downloads using TorrentFlux-b4rt, cron and rsync
TorrentFlux-b4rt is an awesome masterpiece of engineering. You install it on your Web server, and then you can start downloading BitTorrent torrents right away. The catch is that those torrents are saved in your Web server until you actually download them to your PC. And having to schedule downloads separately is a pain. Well, no more.
Awesome awesomeness: start your downloads minutes after TorrentFlux finishes them
I’ve created a very, very complicated and interwoven (translation: awesome and straightforward) script that you may use and modify on your home computer. It works like this:
- Using SSH, it queries your TorrentFlux server’s transfer list, asking which torrents are completely downloaded.
- For each of the completed torrents, it grabs the file name of the downloaded file.
- Then, it uses rsync to incrementally transfer the downloaded file to your PC.
- Again using rsync, it removes the file from your TorrentFlux server, freeing disk space.
What you need to do before using the script
For you to use it, you will need to do the following:
- Enable SSH access to your (soon-to-be) TorrentFlux server.
- Set up public key (passwordless) authentication so that your home computer’s user account can freely log in to your server without asking for a password.
- Install TorrentFlux-b4rt and set it up so that your SSH user has read/write access to the torrents and downloads folders/files. Umask/permissions, you know the drill.
- Enable
fluxcliusage on your TorrentFlux setup. - Install
rsyncon both your server and your home computer. - Install the BitTorrent package that contains the program named
torrentinfo-consoleon your server. - Create a destination directory on your home computer.
- Install a mail program such as
mailxon your home computer. - Install
cronon your home computer. - Configure the script altering the variables at the top.
Yeah, it’s a lot of work. Go bitch somewhere else, we’re here for the fun.
Installing the script
Put the script somewhere in your PATH. Make it executable. Set up a cron job for the purpose:
# m h dom mon dow command 0,10,20,30,40,50 * * * * /home/rudd-o/bin/torrentleecher -qMake sure the cron job line ends with a line ending (carriage return) — otherwise it doesn’t run.
The script itself
It’s a stunning, hackish example of subprocessing, SSH remoting, pipeline integration, text processing, POSIX daemonizing and file locking, and samizdat logging. It gets almost all of them wrong, and yet it manages to stay standing. If you’re interested in reading a complete account of how this script grew out of nothing, well, click here.
It’s a miracle:
#!/usr/bin/env python
from subprocess import Popen,PIPE,STDOUT,call import fcntl import re import os import sys import signal
FIXME: this script doesn't deal with multifile torrents
vars!
wherever they are used, it's MOST LIKELY they need quoting
FIXME whatever calls SSH remoting needs to protect/quote the commands for spaces or else this might turn out to be a bitch
torrentflux_base_dir = "/var/www/html/torrents/torrents" torrentflux_download_dir = "/var/www/html/torrents/torrents/incoming" torrentflux_server = "yourserver.com" torrentleecher_destdir = "/home/rudd-o/download/" fluxcli = "fluxcli" torrentinfo = "torrentinfo-console" email_address = "rudd-o@awesome.com"
def getstdout(cmdline): p = Popen(cmdline,stdout=PIPE) output = p.communicate()[0] if p.returncode != 0: raise Exception, "Command %s return code %s"%(cmdline,p.returncode) return output def getstdoutstderr(cmdline,inp=None): # return stoud and stderr in a single string object p = Popen(cmdline,stdin=PIPE,stdout=PIPE,stderr=STDOUT) output = p.communicate(inp)[0] if p.returncode != 0: raise Exception, "Command %s return code %s"%(cmdline,p.returncode) return output def passthru(cmdline): return call(cmdline) # return status code, pass the outputs thru def getssh(cmd): return getstdout(["ssh","-o","BatchMode yes","-o","ForwardX11 no",torrentflux_server] + [cmd]) # return stdout of ssh. doesn't return stderr def sshpassthru(cmd): return call(["ssh","-o","BatchMode yes","-o","ForwardX11 no",torrentflux_server] + [cmd]) # return status code from a command executed using ssh def mail(subject,text): return getstdoutstderr(["mail","-s",subject,email_address],text)
def get_finished_torrents(): stdout = getssh("%s transfers"%fluxcli) stdout = stdout.splitlines()[2:-5] stdout = [ re.match("^- (.+) - [0123456789.]+ MB - (Seeding|Done)",line) for line in stdout ] return [ match.group(1) for match in stdout if match ]
def get_file_name(torrentname): cmd = "LANG=C %s %s/.transfers/%s"%(torrentinfo,torrentflux_base_dir,torrentname) stdout = getssh(cmd).splitlines() filenames = [ l[22:] for l in stdout if l.startswith("file name...........: ") ] assert len(filenames) is 1 return filenames[0]
def dorsync(filename): cmdline = ["rsync","-avzP","--remove-source-files", "%s:%s/%s"%(torrentflux_server,torrentflux_download_dir,filename),"."] return passthru(cmdline)
def exists_on_server(filename): cmd = "test -f %s/%s"%(torrentflux_download_dir,filename) returncode = sshpassthru(cmd) if returncode == 1: return False elif returncode == 0: return True else: assert False # Not reached
def get_files_to_download(): torrents = get_finished_torrents() for t in torrents: yield (t,get_file_name(t))
def lock(): global f try: fcntl.lockf(f.fileno(),fcntl.LOCK_UN) f.close() except: pass try: f=open(os.path.join(torrentleecher_destdir,".torrentleecher.lock"), 'w') fcntl.lockf(f.fileno(),fcntl.LOCK_EX | fcntl.LOCK_NB) except IOError,e: if e.errno == 11: return False else: raise return True
def daemonize(): """Detach a process from the controlling terminal and run it in the background as a daemon. """ try: pid = os.fork() except OSError, e: raise Exception, "%s [%d]" % (e.strerror, e.errno)
if (pid == 0): # The first child.
os.setsid()
try: pid = os.fork() # Fork a second child.
except OSError, e: raise Exception, "%s [%d]" % (e.strerror, e.errno)
if (pid == 0): # The second child.
os.chdir("/")
else:
# exit() or _exit()? See below.
os._exit(0) # Exit parent (the first child) of the second child.
else: os._exit(0) # Exit parent of the first child.
import resource # Resource usage information.
maxfd = resource.getrlimit(resource.RLIMIT_NOFILE)[1]
if (maxfd == resource.RLIM_INFINITY):
maxfd = 1024
# Iterate through and close all file descriptors.
for f in [ sys.stderr, sys.stdout, sys.stdin ]:
try: f.flush()
except: pass
for fd in range(0, 2):
try: os.close(fd)
except OSError: pass
for f in [ sys.stderr, sys.stdout, sys.stdin ]:
try: f.close()
except: pass
sys.stdin = file("/dev/null", "r")
sys.stdout = file(os.path.join(torrentleecher_destdir,".torrentleecher.log"), "a",0)
sys.stderr = file(os.path.join(torrentleecher_destdir,".torrentleecher.log"), "a",0)
os.dup2(1, 2)
return(0)
sighandled = False def sighandler(signum,frame): global sighandled if not sighandled: print “Received signal %s”%signum # temporarily immunize from signals oldhandler = signal.signal(signum,signal.SIG_IGN) os.killpg(0,signum) signal.signal(signum,oldhandler) sighandled = True
def report_file_failed(filename): try: os.symlink(os.path.join(torrentleecher_destdir,”.torrentleecher.log”),”%s.log”%filename) except OSError,e: if e.errno != 17: raise #file exists should be ignored of course sys.stdout.flush() sys.stderr.flush() errortext = “”"Please take a look at the log files in %s”"”%torrentleecher_destdir mail(”Leecher: error — %s”%filename,errortext)
def report_file_done(filename): try: file(”%s is done”%filename,”w”).write(”Done”) except OSError,e: if e.errno != 17: raise #file exists should be ignored of course mail(”Leecher: done — %s”%filename,”The file is at %s”%torrentleecher_destdir)
def main(): if not ( len(sys.argv) > 1 and “-D” in sys.argv[1:] ): daemonize() os.chdir(torrentleecher_destdir) if not lock(): # we need to lock the file after the daemonization if not ( len(sys.argv) > 1 and “-q” in sys.argv[1:] ): print “Other process is downloading the file — add -q argument to command line to squelch this message” sys.exit(0) signal.signal(signal.SIGTERM,sighandler) signal.signal(signal.SIGINT,sighandler)
print "Starting download of finished torrents"
try:
for torrent,filename in get_files_to_download():
# no point in doing anything if the file was removed, right?
# so we continue if the file doesn't exist
if not exists_on_server(filename): continue
print "Downloading %s from torrent %s"%(filename,torrent)
retvalue = dorsync(filename)
if retvalue == 0:
report_file_done(filename)
print "Download of %s complete"%filename
# TODO: make use of fluxcli to kill the torrent out of TF
else:
if retvalue == 20:
print "Download of %s stopped -- rsync process interrupted"%(filename)
print "Finishing by user request"
sys.exit(2)
elif retvalue < 0:
report_file_failed(filename)
print "Download of %s failed -- rsync process killed with signal %s"%(filename,-retvalue)
print "Aborting"
sys.exit(1)
else:
report_file_failed(filename)
print "Download of %s failed -- rsync process exited with return status %s"%(filename,retvalue)
print "Aborting"
sys.exit(1)
print "Download of finished torrents complete"
except Exception,e:
report_file_failed("00 - General error")
raise
if name == “main“: main()
If you want to learn the process of how this script grew out of nothing, here’s the story.