FIXME filter out requests with downgrading privileges, this will require a change in setuid and setgid first to save the uids for later privilege elevation
FIXME how to make pgsql not spit shit out to stdout or stderr
FIXME implement the status command with some stuff cool, because it sez metadata service inactivo pero subsistema bloqueado, probablemente por lo del pidfile que falta
FIXME the stop command of msctl NEEDS to wait for as long as needed till the metadata server is completely stopped, or wait a reasonable time and then abort.  This is required because the initscript restart() runs msctl stop and then msctl start one right after the other.
FIXME study the possibility of making the msctl stop command use the kill() function if it cannot contact the metadata service on a port, by getting the pid from a pidfile thta the metadata service should create and checking if it corresponds to it with the is running function that I wrote (needs a few mds because that test only for sys.argv[0] and in this case it would be 'python' duh), because if the ms is starting, the msctl stop command would fail horribly.
FIXME implement security credentials for interface servers and make stop and setload admin commands.
FIXME the erasing of not found files... make it not accessible with os.access() because the daemon may well run in the security context of an user
FIXME implement aautogrowing of inotify limits if possible.  this should be done at the time of instancing a watcher and at the time of watch() if any of them fail with their magic ENOSPC or no me acuerdo el otro codigo de error!
FIXME store the encoding, and on startup if the encoding changed, nuke both the index and the catalog
FIXME add manpages for metadata-service and msctl
FIXME add a subcommand to msctl to nuke the database
FIXME add a subcommand to msctl to KILL -9 metadata-service
FIXME add a pidlockfile mechanism to metadata-service and a config function in the Config class to get it.  This can then be used to raise EALREADY if the metadata service is already running on a specific datadir.
FIXME eliminate the needless Database abstraction and the Databases directory, we'll keep only the ZODBDatabase from that file.
FIXME break apart into independent pkg: DirectoryStorage/DirDict
FIXME we need a way to make the pgcatalog run analyze once in a while.  the pack() method of the pgcatalog should run vacuum analyze.
FIXME error handling on pgdatabasecluster, remove Exceptions and replace with proper exceptions, and make an exception if the database server is attempted to be started twice with the same port.
VERIFY immunize pgdatabasecluster.get_database against Ctrl+C-induced operationalerrors (around line 209 seems to be)
FIXME Thing is, instead of returning a list of URLs, it should be returning a list of dictionaries, where each dictionary MUST contain at least the "url" attribute, and this way we'll kick ass.  Basically, it should do this because that way we can return arbitrary properties "relevancy" and "mimetype" and others, which the user can request upon search time to be returned along with the URLs.  This way we can also brand results with the infospace name that brought it (should be responsibility of the index interface).
FIXME ponerles $Id$ a todos los .py y ejecutables y README ponerles la propiedad svn:keywords
FIXME ponerles header a todos los .py y ejecutables
FIXME las funciones de las public interfaces deben retornar True si todo anduvo bien. documentar esto.
FIXME Escribir contenido para el sitio Web de search services y cranear estructura bien.  (tiene que comprender varias aplicaciones, no solo este servicio). Arreglar package documentation y ponerle los URLs que quedarán finalmente.
FIXME arreglar diagrama uml hacerlo corresponder el modelo de datos del filesystem agent con la verdadera realidad vigente
FIXME implement smart query grouping by index type and merging by operator type
FIXME write NOR and NAND aggregator support for queries
FIXME make rankings/relevance work on NL searches and with other kinds of searches as well
FIXME estudiar como hacer para que los contents se marshaleen o no, bajo demanda del usuario
FIXME: Osea crear user-session-level small dcop client which searches the server, as a kuniqueapplication.  this should be easy to do given that this is all a matter of writing a kuniqueapplication in python.  Waiting for PyQt leaks to be solved.
FIXME delete files from the database when a search process is returning results.  Where to run this code?  Since the search server only knows about the database, how to sovle this provlem?  We may have solved that problem by detouring filesystem infospace queries to the filesystem querier.
FIXME ReiserFSStorage? create empty file (if it is possible, sparse file), chattr it +Sync+ Noatime, mount noatime+sync, and make it pack tails. overload DirectoryStorage.  When disk space runs low (NOT when it's full) (use os.statvfs), suspend database access (locking it), cat zeros into the end of the file, and resize_reiserfs online.  Check return status.
FIXME: reimplement search mechanisms to return scores and results sorted by score
FIXME create a detect_encoding function that returns the encoding for a string. and use it when the html charset/content-type detection does not work. ensure the detect_encoding algorithm only does the detection on the first 10K characters or so, so it won't take ages.
FIXME encode the text files with the appropriate encoding, using detect_encoding.
FIXME speed up processing of large HTML files (profile the performance of the plugin with the Showdown with the pinkertons file).
FIXME document the songs I've been listening to while doing this piece of crap
FIXME throttling increases should be as fast as now, but throttling decreases should be done at 1 tenth the speed the're been doing now
FIXME: in the vcrawler, if a volume is unmounted before the crawling is done, detect that and exit the crawler WITHOUT marking emitting the FullyCrawled signal, and del() the watcher, basically exit the crawler completely without emitting the said signal.
FIXME document the availability of PluginInterface in the global namespace for the plugins
SOLVED make plugin test command available in regular install
SOLVED break apart into independent packages: pgcatalog, pgdbcluster,  UnixSocketTransport
SOLVED set proper encoding on connections when connections are opened in pgcatalogconnection (the proper encoding would be the equivalent PG coding to the locale's getpreferredencoding()).   Check database link encoding:  The proper encoding for the database should always be UNICODE.  This is done at CREATE DATABASE time.  The link encoding is set with SET CLIENT_ENCODING TO 'value', where 'value' should be the metadata service's environment (default) locale encoding.  There is a table in the PG documentation.
SOLVED crete script metadata-service-ctl.  if any call to the metadata service returns True (meaning was well), sys.exit(0), if returns a faultcode sys.exit(faultcode). we need to reserve codes for standard unix errors such as cannot contact the metadata service.
SOLVED make pgcatalog an independent package
SOLVED make installation create /var/lib or the data dir the user requested.
SOLVED make all command line tools and the gui search tool detect the search port that it should use.  Maybe putting it into /dev?  nono, making a small command line app that returns the filename of the socket.  that app can use the config file routines to detect where it is.  client apps should depend on the existence of that command.
SOLVED using autotools: make the config file go to the correct config dir, and the RPM should mark that as a config file.  Perhaps using autotools instead of this piece of shit.
SOLVED: rpm put provides, requires into cfg file. Put correct category. put Copyright.
REJECTED: RPM  Put icon if possible.  
SOLVED solve the parsetab.py created by yacc in the PyQueryParser
SOLVED check if date searches work
SOLVED make DirectoryStorage go faster
SOLVED make search client in python, not in C++, SHIT.
SOLVED rejected excepcion InvalidFile error para plugins?
SOLVED implement locking for the database so a second instance with the same database just fails to start up
SOLVED implement locking for the catalog so a second instance with the same database just fails to start up
SOLVED investigate how to determine if a folder/volume has been fully indexed, to perform incremental indexing from then on. Maybe storing Volume objects in the database.
SOLVED moved to PostgreSQL: solve read conflict errors by retrying
SOLVED remove reliance on xmllint
SOLVED it works OK: test what happens if I submit Latin-1 XML to the search server, and which encoding is returned when searching/gettingMetadata for files with accents.  This test needs to be done while the metadata server is running in a different locale, such as C or something with ISO-8859-1
SOLVED reemplazar csv module por simple .split(" ") ya que los espacios ya estan escaped en el mtab
SOLVED need to implement a dict-like object which uses a directory as the storage target.  Performance issues?  If that can be accomplished, maybe we can forget about doing shit like storing the oid catalog in a dbm file, and directly use the directory.  The file names can very well be the hash keys
SOLVED make all transactions abort if there was an exception. This is already done in the indexer.
SOLVED optimize memory usage of the indexer
SOLVED rejected: make DirectoryStorage use lockfiles like FileStorage
SOLVED: check that the database does not grow when a file is reindexed.  Check that the database shrinks when a file is unindexed.
SOLVED: the string conundrum.  Decide whether I want to store unicode objects in the database, or just plain strings, and in which encoding.  SOLUTION: *Everything* works in the daemon's default locale.  XML-RPC queries work in UTF-8 space.
SOLVED, true: find out if gdbmStorage actually supports transactions and stuff!!!! I need to know that in order to see if it is safe to use.
SOLVED: check if .interface.sock exists (by plain opening it, and if it fails...) then announce it's deleting it (assuming there is no other search service running, a safe assumption to make?).
SOLVED, rejected, unmarshallable representations are a bug in a plugin: when marshalling stuff to be sent as metadata in a getMetadata request, if something's marshalling fails, simply do not send that key through the wire.  This helps avoid file representations which have unmarshallable data.
SOLVED: do not start crawlers on network volumes
SOLVED, they interlock by sharing a queue: make load checks more frequent? or make crawlers interlock so that only one can advance at a time?
SOLVED, unneeded now: document how to write database operations and the whole connection/root/transaction semantics.
SOLVED make somehow that the volume expert is told about the database dir, so it deems it inappropriate, just in case it's not mounted loop
SOLVED: (via GNOME) Need to investigate how to leverage freedesktop.org's mime-data package to detect mime types for files.  this package is one package everyone has!

