FIXME store the encoding, and on startup if the encoding changed, nuke both the index and the catalog
FIXME make plugin test command available in regular install
FIXME add manpages for metadata-service and msctl
FIXME add a subcommand to msctl to nuke the database
FIXME add a subcommand to msctl to KILL -9 metadata-service
FIXME add a pidlockfile mechanism to metadata-service and a config function in the Config class to get it.  This can then be used to raise EALREADY if the metadata service is already running on a specific datadir.
FIXME eliminate the needless Database abstraction and the Databases directory, we'll keep only the ZODBDatabase from that file.
FIXME break apart into independent packages: pgcatalog, pgdbcluster, DirectoryStorage/DirDict, UnixSocketTransport
FIXME we need a way to make the pgcatalog run analyze once in a while.  the pack() method of the pgcatalog should run vacuum analyze.
FIXME set proper encoding on connections when connections are opened in pgcatalogconnection (the proper encoding would be the equivalent PG coding to the locale's getpreferredencoding()).   Check database link encoding:  The proper encoding for the database should always be UNICODE.  This is done at CREATE DATABASE time.  The link encoding is set with SET CLIENT_ENCODING TO 'value', where 'value' should be the metadata service's environment (default) locale encoding.  There is a table in the PG documentation.
FIXME error handling on pgdatabasecluster, remove Exceptions and replace with proper exceptions, and make an exception if the database server is attempted to be started twice with the same port.
FIXME immunize pgdatabasecluster.get_database against Ctrl+C-induced operationalerrors (around line 209 seems to be)
FIXME crete script metadata-service-ctl.  if any call to the metadata service returns True (meaning was well), sys.exit(0), if returns a faultcode sys.exit(faultcode). we need to reserve codes for standard unix errors such as cannot contact the metadata service.
FIXME make pgcatalog an independent package
FIXME make installation create /var/lib or the data dir the user requested.
FIXME make all command line tools and the gui search tool detect the search port that it should use.  Maybe putting it into /dev?  nono, making a small command line app that returns the filename of the socket.  that app can use the config file routines to detect where it is.  client apps should depend on the existence of that command.
FIXME make the config file go to the correct config dir, and the RPM should mark that as a config file.  Perhaps using autotools instead of this piece of shit.
FIXME: rpm put provides, requires into cfg file.  Put correct category.  Put icon if possible.  put Copyright.
FIXME Thing is, instead of returning a list of URLs, it should be returning a list of dictionaries, where each dictionary MUST contain at least the "url" attribute, and this way we'll kick ass.  Basically, it should do thi sbecause that way we can return arbitrary properties "relevancy" and "mimetype" and others, which the user can request upon search time to be returned along with the URLs.  This way we can also brand results with the infospace name that brought it (should be responsibility of the index interface).
FIXME ponerles $Id$ a todos los .py y ejecutables y README ponerles la propiedad svn:keywords
FIXME ponerles header a todos los .py y ejecutables
FIXME las funciones de las public interfaces deben retornar True si todo anduvo bien. documentar esto.
FIXME Escribir contenido para el sitio Web de search services y cranear estructura bien.  (tiene que comprender varias aplicaciones, no solo este servicio) Arreglar package documentation y ponerle los URLs que quedarán finalmente.
FIXME arreglar diagrama uml hacerlo corresponder el modelo de datos del filesystem agent con la verdadera realidad vigente
FIXME solve the parsetab.py created by yacc in the PyQueryParser
FIXME implement smart query grouping by index type and merging by operator type
FIXME check if date searches work
FIXME write NOR and NAND aggregator support for queries
FIXME make DirectoryStorage go faster
FIXME make rankings/relevance work on NL searches and with other kinds of searches as well
FIXME estudiar como hacer para que los contents se marshaleen o no, bajo demanda del usuario
FIXME make search client in python, not in C++, SHIT. Osea crear user-session-level small dcop client which searches the server, as a kuniqueapplication.  this should be easy to do given that this is all a matter of writing a kuniqueapplication in python.
FIXME delete files from the database when a search process is returning results.  Where to run this code?  Since the search server only knows about the database, how to sovle this provlem?  We may have solved that problem by detouring filesystem infospace queries to the filesystem querier.
FIXME excepcion InvalidFile error para plugins?
FIXME implement locking for the database so a second instance with the same database just fails to start up
FIXME ReiserFSStorage? create empty file (if it is possible, sparse file), chattr it +Sync+ Noatime, mount noatime+sync, and make it pack tails. overload DirectoryStorage.  When disk space runs low (NOT when it's full) (use os.statvfs), suspend database access (locking it), cat zeros into the end of the file, and resize_reiserfs online.  Check return status.
FIXME investigate how to determine if a folder/volume has been fully indexed, to perform incremental indexing from then on. Maybe storing Volume objects in the database.
FIXME: reimplement search mechanisms to return scores and results sorted by score
FIXME: solve read conflict errors by retrying
FIXME create a detect_encoding function that returns the encoding for a string. and use it when the html charset/content-type detection does not work. ensure the detect_encoding algorithm only does the detection on the first 10K characters or so, so it won't take ages.
FIXME remove reliance on xmllint
FIXME encode the text files with the appropriate encoding, using detect_encoding.
FIXME speed up processing of large HTML files (profile the performance of the plugin with the Showdown with the pinkertons file).
FIXME document the songs I've been listening to while doing this piece of crap
FIXME: test what happens if I submit Latin-1 XML to the search server, and which encoding is returned when searching/gettingMetadata for files with accents.  This test needs to be done while the metadata server is running in a different locale, such as C or something with ISO-8859-1
FIXME throttling increases should be as fast as now, but throttling decreases should be done at 1 tenth the speed the're been doing now
FIXME: in the vcrawler, if a volume is unmounted before the crawling is done, detect that and exit the crawler WITHOUT marking emitting the FullyCrawled signal.
FIXME document the availability of PluginInterface in the global namespace for the plugins
SOLVED reemplazar csv module por simple .split(" ") ya que los espacios ya estan escaped en el mtab
SOLVED need to implement a dict-like object which uses a directory as the storage target.  Performance issues?  If that can be accomplished, maybe we can forget about doing shit like storing the oid catalog in a dbm file, and directly use the directory.  The file names can very well be the hash keys
SOLVED make all transactions abort if there was an exception. This is already done in the indexer.
SOLVED optimize memory usage of the indexer
SOLVED rejected: make DirectoryStorage use lockfiles like FileStorage
SOLVED: check that the database does not grow when a file is reindexed.  Check that the database shrinks when a file is unindexed.
SOLVED: the string conundrum.  Decide whether I want to store unicode objects in the database, or just plain strings, and in which encoding.  SOLUTION: *Everything* works in the daemon's default locale.  XML-RPC queries work in UTF-8 space.
SOLVED, true: find out if gdbmStorage actually supports transactions and stuff!!!! I need to know that in order to see if it is safe to use.
SOLVED: check if .interface.sock exists (by plain opening it, and if it fails...) then announce it's deleting it (assuming there is no other search service running, a safe assumption to make?).
SOLVED, rejected, unmarshallable representations are a bug in a plugin: when marshalling stuff to be sent as metadata in a getMetadata request, if something's marshalling fails, simply do not send that key through the wire.  This helps avoid file representations which have unmarshallable data.
SOLVED: do not start crawlers on network volumes
SOLVED, they interlock by sharing a queue: make load checks more frequent? or make crawlers interlock so that only one can advance at a time?
SOLVED, unneeded now: document how to write database operations and the whole connection/root/transaction semantics.
SOLVED make somehow that the volume expert is told about the database dir, so it deems it inappropriate, just in case it's not mounted loop
SOLVED: (via GNOME) Need to investigate how to leverage freedesktop.org's mime-data package to detect mime types for files.  this package is one package everyone has!

