FIXME make fscrawler detect removable volumes properly and remove them from the crawler list
FIXME make dynamic loading/configuration of installed Indexers.

This is nailed as well:  the index public interface relays search queries to each infospaceagent, aggregates results and returns them

WHAT STARTED THIS STUFF:

Consider a different system in which:

the Indexer process receives not files in a list, but actual object representations.
this would be of the form
...call(object_space_name,uniform_identifier,object)
...call("file_list","file:///etc/fstab",fstab_object)
from contributing processes.

the filesystem crawler (and watcher siblings) is the responsible for instantiating the
FILE object representations and sending them to the Indexer.
the indexer still maintains the responsibility of maintaining the database updated, as
new objects come from all contributing processes, and all other responsibilities.

this would enable storing objects like file listings from targz files, or Gaim log entries,
or web pages or or vCards in Evolution, via different contributor processes.
The FilesystemCrawler would be the one responsible for contributing files (and of course
any other processes needed to keep the index up-to-the-second up-to-date).

the "filesystem" infospace
the "web history" infospace (for each user)
the "evolution" infospace      ""
the "bookmarks" infospace      ""

(the web page thing may be different.  it can be easily collated into file object space)

(mull over whether a different plugin architecture is needed for different object spaces)

(actual database-related implementation details may be different.  Perhaps the file list
mapping should be renamed to "object_map", each representation object should have an
attribute "object_space" detailing a string which defines which object space this object
belongs to, and queries which require only files should add a filter defining that
we want something from only that object space)

-----------------------
THE THOUGHT PROCESS LEADING TO THIS CONCLUSION

How to store files?
- as a big list with an attribute for the volume ID and only one big Catalog
- in a Filesystem Infospace, which has several Volumes, with a list of files each and a Catalog as well
Implications:
 searches like
 [ filename: "/home/rudd-o" ]
need to be decomposed like this:
1. determine implied volumes' GUIDs by the paths
2. get the filesystem infospace and retrieve each implied volume's catalog
3. feed each catalog with its corresponding search subquery (something like this): [ filename: "/rudd-o" ], because paths would be relative to the volume root
3. query each volume's catalog for matching files
On which object do we assign responsibility for 1, 2 and 3?  It's clear they should mostly be in the domain of the FilesystemIndexer and friends.  If we put the smarts in the Volume (as a SearchableContainer class which would require a few methods to be implemented in inherited classes), we might have a clean solution.

How do we do queries 

Query types:
- "Global queries" Get me the beatles from everywhere
- "Filesystem queries" Get me the beatles from my filesystem
- "Path queries" Get me the beatles from my filesystem, below /home
- "Archive queries" Get me the beatles from all my archived volumes

a current search expression like this:

{ "attribute" : "artist", value="The Beatles", type="exact" }

a current operation looks like this:

{ "operation" : "or", [ list of {query...} ] }

We might extend all search expressions with an "infospace" : [ list of infospaces] argument, which the user can specify to determine the scope of the search.  Naturally, the infospace argument can only be accepted in the top-level query.

We also need a converter that converts this piece of shit XML-unserialized trash to proper Query packets.

What we might do is that when a search packet arrives, all requested and registered infospace searchers are supplied with the query packet, and their results are collated as per the operator dictated.  Each infospace has its own queriers, which they (?) register when they are started.  Infospace searchers have the responsibility to interpret and mangle each assigned query in its own fashion.  For example, the filesystem querier might interpret a "path" = "/home" to mean "volume_id" = "/home" AND "path" = "/".

Queriers return (the information objects? will that impact performance?  the catalog RIDs?  what if each infospace catalogs its files in its own way?


Tentative storage data model for FilesystemIndexer/FilesystemQuerier:
	a list of known Volumes, each with a dictionary of files named "contents", where the key is the filename relative to the volume root
	a master catalog which catalogs all files.  Queries for a specific volume should filter using a column "volume".  File objects should also hold a reference to their containing Volume.


