[Xapian-discuss] Behavior of WritableDatabase against an "add_database"

[James Aylett]

On Fri, Nov 03, 2006 at 10:33:34PM -0200, Rafael SDM Sierra wrote:

> But, still the suggest of implement some load balance feature in
> some future version (2.0 maybe...oO)

I don't think you can load balance like this because of locking (I'm
not even convinced you *can* load balance on write, but I'm probably
wrong). However you can cluster like this, which may be what you mean.

This could be implemented in Xapian, but I'm not convinced it's the
right place, because the logic for how to segment documents would
still need to come in your code. We could do something like:

----------------------------------------------------------------------
db = xapian.WritableDatabase(discriminant_functor)
db.add_database(xapian.remote_open_writable('192.168.0.1',6666))
db.add_database(xapian.remote_open_writable('192.168.0.2',6666))
----------------------------------------------------------------------

or something, with discriminant_functor being a callback to figure out
which database to use ... but how do you return the db? By index? By
name? Say it's the former, and you write the functor to split based on
the start of the file path of the document (in python, and assuming a
get_field decorator as I proposed in the past). This is slightly
contrived, but:

----------------------------------------------------------------------
def discriminant_functor(doc):
    discrim = doc.get_field('path')[:4]
    d_to_db = { '/usr': 1, '/var': 2, '/hom/': 3 }
    db_idx = d_to_db.get(discrim, 0)
    return db_idx
----------------------------------------------------------------------

Then two changes does this entirely in your code, without Xapian
needing the feature:

----------------------------------------------------------------------
def add_document(doc, dbs): # dbs is a list
    discrim = doc.get_field('path')[:4]
    d_to_db = { '/usr': 1, '/var': 2, '/hom/': 3 }
    db_idx = d_to_db.get(discrim, 0)
    dbs[db_idx].add_document(doc)
----------------------------------------------------------------------

J    

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  james at tartarus.org                               uncertaintydivision.org