[Xapian-discuss] Re: Re: get_docid over multi-database search

[James Aylett]

On Thu, Jan 10, 2008 at 03:35:50AM +0000, Olly Betts wrote:

> > I don't actually know how the matcher deals with multiple databases
> > right now, but I suspect it does it in a sort of pseudo-parallel [1],
> 
> Actually, we process databases sequentially in this case.  After the
> first database, we'll usually have an MSet full and so a decent minimum
> weight bound, so processing subsequent databases will usually be much
> quicker.
> 
> This is likely to be more friendly if the databases are on the same
> disk(s), though it probably doesn't parallelise load so well if they
> aren't.  But if your query load is high, concurrent queries will
> tend to do that for you anyway.

Both true. Hmm.

> I don't think we've tried processing databases in parallel, so it could
> be that would work better.  It would be an interesting experiment if
> somebody wanted to try it.

We'd need to devise a test case (better, several cases) with
concurrent queries, using some sort of valid (or validatable)
distribution of queries, against a database for which those queries
are valid.

Do you know (or can you look up) the proportion of GMane queries that
are restricted to a specific group?

J

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  james at tartarus.org                               uncertaintydivision.org