[Xapian-discuss] xapian performance

[Olly Betts]

On Tue, Nov 21, 2006 at 07:16:52PM -0200, Fernando Nemec wrote:
> After so many patches I opt to get a fresh new source copy from svn.
> As far as I see you committed almost all patches you produced in the
> last days.

So far I've only committed the changes to use "my_fls" instead of the
floating point log calculation.  The changes to open positionlists
lazily aren't in yet (I was waiting to check that the latest patch
fixed the slowdown for 2 term phrases).

> Sadly I didn't figure out any new improvement. I made a simple list
> with a variety of queries and all of them return in more or less the
> same time (a few tens of seconds).

The "my_fls" changes should reduce CPU use, so you won't see much
improvement if you're heavily I/O bound (which you must be if a search
takes tens of seconds).

> Is there any information I can supply to you to help to find what's
> going on phrase searches?

Could you compare the speed of phrase searches with this patch:

> > http://www.oligarchy.co.uk/xapian/patches/xapian-experimental-phrase-optimisation-v2.patch

with not using it (either on SVN trunk or 0.9.9).  Ideally it should
speed up phrases with 3 or more terms, but should be just as fast for
2 term phrases.

I'm going to look at creating a simple patch to count the number of
blocks read from each table during the query, which should help to get a
handle on how much I/O we're actually doing in an easily repeatable way.

Cheers,
    Olly