[Xapian-discuss] xapian performance

[Olly Betts]

On Mon, Nov 13, 2006 at 12:12:51PM -0200, Fernando Nemec wrote:
> Hi all, does anyone knows tweaks to compile Xapian under intel/linux
> to have, even if small, performance increase.

I'm not really aware of any compile time magic that can be worked.
If there was a magic GCC switch we'd be turning it on by default!

For a large search system, it's the I/O which dominates so getting
the compiler to generate better code is probably less important than it
is for something like a video codec.

There will undoubtably be some hotspots still, so you could try using
GCC's -fprofile-generate and -fprofile-use.  But I wouldn't expect
miracles.

If you have an x86_64 chip, compiling the code as 64 bit rather than 32
bit will probably be faster.

At runtime, if you've plenty of memory, setting env. variable
XAPIAN_FLUSH_THRESHOLD higher than the default of 10000 can speed up
indexing a lot.

Once built, running the database through quartzcompact or xapian-compact
(for flint) will make searches faster.

You should also make sure that your disk subsystem is set up to be fast.
Make sure DMA is enabled (if supported), etc.

There's also still plenty of scope for improving code to speed up
indexing, and some scope for faster searching too.  I know some places
where better algorithms can be used, but I suspect there are bottlenecks
in unobvious places too.  Profiling to identify these places would be
a useful activity.

Cheers,
    Olly