[Xapian-discuss] Optimization and Load balancing with Xapian

[Olly Betts]

On Fri, Feb 24, 2006 at 03:36:42PM +0200, David Levy wrote:
> new york
> real    4m10.437s
> user    0m0.459s
> sys     0m1.601s
> 
> !!!!!!!!!!!!!!!!!!!!!!!!
> 
> that's crazy :)

And rather revealing!

> what should i see with this figures ?

You want real - (user+sys) to be much lower.  Assuming there's no other
load on the server, and you're only running one test search at a time,
the difference will be pretty much entirely I/O overhead.

> maybe because today i don't hae many available RAM (i use most of my RAM to
> test RAMDISK for xapian on this server)

How much RAM does the server have?

I don't know if the Linux VM system understands what a RAM disk is - if
not it may be trying to cache blocks from it, which would be unhelpful
at best.

If Xapian's value.DB file is comfortably less than the free RAM, then
you could try using the database from disk, but before running tests
do:

cat value.DB > /dev/null

That should make sure most of the file is cached.

Hmm, one random thought - how many values are you adding per document?
Currently both quartz and flint have to retrieve all the values for a
document to get just one, which isn't a good design for how values
are generally used...  But that means having unused "sort keys" around
currently actually makes all sorting slower.

Cheers,
    Olly