[Xapian-discuss] Optimization and Load balancing with Xapian

[David Levy]

hmmm that's not such a random idea ;-)
Actually I store 5 other values which I don't really use yet. Can I delete
them from my database without having to reindex it all (very long :()) ?

My value DB file is 1Go file. I have 2Go RAM, so I guess I could cache the
entire file yes ...

Actually I am using /dev/shm filesystem instead of RAMDISK for test purpose.

I think I should remove some values as I understand it would really help....


On 2/24/06, Olly Betts <olly at survex.com> wrote:
>
> On Fri, Feb 24, 2006 at 03:36:42PM +0200, David Levy wrote:
> > new york
> > real    4m10.437s
> > user    0m0.459s
> > sys     0m1.601s
> >
> > !!!!!!!!!!!!!!!!!!!!!!!!
> >
> > that's crazy :)
>
> And rather revealing!
>
> > what should i see with this figures ?
>
> You want real - (user+sys) to be much lower.  Assuming there's no other
> load on the server, and you're only running one test search at a time,
> the difference will be pretty much entirely I/O overhead.
>
> > maybe because today i don't hae many available RAM (i use most of my RAM
> to
> > test RAMDISK for xapian on this server)
>
> How much RAM does the server have?
>
> I don't know if the Linux VM system understands what a RAM disk is - if
> not it may be trying to cache blocks from it, which would be unhelpful
> at best.
>
> If Xapian's value.DB file is comfortably less than the free RAM, then
> you could try using the database from disk, but before running tests
> do:
>
> cat value.DB > /dev/null
>
> That should make sure most of the file is cached.
>
> Hmm, one random thought - how many values are you adding per document?
> Currently both quartz and flint have to retrieve all the values for a
> document to get just one, which isn't a good design for how values
> are generally used...  But that means having unused "sort keys" around
> currently actually makes all sorting slower.
>
> Cheers,
>     Olly
>



--
David LEVY {selenium}
Website ~ http://www.davidlevy.org
Wishlist Zlio ~ http://david.zlio.com/wishlist
Blog ~ http://selenium.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.tartarus.org/pipermail/xapian-discuss/attachments/20060224/3aaa0bd9/attachment.htm