[Xapian-discuss] Search performance issues and profiling/debugging search?

[Olly Betts]

On Wed, Oct 24, 2007 at 04:04:07AM +0200, Ron Kass wrote:
> I don't know anything yet about oprofile, will have to dig deeper there. 
> (any pinpoints would be handy)

I've written a quick guide (something I've been meaning to do for a
while):

http://wiki.xapian.org/ProfilingXapian
    
> What was not clear from your answers is if it makes sense that second 
> (and third) time searches take that long.

It suggests that the speed isn't limited by I/O but rather by CPU.
Profiling data would show us why and hopefully enable such cases to be
improved.

> First I ran a test on all 8 databases (1, 2, 4, 6, 7, 8, 9 and 10)
> 
> http://www.pidgintech.com/other/fts/test/test_1-2-4-6-7-8-9-10.txt
> 
> This resulted in Segmentation fault.

Where does it seg fault?  You can find out by running under gdb like so:

gdb --args <the command to run your test>

And then the seg fault should drop you back to the gdb prompt, where
you can type "bt" to show a backtrace.

> 1. Regarding your stable-sort, theory, if its something we can test, let 
> me know how.

I could provide a patch, but you'd need to be using SVN HEAD because the
code in this area has changed a lot since 1.0.3.

SVN HEAD is also a lot faster on some cases (more than twice as fast in
tests replaying tweakers.net query logs against their database).  So if
you're going to profile it would be a lot more useful to profile that.

> 2. What undefined value might anything depend on?

An uninitialised variable somewhere perhaps.

> 3. I don't know the testsuite and valgrind and what you refer to 
> regarding that..

Don't worry about the testsuite.  Just try running your test code under
valgrind:

valgrind <the command to run your test>

It'll run rather more slowly and use quite a bit more memory than
normal.  Any problems valgrind spots are reported to stderr by default.

Cheers,
    Olly