[Xapian-discuss] Search performance issues and profiling/debugging

[Ron Kass]

>> We ran the same test with databases 2,4,6,8 and 10 on a:
>>    Dell 2950,
>>        CentOS 5.0 64bit
>>        Two quad-Xeon CPUs (8 cores total)
>>        16GB mem
>>        6 disks in raid 5.
>>    (a nice and strong machine overall)
>>
>> While speed was better in most cases, it was still long for repeated 
>> searches.. around 0.2-0.3 seconds for sequential searches (what was 
>> around 1 second on the other machine).
>>
>> Notes:
>>    * There is mysql server running on that machine, but its not doing much.
>>     
>
> Hmm, oprofile shows 25-50% of the time is spent in mysqld...
>
> Otherwise, a lot of the time in Xapian seems to spent unpacking posting
> lists in flint.  That's not terribly suprising, but it might be
> worthwhile looking at the parts of that code which show up in these
> profiles to see if they can be optimised.
>
> It would be interesting to know if you get similar profiles with the
> bug in BM25Weight fixed (the one which gives varying weights when k2 is
> non-zero).  That was fixed in SVN after 1.0.4.
>
> Cheers,
>     Olly
>   
Actually this was the FASTER server, the other one was slower, and 
didn't have mysql on it. We only tried this one since it didn't have 
XenSource on it.
Now that the other server is without Xen and with the latest version 
(and we still do see slowness but didn't profile it yet) we don't have 
MySQL running at the same time, and we also use the default BM25 parameters.
We will profile it again soon..
Best regards,
Ron