Hi Olly,<br>
>>><br><br><div><span class="gmail_quote">On 2/16/06, <b class="gmail_sendername">Olly Betts</b> <<a href="mailto:olly@survex.com">olly@survex.com</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
On Thu, Feb 16, 2006 at 05:38:38PM +0200, David Levy wrote:<br>> Also I ask the 5 first hits in the omega request (HITSPERPAGE parameter, is<br>> it the better way ?)<br><br>No, that's the way to specify that.<br><br>
> > It's not the actual sorting which takes the extra time - the issue is<br>> > that for a multi-term query, relevance ranking can terminate early in<br>> > many cases (often when we reach the end of the matches for any of the
<br>> > terms). But if results are sorted on a value, we need to consider every<br>> > result which matches the query.<br>><br>> so you are telling me I won't be able to improve my calculation time if I
<br>> still use sorting ...?<br><br>You can try all the usual things to speed up searches - lots of RAM,<br>fast disks, compact the database, etc. Using flint instead of quartz<br>may help too. Some of the changes I have planned for flint will
<br>hopefully make a significant difference too - the way values are<br>currently stored doesn't lead itself to fast access in this case.<br><br>But sorting as currently designed does need to process every matching<br>document, which is going to be slow for a large database if the query
<br>matches a lot of documents.</blockquote><div><br>
<br>
Will this mecanism change in future releases ?<br>
<br>
I have compacted and removed large fields in the index. So the database is half the size ... but performance are still slow. <br>
I am thinking about using "ramdisks" maybe; and I am checking my hard disks too.<br>
Did you used ramdisks with Xapian yet ? Does it help ?<br>
<br>
</div><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">> Is there any other way to get results sorted by another criteria than<br>> relevance ?
<br><br>If you have only one sort order, and can arrange to add documents in<br>that order, then you can just use the raw document order for your<br>sorted search. This works particularly well for date ordering, since<br>
newly arrived documents end up in the right place. That's how the<br>Gmane search implements sort-by-date.</blockquote><div><br>
That would be a good idea, but I don't think I can because thoses values are dynamic.<br>
</div><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Actually, an interesting thing to note is that "sort by reverse date"<br>
can terminate early, while "sort by date" has to scan the whole docid<br>range (I plan to allow running postlists backwards which will make<br>"sort by date" as fast as "sort by reverse date" but I've not
<br>implemented that yet).<br><br>But even now, "sort by date" is still acceptably fast on 30 million<br>documents, which points the finger strongly towards accessing the values<br>as taking most of the time.</blockquote>
<div><br>
How was do you mean ? <br>
I was bad results with < 1M documents : <br>
Ending search for term in 0.199603 s with 271 matches : show<br>
Ending search for term in 0.153882 s with 1241 matches : human<br>
<br>
without sorting.<br>
I would really like results << 0.1 seconds for *every* query.<br>
<br>
However, I used the "collapse" parameter .. Is it time consuming even it there are no records to collapse in the results ? <br>
<br>
Regards<br>
</div><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Cheers,<br> Olly<br></blockquote></div><br><br clear="all"><br>-- <br>David LEVY {selenium}
<br>Website ~ <a href="http://www.davidlevy.org">http://www.davidlevy.org</a><br>Wishlist Zlio ~ <a href="http://david.zlio.com/wishlist">http://david.zlio.com/wishlist</a><br>Blog ~ <a href="http://selenium.blogspot.com">
http://selenium.blogspot.com</a><br>