[Xapian-discuss] About Xapian vs ElasticSearch

[Peter Van Dijk]

As always, what you're doing has a huge impact on which tool you want to
use.

I cant really speak for how Xapian would go indexing logs, but in our case
we're indexing well over a billion records, using approximately 50,000
separate Xapian indexes.
One of the aspects of Xapian that we were attracted to was the ability for
us to separate out our customer data from top to bottom so that we have a
totally solid barrier between one customer to the next. This process also
lets us scale really really easily.

We dont have a massive lookup workload, but it's used fairly frequently,
and we're continually modifying the data. The moment a given index becomes
warm we're getting lookup times in the 100th of a second.

We werent able to get this kind of structure and performance with other
tools, let alone with the minimal system resources we need to handle it
with Xapian.
That being said we didnt look at elasticsearch when we built it originally,
so I dont have any data for you unfortunately.


Peter



On 24 June 2015 at 09:09, Michael Lewis <mal at icginc.com> wrote:

> I've done a number of small projects (<1 mil) docs w xapian. However when
> our needs grew to a couple of hundred million docs no go. Index times  get
> insanely slow very quickly We add about 250,000 docs a day. You can play
> games with switching indexes during updates and multiple remote dDBs. But
> why. Using ES we index 1000 new docs in 4 seconds. Our search times are
> subsecond in most non-phrase searches.
>
> ES Is easy to set up and maintain and expands easily. It is easily
> expanded and our go to choice for large projects.
>
> We still use xapian and are happy with it in more static situations. One
> thing about xapian -- the support by Olly and others is superb.
>
> Michael
>
> > On Jun 23, 2015, at 6:42 PM, Carlos Pacheco <cep_forum at icloud.com>
> wrote:
> >
> > I was looking for performance comparisons.
> >
> > I just found this link (
> http://blog.inoi.fi/2010/10/migrating-from-xapian-to-elasticsearch.html <
> http://blog.inoi.fi/2010/10/migrating-from-xapian-to-elasticsearch.html>).
> >
> > This was write on 2010.
> >
> > Could someone comment it ?
> > Is There some truth in what he said?
> >
> > I hate java, and didn’t want to back to java search engines...
> > _______________________________________________
> > Xapian-discuss mailing list
> > Xapian-discuss at lists.xapian.org
> > http://lists.xapian.org/mailman/listinfo/xapian-discuss
>
> _______________________________________________
> Xapian-discuss mailing list
> Xapian-discuss at lists.xapian.org
> http://lists.xapian.org/mailman/listinfo/xapian-discuss
>



-- 


*Peter van Dijk*
head of development



Ground Floor, 47 Warner Street
Fortitude Valley, QLD 4006
*P*: +61 7 3257 3906
*F*: +61 7 3257 3907
www.vision6.com.au

This e-mail message (and attachments) may contain information that is
confidential to Vision6. If you are not the intended recipient you cannot
use, distribute or copy the message or attachments. In such a case, please
notify the sender by return e-mail immediately and erase all copies of the
message and attachments. Opinions, conclusions and information in this
message and attachments that do not relate to the official business of
Vision6 are neither given nor endorsed by it.