[Xapian-discuss] Exception: Key too long

[Olly Betts]

On Thu, Sep 28, 2006 at 05:30:03PM +0200, Markus W?rle wrote:
> Key too long: length was 254 bytes, maximum length of a key is 
> BTREE_MAX_KEY_LEN
> 
> I am pretty sure that this is not my fault. In my code, I limited the 
> maximum term-length to 200 bytes (and no \0 bytes), and one of the 
> documents where the error occured has a maximum-term-lengh of 11. If 

I don't think it can fire under any other circumstances.

Are you aware that both terms added by add_term and add_posting matter?
So for example, a unique id term containing a URL or pathname can be
an issue.

> I'm trying to add one of the documents that thows these exceptions to 
> an empty database, everything works fine.

There's a rather unhelpful feature in how this currently works.  There's
not currently any explicit check on the term length.  Instead, this
exception is thrown deep inside the backend and it's checking the length
of the keys for the B-tree tables.  For the position table, this
includes the term name and document id, and the document id is encoded
in such a way that a larger document id can take up more bytes.  So the
same term may be fine in document 1, but cause an exception in document
1000.

And yes, I know this is rubbish - it's not trivial to add an explicit
check currently because of the zero byte encoding issue.

You could try adding the suspect document to an empty database using
"replace_document()" with a large document id (e.g. 33400 - the total
number of documents in the original system).  That should reproduce your
problem if you have the culprit.

Cheers,
    Olly