From gordevio2 at gmail.com  Fri Apr  2 08:51:56 2021
From: gordevio2 at gmail.com (henk l)
Date: Fri, 2 Apr 2021 09:51:56 +0200
Subject: indexing pdf errors
In-Reply-To: <20210331211451.GD8580@survex.com>
References: <f94b43a49a78b0b6aa27c40f36a490f8ca4db31b.camel@gmail.com>
 <20210331211451.GD8580@survex.com>
Message-ID: <CAEs6y99oAy99FDijqvkWu7GLfCHy6jAP6k0aAe19GU7dt2tgFQ@mail.gmail.com>

Thanks Olly for your reaction. You are right about the graphics issues.
Good idea to test it with pdftotext. I was not aware of the command.
Kind regards,
Henk

On Wed, 31 Mar 2021 at 23:14, Olly Betts <olly at survex.com> wrote:

> On Mon, Mar 29, 2021 at 10:10:39AM +0200, Henk L. wrote:
> > I have indexed pdf's with omindex. In this batch not all went well,
> > though the files got indexed. I get errors:
> >
> > Syntax Warning: Couldn't link the profiles
> > Syntax Warning: Can't create transform
> >
> > These pdf's are the output of a scan and ocr process. So they contain
> > text.
> >
> > Is there a way I can find out what happened?
>
> Assuming Xapian 1.4.x where x >= 10, we extract text from PDFs by piping
> them to:
>
>     pdftotext -enc UTF-8 - -
>
> For some PDF files pdftotext emits warning messages.
>
> These are presumably due to either invalid structure within the PDF file
> (either due to bugs in the tool that made it, or corruption of the file
> since) or possibly bugs in libpoppler (which pdftotext uses to do most
> of the actual work.)
>
> Most of them sound like things that aren't important for extracting just
> the text (e.g. "Can't create transform" sounds like a graphics
> coordinate mapping problem.)
>
> You can test by hand with the command above and see what text is
> actually extracted.
>
> Possibly we should run pdftotext with -q to disable such messages,
> though that also seems like it makes errors silent too, which is less
> helpful.  There doesn't seem to be any finer level of control.
>
> (Or if you're running Omega from git master you may be using the new
> worker module for libpoppler, but mostly that just means we don't fork()
> and exec() for each PDF file indexed - these messages are still coming
> from libpoppler, and we could set the option that pdftotext -q sets to
> disable them.)
>
> Cheers,
>     Olly
>

From deisler.he at gmail.com  Tue Apr 20 03:34:31 2021
From: deisler.he at gmail.com (Eden He)
Date: Tue, 20 Apr 2021 10:34:31 +0800
Subject: need help on FLAG_PARTIAL
Message-ID: <CABaDr5Yynh58=rGrkAK2HG7ZcgNuwtnVL4dU3e-c1+xMHboWfQ@mail.gmail.com>

Hi,
I have some docs with a fields named "phone",
something like that:
{
ID int
Name str
Phone str
}
I created an index for them and gave a prefix "P" to the field "phone",
and there is one doc whose value of phone is "53083763748".
However, when I searched that field using FLAG_PARTIAL, I can only get
results when I used the following code:

queryparser.add_prefix("phone", "P")
query = queryparser.parse_query("phone:5", xapian.QueryParser.FLAG_PARTIAL)

when I expand the query to "53" or "530", I can only get nothing.

queryparser.add_prefix("phone", "P")
query = queryparser.parse_query("phone:53", xapian.QueryParser.FLAG_PARTIAL)

there must be something wrong, but I cannot tell where it is. Would you
please help me with this?
Thanks.

--
all the best.

From olly at survex.com  Tue Apr 20 04:33:50 2021
From: olly at survex.com (Olly Betts)
Date: Tue, 20 Apr 2021 04:33:50 +0100
Subject: need help on FLAG_PARTIAL
In-Reply-To: <CABaDr5Yynh58=rGrkAK2HG7ZcgNuwtnVL4dU3e-c1+xMHboWfQ@mail.gmail.com>
References: <CABaDr5Yynh58=rGrkAK2HG7ZcgNuwtnVL4dU3e-c1+xMHboWfQ@mail.gmail.com>
Message-ID: <20210420033350.GF8580@survex.com>

On Tue, Apr 20, 2021 at 10:34:31AM +0800, Eden He wrote:
> I created an index for them and gave a prefix "P" to the field "phone",
> and there is one doc whose value of phone is "53083763748".
> However, when I searched that field using FLAG_PARTIAL, I can only get
> results when I used the following code:
> 
> queryparser.add_prefix("phone", "P")
> query = queryparser.parse_query("phone:5", xapian.QueryParser.FLAG_PARTIAL)
> 
> when I expand the query to "53" or "530", I can only get nothing.
> 
> queryparser.add_prefix("phone", "P")
> query = queryparser.parse_query("phone:53", xapian.QueryParser.FLAG_PARTIAL)
> 
> there must be something wrong, but I cannot tell where it is. Would you
> please help me with this?

I can't reproduce your problem.

Creating the database:

$ python3 -c 'import xapian; db=xapian.WritableDatabase("tmp.db"); doc=xapian.Document(); doc.add_term("P53083763748"); db.add_document(doc); db.commit()'

And:

$ python3 -c 'import xapian; db=xapian.Database("tmp.db"); enq=xapian.Enquire(db); qp=xapian.QueryParser(); qp.add_prefix("phone", "P")
for q in (5, 53, 530): enq.set_query(qp.parse_query("phone:"+str(q), xapian.QueryParser.FLAG_PARTIAL)); print("%d -> %d match" % (q, enq.get_mset(0,10).size()))'
5 -> 1 match
53 -> 1 match
530 -> 1 match

If you still can't get it to work, please post a simple script which
demonstrates it not working.

Cheers,
    Olly


From deisler.he at gmail.com  Tue Apr 20 08:37:58 2021
From: deisler.he at gmail.com (Eden He)
Date: Tue, 20 Apr 2021 15:37:58 +0800
Subject: need help on FLAG_PARTIAL
In-Reply-To: <20210420033350.GF8580@survex.com>
References: <CABaDr5Yynh58=rGrkAK2HG7ZcgNuwtnVL4dU3e-c1+xMHboWfQ@mail.gmail.com>
 <20210420033350.GF8580@survex.com>
Message-ID: <CABaDr5Y862QNM3en_NVai9CN8Jd5wC-T5twdNRehFua+cfeGkg@mail.gmail.com>

Hi Olly,

Thanks for your code, I finally figured out the problem, the query I passed
to parse_query is incorrectly enclosed by quotation marks.
Thank you so much.

Olly Betts <olly at survex.com> ?2021?4?20??? ??11:33???

> On Tue, Apr 20, 2021 at 10:34:31AM +0800, Eden He wrote:
> > I created an index for them and gave a prefix "P" to the field "phone",
> > and there is one doc whose value of phone is "53083763748".
> > However, when I searched that field using FLAG_PARTIAL, I can only get
> > results when I used the following code:
> >
> > queryparser.add_prefix("phone", "P")
> > query = queryparser.parse_query("phone:5",
> xapian.QueryParser.FLAG_PARTIAL)
> >
> > when I expand the query to "53" or "530", I can only get nothing.
> >
> > queryparser.add_prefix("phone", "P")
> > query = queryparser.parse_query("phone:53",
> xapian.QueryParser.FLAG_PARTIAL)
> >
> > there must be something wrong, but I cannot tell where it is. Would you
> > please help me with this?
>
> I can't reproduce your problem.
>
> Creating the database:
>
> $ python3 -c 'import xapian; db=xapian.WritableDatabase("tmp.db");
> doc=xapian.Document(); doc.add_term("P53083763748"); db.add_document(doc);
> db.commit()'
>
> And:
>
> $ python3 -c 'import xapian; db=xapian.Database("tmp.db");
> enq=xapian.Enquire(db); qp=xapian.QueryParser(); qp.add_prefix("phone", "P")
> for q in (5, 53, 530): enq.set_query(qp.parse_query("phone:"+str(q),
> xapian.QueryParser.FLAG_PARTIAL)); print("%d -> %d match" % (q,
> enq.get_mset(0,10).size()))'
> 5 -> 1 match
> 53 -> 1 match
> 530 -> 1 match
>
> If you still can't get it to work, please post a simple script which
> demonstrates it not working.
>
> Cheers,
>     Olly
>


-- 
all the best.

