From olly at survex.com  Tue Jan  3 17:00:26 2006
From: olly at survex.com (Olly Betts)
Date: Tue Jan  3 17:00:28 2006
Subject: [Xapian-devel] opensearch template
Message-ID: <20060103170026.GQ22250@survex.com>

I've just noticed the Omega "opensearch" template seems to lack escaping
for URLs in <link> tags:

$httpheader{Content-Type,application/xml}<?xml version="1.0"?>
  <rss version="2.0" xmlns:openSearch="http://a9.com/-/spec/opensearchrss/1.0/">
    <channel>
      <title>Xapian search results</title>
      <link>$env{SCRIPT_NAME}?DB=$dbname&amp;P=$html{$query}</link>
      <description>Search results for "$html{$query}" at xapian.org</description>
      <language>en</language>
      <copyright>(C)2005, Xapian.org</copyright>
      <openSearch:totalResults>$msize</openSearch:totalResults>
      <openSearch:startIndex>$add{topdoc,1}</openSearch:startIndex>
      <openSearch:itemsPerPage>$hitsperpage</openSearch:itemsPerPage>
$hitlist{
      <item>
        <title>$html{$or{$field{title},$field{caption}}}</title>
        <link>$field{url}</link>
        <description>$highlight{$field{sample},$terms,&lt;strong&gt;,&lt;/strong&gt;}$if{$field{sample},...}</description>
      </item>}
    </channel>
  </rss>

Surely $field{url} needs some sort of escaping?  Also, we aren't
escaping $env{SCRIPT_NAME} or $dbname in the <channel>'s <link> tag.

Sadly, the opensearch spec doesn't really seem cover the format of
URLs in <link> tags, but the "&amp;" in the <channel>'s <link> makes
me suspect that $html{ } is the right answer.

But reading the spec, it does say that any HTML in the description tag
should be escaped once, or enclosed as CDATA:

http://opensearch.a9.com/docs/devfaq.jsp#html

Since we seem to be using RSS instead of ATOM, I think that means we
need a $html{ } around the contents of the description tag.

Cheers,
    Olly

From james-xapian at tartarus.org  Tue Jan  3 17:15:07 2006
From: james-xapian at tartarus.org (James Aylett)
Date: Tue Jan  3 17:15:10 2006
Subject: [Xapian-devel] opensearch template
In-Reply-To: <20060103170026.GQ22250@survex.com>
References: <20060103170026.GQ22250@survex.com>
Message-ID: <20060103171507.GJ10528@tartarus.org>

On Tue, Jan 03, 2006 at 05:00:26PM +0000, Olly Betts wrote:

> $httpheader{Content-Type,application/xml}<?xml version="1.0"?>
>   <rss version="2.0" xmlns:openSearch="http://a9.com/-/spec/opensearchrss/1.0/">
>     <channel>
>       <title>Xapian search results</title>
>       <link>$env{SCRIPT_NAME}?DB=$dbname&amp;P=$html{$query}</link>
>       <description>Search results for "$html{$query}" at xapian.org</description>
>       <language>en</language>
>       <copyright>(C)2005, Xapian.org</copyright>
>       <openSearch:totalResults>$msize</openSearch:totalResults>
>       <openSearch:startIndex>$add{topdoc,1}</openSearch:startIndex>
>       <openSearch:itemsPerPage>$hitsperpage</openSearch:itemsPerPage>
> $hitlist{
>       <item>
>         <title>$html{$or{$field{title},$field{caption}}}</title>
>         <link>$field{url}</link>
>         <description>$highlight{$field{sample},$terms,&lt;strong&gt;,&lt;/strong&gt;}$if{$field{sample},...}</description>
>       </item>}
>     </channel>
>   </rss>
> 
> Surely $field{url} needs some sort of escaping?  Also, we aren't
> escaping $env{SCRIPT_NAME} or $dbname in the <channel>'s <link> tag.

Yeah, they should all be escaped appropriately.

> Sadly, the opensearch spec doesn't really seem cover the format of
> URLs in <link> tags, but the "&amp;" in the <channel>'s <link> makes
> me suspect that $html{ } is the right answer.

There isn't any structure beyond "they are URLs". It's an XML-based
format, so omegascripts' (slightly confusingly-named) $html{ } should
do fine.

> But reading the spec, it does say that any HTML in the description tag
> should be escaped once, or enclosed as CDATA:
> 
> http://opensearch.a9.com/docs/devfaq.jsp#html
> 
> Since we seem to be using RSS instead of ATOM, I think that means we
> need a $html{ } around the contents of the description tag.

If you do, the &lt; and &gt; in the $highlight{...} need to change to
< and > respectively, or you'll have "double escaped" them. I hate the
opensearch spec, it's written as if the RSS 2.0 document were a spec.

In theory we can re-work opensearch over Atom, but you get into fiddly
areas because Atom requires GUIDs. There wasn't enough interest on the
Atom list to get anywhere with the idea, as far as I can tell (and
neither Richard nor I have had time).

The opensearch template isn't really finished - it was more a quick
hack to see how fiddly it would be.

J

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  james@tartarus.org                               uncertaintydivision.org

From olly at survex.com  Tue Jan  3 17:41:25 2006
From: olly at survex.com (Olly Betts)
Date: Tue Jan  3 17:41:27 2006
Subject: [Xapian-devel] opensearch template
In-Reply-To: <20060103171507.GJ10528@tartarus.org>
References: <20060103170026.GQ22250@survex.com>
	<20060103171507.GJ10528@tartarus.org>
Message-ID: <20060103174125.GS22250@survex.com>

On Tue, Jan 03, 2006 at 05:15:07PM +0000, James Aylett wrote:
> On Tue, Jan 03, 2006 at 05:00:26PM +0000, Olly Betts wrote:
> > But reading the spec, it does say that any HTML in the description tag
> > should be escaped once, or enclosed as CDATA:
> > 
> > http://opensearch.a9.com/docs/devfaq.jsp#html
> > 
> > Since we seem to be using RSS instead of ATOM, I think that means we
> > need a $html{ } around the contents of the description tag.
> 
> If you do, the &lt; and &gt; in the $highlight{...} need to change to
> < and > respectively, or you'll have "double escaped" them.

I think the double escaping is correct.

If we search for "1" and a description is:

1 < 2

Then $highlight produces (assuming we're using <b>...</b> to highlight):

<b>1</b> &lt; 2

And the opensearch response will contain:

<description>&lt;b&gt;1&lt;/b&gt; &amp;lt; 2</description>

Then the code handling the opensearch response unescapes the contents of
<description> and gets:

<b>1</b> &lt; 2

Which is correct.  If "&lt;" were "<" from $highlight, at this point
we'd have "<b>1</b> < 2" which is broken HTML.

Or am I missing something?

> In theory we can re-work opensearch over Atom, but you get into fiddly
> areas because Atom requires GUIDs. There wasn't enough interest on the
> Atom list to get anywhere with the idea, as far as I can tell (and
> neither Richard nor I have had time).

I think it's been done now - the opensearch site talks about use with
RSS or Atom anyway.

> The opensearch template isn't really finished - it was more a quick
> hack to see how fiddly it would be.

OK, but we really shouldn't be shipping stuff with missing escaping.
That's where XSS security holes come from...

Cheers,
    Olly

From james-xapian at tartarus.org  Tue Jan  3 17:48:58 2006
From: james-xapian at tartarus.org (James Aylett)
Date: Tue Jan  3 17:48:59 2006
Subject: [Xapian-devel] opensearch template
In-Reply-To: <20060103174125.GS22250@survex.com>
References: <20060103170026.GQ22250@survex.com>
	<20060103171507.GJ10528@tartarus.org>
	<20060103174125.GS22250@survex.com>
Message-ID: <20060103174858.GA31336@tartarus.org>

On Tue, Jan 03, 2006 at 05:41:25PM +0000, Olly Betts wrote:

> > If you do, the &lt; and &gt; in the $highlight{...} need to change to
> > < and > respectively, or you'll have "double escaped" them.
> 
> I think the double escaping is correct.

No, you're completely right. Silly me.

> > In theory we can re-work opensearch over Atom, but you get into fiddly
> > areas because Atom requires GUIDs. There wasn't enough interest on the
> > Atom list to get anywhere with the idea, as far as I can tell (and
> > neither Richard nor I have had time).
> 
> I think it's been done now - the opensearch site talks about use with
> RSS or Atom anyway.

There were issues with the simplicity of opensearch some people wanted
to address. Also trying to get a solid specification for opensearch+
rather than one which relies on the underlying definitions.

> > The opensearch template isn't really finished - it was more a quick
> > hack to see how fiddly it would be.
> 
> OK, but we really shouldn't be shipping stuff with missing escaping.

Absolutely. I don't remember deciding to ship it, which probably means
it was a mistake.

> That's where XSS security holes come from...

Well, (a) we shouldn't be building XML without at least running it
through a well-formedness check; (b) HTML shouldn't really be embedded
using escaping, it should be done by reference or in CDATA which is a
big step towards avoiding XSS-like problems because it removes the
multi-escaping issue entirely.

CDATA inside RSS causes problems for some clients, however (bah). (a)
is probably worth looking at at some point - a mechanism which allows
omega to run the generated string through an empty SAX parser to check
it isn't rubbish.

That doesn't solve the problem, however. I'm not convinced building
XML using omegascript is a good idea; we should be using a tool built
on GenX or something. (Now... a variant of omegascript that outputs
generated XML where the presence or absence of elements/data was
configurable - that would be very nice. If it could feed into an STX
processor, so much the better.)

J

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  james@tartarus.org                               uncertaintydivision.org

From jean-francois.dockes at wanadoo.fr  Fri Jan 13 06:48:10 2006
From: jean-francois.dockes at wanadoo.fr (Jean-Francois Dockes)
Date: Fri Jan 13 16:21:00 2006
Subject: [Xapian-devel] xapian-config --libs outputs libstdc++.la as a
	dependency
Message-ID: <17351.19754.124115.873666@mouton.dockes.com>


Hello,

I am hearing of users having trouble linking with libxapian (on slackware
and gentoo systems, and 0.9.2 I think), and I am not too sure where the
problem comes from, or what the correct solution could be, so I am just
asking here in case someone has a quick answer.

What happens is that "xapian-config --libs" outputs libstdc++.la in the
list of libraries. Something like:

-L/usr/i686-pc-linux-gnu/bin -L/usr/i686-pc-linux-gnu/lib \
 /usr/lib/gcc/i686-pc-linux-gnu/3.4.4/libstdc++.la 

As recoll is linked with g++, this generates an error (I guess it might
work if libtool was used?).

The libstdc++ data seems to come from libxapian.la ('dependency_libs'
variable).

I don't see this happening on any on my systems (FreeBSD, Solaris, Redhat
7.3, Fedora core 4), so I guess I'll have to install gentoo somewhere to
look into this, except if someone here can tell me what goes wrong and what
I could do about it.

I am pasting a libxapian.la (from a gentoo system) at the end of this message.

Regards,

J.F. Dockes


# libxapian.la - a libtool library file
# Generated by ltmain.sh - GNU libtool 1.5.18 (1.1220.2.245 2005/05/16 08:55:27)
#
# Please DO NOT delete this file!
# It is necessary for linking the library.

# The name that we can dlopen(3).
dlname='libxapian.so.8'

# Names of this library.
library_names='libxapian.so.8.0.0 libxapian.so.8 libxapian.so'

# The name of the static archive.
old_library='libxapian.a'

# Libraries that this one depends upon.
dependency_libs=' -L/usr/i686-pc-linux-gnu/bin -L/usr/i686-pc-linux-gnu/lib /usr/lib/gcc/i686-pc-linux-gnu/3.4.4/libstdc++.la'

# Version information for libxapian.
current=8
age=0
revision=0

# Is this an already installed library?
installed=yes

# Should we warn about portability when linking against -modules?
shouldnotlink=no

# Files to dlopen/dlpreopen
dlopen=''
dlpreopen=''

# Directory that this library needs to be installed in:
libdir='/usr/lib'

From olly at survex.com  Fri Jan 13 21:47:48 2006
From: olly at survex.com (Olly Betts)
Date: Fri Jan 13 21:47:49 2006
Subject: [Xapian-devel] xapian-config --libs outputs libstdc++.la as a
	dependency
In-Reply-To: <17351.19754.124115.873666@mouton.dockes.com>
References: <17351.19754.124115.873666@mouton.dockes.com>
Message-ID: <20060113214748.GW11445@survex.com>

On Fri, Jan 13, 2006 at 07:48:10AM +0100, Jean-Francois Dockes wrote:
> What happens is that "xapian-config --libs" outputs libstdc++.la in the
> list of libraries. Something like:
> 
> -L/usr/i686-pc-linux-gnu/bin -L/usr/i686-pc-linux-gnu/lib \
>  /usr/lib/gcc/i686-pc-linux-gnu/3.4.4/libstdc++.la 

Hmm, that's not good.

I guess we need to recursively check for .la files in dependency_libs
and substitute the actual library to link against, plus the contents of
the .la file's dependency_libs.  That's a bit fiddly though...

We might be able to get away with just dropping libstdc++.la, since
linking C++ code should pull it in anyway.  I can't see any other
.la files making it in there unless we start depending on other
libraries.

Anyway, I'll take a look.

> As recoll is linked with g++, this generates an error (I guess it might
> work if libtool was used?).

If you're using libtool, then xapian-config --ltlibs should be used
instead (which happens automatically if you use XO_LIB_XAPIAN in
configure.in/configure.ac and use AC_PROG_LIBTOOL or AM_PROG_LIBTOOL).

Cheers,
    Olly

From olly at survex.com  Sat Jan 14 02:49:00 2006
From: olly at survex.com (Olly Betts)
Date: Sat Jan 14 02:49:01 2006
Subject: [Xapian-devel] xapian-config --libs outputs libstdc++.la as a
	dependency
In-Reply-To: <20060113214748.GW11445@survex.com>
References: <17351.19754.124115.873666@mouton.dockes.com>
	<20060113214748.GW11445@survex.com>
Message-ID: <20060114024900.GD11445@survex.com>

On Fri, Jan 13, 2006 at 09:47:48PM +0000, Olly Betts wrote:
> On Fri, Jan 13, 2006 at 07:48:10AM +0100, Jean-Francois Dockes wrote:
> > What happens is that "xapian-config --libs" outputs libstdc++.la in the
> > list of libraries. Something like:
> > 
> > -L/usr/i686-pc-linux-gnu/bin -L/usr/i686-pc-linux-gnu/lib \
> >  /usr/lib/gcc/i686-pc-linux-gnu/3.4.4/libstdc++.la 
> 
> Hmm, that's not good.

I've attached a simple patch which should fix this in most cases.  I'll
probably go with this for the forthcoming release at least.  It'll work
so long as no .la files listed in libxapian.la's dependency_libs have
dependency_libs themselves that aren't pulled in anyway.

The patch is for xapian-config.  You'll need to patch xapian-config.in
instead if you patch before running configure (the same patch should
apply to either).

Cheers,
    Olly
-------------- next part --------------
--- xapian-config.orig
+++ xapian-config
@@ -132,6 +132,14 @@
 	test /usr/lib != "$libdir" && L="-L$libdir "
 	D=
 	eval `grep ^dependency_libs= "$libdir/libxapian.la"`
+	# Replace "/path/to/libfoo.la" with "-L/path/to -lfoo" which doesn't
+	# take into account any dependency_libs which libfoo.la may pull in
+	# but the only .la that we're likely to see is libstdc++.la and it'll
+	# hopefully work there as any dependency libs will probably be pulled
+	# in by the compiler for us.  Recursively pulling in dependency_libs
+	# is probably too hard to do in sh - we'd need to reimplement
+	# xapian-config in C or C++.
+	dependency_libs=`echo " $dependency_libs "|sed 's![     ][      ]*!  !g;s! \(/[^ ]*\)/lib\([^ ]*\).la ! -L\1 -l\2 !g;s!  ! !g;s!^ !!;s! $!!'`
 	test -n "$dependency_libs" && D=" $dependency_libs"
 	echo "$F$L-lxapian$D"
 	;;
From jean-francois.dockes at wanadoo.fr  Sat Jan 14 06:58:11 2006
From: jean-francois.dockes at wanadoo.fr (Jean-Francois Dockes)
Date: Sat Jan 14 06:58:18 2006
Subject: [Xapian-devel] xapian-config --libs outputs libstdc++.la as a
	dependency
In-Reply-To: <20060114024900.GD11445@survex.com>
References: <17351.19754.124115.873666@mouton.dockes.com>
	<20060113214748.GW11445@survex.com>
	<20060114024900.GD11445@survex.com>
Message-ID: <17352.41219.947435.17457@mouton.dockes.com>

Olly Betts writes:
 > On Fri, Jan 13, 2006 at 07:48:10AM +0100, Jean-Francois Dockes wrote:
 > > What happens is that "xapian-config --libs" outputs libstdc++.la in the
 > > list of libraries. Something like:
 > 
 > I've attached a simple patch which should fix this in most cases.  I'll
 > probably go with this for the forthcoming release at least.  It'll work
 > so long as no .la files listed in libxapian.la's dependency_libs have
 > dependency_libs themselves that aren't pulled in anyway.
 > ...

Whoa, this was fast. I wanted to check the gentoo xapian build process to
see if there was something special in there that triggered the condition,
but you left me no time ...

Anyway, I am going to try and check how the patch can be added to the
gentoo portage script ('ebuild').

J.F. Dockes


From olly at survex.com  Tue Jan  3 17:00:26 2006
From: olly at survex.com (Olly Betts)
Date: Tue, 3 Jan 2006 17:00:26 +0000
Subject: [Xapian-devel] opensearch template
Message-ID: <20060103170026.GQ22250@survex.com>

I've just noticed the Omega "opensearch" template seems to lack escaping
for URLs in <link> tags:

$httpheader{Content-Type,application/xml}<?xml version="1.0"?>
  <rss version="2.0" xmlns:openSearch="http://a9.com/-/spec/opensearchrss/1.0/">
    <channel>
      <title>Xapian search results</title>
      <link>$env{SCRIPT_NAME}?DB=$dbname&amp;P=$html{$query}</link>
      <description>Search results for "$html{$query}" at xapian.org</description>
      <language>en</language>
      <copyright>(C)2005, Xapian.org</copyright>
      <openSearch:totalResults>$msize</openSearch:totalResults>
      <openSearch:startIndex>$add{topdoc,1}</openSearch:startIndex>
      <openSearch:itemsPerPage>$hitsperpage</openSearch:itemsPerPage>
$hitlist{
      <item>
        <title>$html{$or{$field{title},$field{caption}}}</title>
        <link>$field{url}</link>
        <description>$highlight{$field{sample},$terms,&lt;strong&gt;,&lt;/strong&gt;}$if{$field{sample},...}</description>
      </item>}
    </channel>
  </rss>

Surely $field{url} needs some sort of escaping?  Also, we aren't
escaping $env{SCRIPT_NAME} or $dbname in the <channel>'s <link> tag.

Sadly, the opensearch spec doesn't really seem cover the format of
URLs in <link> tags, but the "&amp;" in the <channel>'s <link> makes
me suspect that $html{ } is the right answer.

But reading the spec, it does say that any HTML in the description tag
should be escaped once, or enclosed as CDATA:

http://opensearch.a9.com/docs/devfaq.jsp#html

Since we seem to be using RSS instead of ATOM, I think that means we
need a $html{ } around the contents of the description tag.

Cheers,
    Olly



From james-xapian at tartarus.org  Tue Jan  3 17:15:07 2006
From: james-xapian at tartarus.org (James Aylett)
Date: Tue, 3 Jan 2006 17:15:07 +0000
Subject: [Xapian-devel] opensearch template
In-Reply-To: <20060103170026.GQ22250@survex.com>
References: <20060103170026.GQ22250@survex.com>
Message-ID: <20060103171507.GJ10528@tartarus.org>

On Tue, Jan 03, 2006 at 05:00:26PM +0000, Olly Betts wrote:

> $httpheader{Content-Type,application/xml}<?xml version="1.0"?>
>   <rss version="2.0" xmlns:openSearch="http://a9.com/-/spec/opensearchrss/1.0/">
>     <channel>
>       <title>Xapian search results</title>
>       <link>$env{SCRIPT_NAME}?DB=$dbname&amp;P=$html{$query}</link>
>       <description>Search results for "$html{$query}" at xapian.org</description>
>       <language>en</language>
>       <copyright>(C)2005, Xapian.org</copyright>
>       <openSearch:totalResults>$msize</openSearch:totalResults>
>       <openSearch:startIndex>$add{topdoc,1}</openSearch:startIndex>
>       <openSearch:itemsPerPage>$hitsperpage</openSearch:itemsPerPage>
> $hitlist{
>       <item>
>         <title>$html{$or{$field{title},$field{caption}}}</title>
>         <link>$field{url}</link>
>         <description>$highlight{$field{sample},$terms,&lt;strong&gt;,&lt;/strong&gt;}$if{$field{sample},...}</description>
>       </item>}
>     </channel>
>   </rss>
> 
> Surely $field{url} needs some sort of escaping?  Also, we aren't
> escaping $env{SCRIPT_NAME} or $dbname in the <channel>'s <link> tag.

Yeah, they should all be escaped appropriately.

> Sadly, the opensearch spec doesn't really seem cover the format of
> URLs in <link> tags, but the "&amp;" in the <channel>'s <link> makes
> me suspect that $html{ } is the right answer.

There isn't any structure beyond "they are URLs". It's an XML-based
format, so omegascripts' (slightly confusingly-named) $html{ } should
do fine.

> But reading the spec, it does say that any HTML in the description tag
> should be escaped once, or enclosed as CDATA:
> 
> http://opensearch.a9.com/docs/devfaq.jsp#html
> 
> Since we seem to be using RSS instead of ATOM, I think that means we
> need a $html{ } around the contents of the description tag.

If you do, the &lt; and &gt; in the $highlight{...} need to change to
< and > respectively, or you'll have "double escaped" them. I hate the
opensearch spec, it's written as if the RSS 2.0 document were a spec.

In theory we can re-work opensearch over Atom, but you get into fiddly
areas because Atom requires GUIDs. There wasn't enough interest on the
Atom list to get anywhere with the idea, as far as I can tell (and
neither Richard nor I have had time).

The opensearch template isn't really finished - it was more a quick
hack to see how fiddly it would be.

J

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  james at tartarus.org                               uncertaintydivision.org



From olly at survex.com  Tue Jan  3 17:41:25 2006
From: olly at survex.com (Olly Betts)
Date: Tue, 3 Jan 2006 17:41:25 +0000
Subject: [Xapian-devel] opensearch template
In-Reply-To: <20060103171507.GJ10528@tartarus.org>
References: <20060103170026.GQ22250@survex.com>
	<20060103171507.GJ10528@tartarus.org>
Message-ID: <20060103174125.GS22250@survex.com>

On Tue, Jan 03, 2006 at 05:15:07PM +0000, James Aylett wrote:
> On Tue, Jan 03, 2006 at 05:00:26PM +0000, Olly Betts wrote:
> > But reading the spec, it does say that any HTML in the description tag
> > should be escaped once, or enclosed as CDATA:
> > 
> > http://opensearch.a9.com/docs/devfaq.jsp#html
> > 
> > Since we seem to be using RSS instead of ATOM, I think that means we
> > need a $html{ } around the contents of the description tag.
> 
> If you do, the &lt; and &gt; in the $highlight{...} need to change to
> < and > respectively, or you'll have "double escaped" them.

I think the double escaping is correct.

If we search for "1" and a description is:

1 < 2

Then $highlight produces (assuming we're using <b>...</b> to highlight):

<b>1</b> &lt; 2

And the opensearch response will contain:

<description>&lt;b&gt;1&lt;/b&gt; &amp;lt; 2</description>

Then the code handling the opensearch response unescapes the contents of
<description> and gets:

<b>1</b> &lt; 2

Which is correct.  If "&lt;" were "<" from $highlight, at this point
we'd have "<b>1</b> < 2" which is broken HTML.

Or am I missing something?

> In theory we can re-work opensearch over Atom, but you get into fiddly
> areas because Atom requires GUIDs. There wasn't enough interest on the
> Atom list to get anywhere with the idea, as far as I can tell (and
> neither Richard nor I have had time).

I think it's been done now - the opensearch site talks about use with
RSS or Atom anyway.

> The opensearch template isn't really finished - it was more a quick
> hack to see how fiddly it would be.

OK, but we really shouldn't be shipping stuff with missing escaping.
That's where XSS security holes come from...

Cheers,
    Olly



From james-xapian at tartarus.org  Tue Jan  3 17:48:58 2006
From: james-xapian at tartarus.org (James Aylett)
Date: Tue, 3 Jan 2006 17:48:58 +0000
Subject: [Xapian-devel] opensearch template
In-Reply-To: <20060103174125.GS22250@survex.com>
References: <20060103170026.GQ22250@survex.com>
	<20060103171507.GJ10528@tartarus.org>
	<20060103174125.GS22250@survex.com>
Message-ID: <20060103174858.GA31336@tartarus.org>

On Tue, Jan 03, 2006 at 05:41:25PM +0000, Olly Betts wrote:

> > If you do, the &lt; and &gt; in the $highlight{...} need to change to
> > < and > respectively, or you'll have "double escaped" them.
> 
> I think the double escaping is correct.

No, you're completely right. Silly me.

> > In theory we can re-work opensearch over Atom, but you get into fiddly
> > areas because Atom requires GUIDs. There wasn't enough interest on the
> > Atom list to get anywhere with the idea, as far as I can tell (and
> > neither Richard nor I have had time).
> 
> I think it's been done now - the opensearch site talks about use with
> RSS or Atom anyway.

There were issues with the simplicity of opensearch some people wanted
to address. Also trying to get a solid specification for opensearch+
rather than one which relies on the underlying definitions.

> > The opensearch template isn't really finished - it was more a quick
> > hack to see how fiddly it would be.
> 
> OK, but we really shouldn't be shipping stuff with missing escaping.

Absolutely. I don't remember deciding to ship it, which probably means
it was a mistake.

> That's where XSS security holes come from...

Well, (a) we shouldn't be building XML without at least running it
through a well-formedness check; (b) HTML shouldn't really be embedded
using escaping, it should be done by reference or in CDATA which is a
big step towards avoiding XSS-like problems because it removes the
multi-escaping issue entirely.

CDATA inside RSS causes problems for some clients, however (bah). (a)
is probably worth looking at at some point - a mechanism which allows
omega to run the generated string through an empty SAX parser to check
it isn't rubbish.

That doesn't solve the problem, however. I'm not convinced building
XML using omegascript is a good idea; we should be using a tool built
on GenX or something. (Now... a variant of omegascript that outputs
generated XML where the presence or absence of elements/data was
configurable - that would be very nice. If it could feed into an STX
processor, so much the better.)

J

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  james at tartarus.org                               uncertaintydivision.org



From jean-francois.dockes at wanadoo.fr  Fri Jan 13 06:48:10 2006
From: jean-francois.dockes at wanadoo.fr (Jean-Francois Dockes)
Date: Fri, 13 Jan 2006 07:48:10 +0100
Subject: [Xapian-devel] xapian-config --libs outputs libstdc++.la as a
	dependency
Message-ID: <17351.19754.124115.873666@mouton.dockes.com>


Hello,

I am hearing of users having trouble linking with libxapian (on slackware
and gentoo systems, and 0.9.2 I think), and I am not too sure where the
problem comes from, or what the correct solution could be, so I am just
asking here in case someone has a quick answer.

What happens is that "xapian-config --libs" outputs libstdc++.la in the
list of libraries. Something like:

-L/usr/i686-pc-linux-gnu/bin -L/usr/i686-pc-linux-gnu/lib \
 /usr/lib/gcc/i686-pc-linux-gnu/3.4.4/libstdc++.la 

As recoll is linked with g++, this generates an error (I guess it might
work if libtool was used?).

The libstdc++ data seems to come from libxapian.la ('dependency_libs'
variable).

I don't see this happening on any on my systems (FreeBSD, Solaris, Redhat
7.3, Fedora core 4), so I guess I'll have to install gentoo somewhere to
look into this, except if someone here can tell me what goes wrong and what
I could do about it.

I am pasting a libxapian.la (from a gentoo system) at the end of this message.

Regards,

J.F. Dockes


# libxapian.la - a libtool library file
# Generated by ltmain.sh - GNU libtool 1.5.18 (1.1220.2.245 2005/05/16 08:55:27)
#
# Please DO NOT delete this file!
# It is necessary for linking the library.

# The name that we can dlopen(3).
dlname='libxapian.so.8'

# Names of this library.
library_names='libxapian.so.8.0.0 libxapian.so.8 libxapian.so'

# The name of the static archive.
old_library='libxapian.a'

# Libraries that this one depends upon.
dependency_libs=' -L/usr/i686-pc-linux-gnu/bin -L/usr/i686-pc-linux-gnu/lib /usr/lib/gcc/i686-pc-linux-gnu/3.4.4/libstdc++.la'

# Version information for libxapian.
current=8
age=0
revision=0

# Is this an already installed library?
installed=yes

# Should we warn about portability when linking against -modules?
shouldnotlink=no

# Files to dlopen/dlpreopen
dlopen=''
dlpreopen=''

# Directory that this library needs to be installed in:
libdir='/usr/lib'



From olly at survex.com  Fri Jan 13 21:47:48 2006
From: olly at survex.com (Olly Betts)
Date: Fri, 13 Jan 2006 21:47:48 +0000
Subject: [Xapian-devel] xapian-config --libs outputs libstdc++.la as a
	dependency
In-Reply-To: <17351.19754.124115.873666@mouton.dockes.com>
References: <17351.19754.124115.873666@mouton.dockes.com>
Message-ID: <20060113214748.GW11445@survex.com>

On Fri, Jan 13, 2006 at 07:48:10AM +0100, Jean-Francois Dockes wrote:
> What happens is that "xapian-config --libs" outputs libstdc++.la in the
> list of libraries. Something like:
> 
> -L/usr/i686-pc-linux-gnu/bin -L/usr/i686-pc-linux-gnu/lib \
>  /usr/lib/gcc/i686-pc-linux-gnu/3.4.4/libstdc++.la 

Hmm, that's not good.

I guess we need to recursively check for .la files in dependency_libs
and substitute the actual library to link against, plus the contents of
the .la file's dependency_libs.  That's a bit fiddly though...

We might be able to get away with just dropping libstdc++.la, since
linking C++ code should pull it in anyway.  I can't see any other
.la files making it in there unless we start depending on other
libraries.

Anyway, I'll take a look.

> As recoll is linked with g++, this generates an error (I guess it might
> work if libtool was used?).

If you're using libtool, then xapian-config --ltlibs should be used
instead (which happens automatically if you use XO_LIB_XAPIAN in
configure.in/configure.ac and use AC_PROG_LIBTOOL or AM_PROG_LIBTOOL).

Cheers,
    Olly



From olly at survex.com  Sat Jan 14 02:49:00 2006
From: olly at survex.com (Olly Betts)
Date: Sat, 14 Jan 2006 02:49:00 +0000
Subject: [Xapian-devel] xapian-config --libs outputs libstdc++.la as a
	dependency
In-Reply-To: <20060113214748.GW11445@survex.com>
References: <17351.19754.124115.873666@mouton.dockes.com>
	<20060113214748.GW11445@survex.com>
Message-ID: <20060114024900.GD11445@survex.com>

On Fri, Jan 13, 2006 at 09:47:48PM +0000, Olly Betts wrote:
> On Fri, Jan 13, 2006 at 07:48:10AM +0100, Jean-Francois Dockes wrote:
> > What happens is that "xapian-config --libs" outputs libstdc++.la in the
> > list of libraries. Something like:
> > 
> > -L/usr/i686-pc-linux-gnu/bin -L/usr/i686-pc-linux-gnu/lib \
> >  /usr/lib/gcc/i686-pc-linux-gnu/3.4.4/libstdc++.la 
> 
> Hmm, that's not good.

I've attached a simple patch which should fix this in most cases.  I'll
probably go with this for the forthcoming release at least.  It'll work
so long as no .la files listed in libxapian.la's dependency_libs have
dependency_libs themselves that aren't pulled in anyway.

The patch is for xapian-config.  You'll need to patch xapian-config.in
instead if you patch before running configure (the same patch should
apply to either).

Cheers,
    Olly
-------------- next part --------------
--- xapian-config.orig
+++ xapian-config
@@ -132,6 +132,14 @@
 	test /usr/lib != "$libdir" && L="-L$libdir "
 	D=
 	eval `grep ^dependency_libs= "$libdir/libxapian.la"`
+	# Replace "/path/to/libfoo.la" with "-L/path/to -lfoo" which doesn't
+	# take into account any dependency_libs which libfoo.la may pull in
+	# but the only .la that we're likely to see is libstdc++.la and it'll
+	# hopefully work there as any dependency libs will probably be pulled
+	# in by the compiler for us.  Recursively pulling in dependency_libs
+	# is probably too hard to do in sh - we'd need to reimplement
+	# xapian-config in C or C++.
+	dependency_libs=`echo " $dependency_libs "|sed 's![     ][      ]*!  !g;s! \(/[^ ]*\)/lib\([^ ]*\).la ! -L\1 -l\2 !g;s!  ! !g;s!^ !!;s! $!!'`
 	test -n "$dependency_libs" && D=" $dependency_libs"
 	echo "$F$L-lxapian$D"
 	;;

From jean-francois.dockes at wanadoo.fr  Sat Jan 14 06:58:11 2006
From: jean-francois.dockes at wanadoo.fr (Jean-Francois Dockes)
Date: Sat, 14 Jan 2006 07:58:11 +0100
Subject: [Xapian-devel] xapian-config --libs outputs libstdc++.la as a
	dependency
In-Reply-To: <20060114024900.GD11445@survex.com>
References: <17351.19754.124115.873666@mouton.dockes.com>
	<20060113214748.GW11445@survex.com>
	<20060114024900.GD11445@survex.com>
Message-ID: <17352.41219.947435.17457@mouton.dockes.com>

Olly Betts writes:
 > On Fri, Jan 13, 2006 at 07:48:10AM +0100, Jean-Francois Dockes wrote:
 > > What happens is that "xapian-config --libs" outputs libstdc++.la in the
 > > list of libraries. Something like:
 > 
 > I've attached a simple patch which should fix this in most cases.  I'll
 > probably go with this for the forthcoming release at least.  It'll work
 > so long as no .la files listed in libxapian.la's dependency_libs have
 > dependency_libs themselves that aren't pulled in anyway.
 > ...

Whoa, this was fast. I wanted to check the gentoo xapian build process to
see if there was something special in there that triggered the condition,
but you left me no time ...

Anyway, I am going to try and check how the patch can be added to the
gentoo portage script ('ebuild').

J.F. Dockes