<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
<br>
Joe Walker wrote:
<blockquote cite="mid5dd4742605021508595e10f5cb@mail.gmail.com"
type="cite">
<pre wrap="">Hi,
On Sun, 13 Feb 2005 22:49:10 -0500, DM Smith <a class="moz-txt-link-rfc2396E" href="mailto:dmsmith555@yahoo.com"><dmsmith555@yahoo.com></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">I have been working with Troy to see if JSword and Sword can share indexes.
So far it looks very promising. One thing we had to agree upon was the
naming of the fields.
Ours are "name" and "body" for the verse reference and the verse text
respectively.
Sword's are "key" and "content". (They also have "strong")
Unless, there are any objections, I will be changing ours to match Sword's.
</pre>
</blockquote>
<pre wrap=""><!---->
That's fine. We will all need to delete all our search indexes, but
that isn't a problem for me. Lets make sure we break the old indexes
cleanly.
</pre>
</blockquote>
<br>
I have checked it in. While it is a clean break, it is a silent one. I
guess a better one would be to come up<br>
with a different pathing.<br>
<blockquote cite="mid5dd4742605021508595e10f5cb@mail.gmail.com"
type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">While we are not in a position to generate an index with Strong's
numbers, we can use one just fine.
Should we suggest that notes and footnotes be indexed as well? Anything
else? Morphology?
Transliterations of Hebrew and Greek? Transliteration of other
languages? Unaccenting of accented text?
Most of these are in the bug database for post 1.0.
</pre>
</blockquote>
<pre wrap=""><!---->
It would be good to keep original language indexes separate from
specific translation indexes. There is no good reason why I shouldn't
be able to see verses with some strongs number in "The Message" even
though the chances of getting a strongs tagged Message translation are
close to zero.
</pre>
</blockquote>
Funny thing, I sent Troy an email today requesting the construction of
a bible that contains just<br>
Strong numbers.<br>
<blockquote cite="mid5dd4742605021508595e10f5cb@mail.gmail.com"
type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">The other thing we had to agree upon was which analyzer to use. We are
both using the same one now.
As long as we can agree on the names of the fields, the analyzer and the
values used for the keys
we can most likely use the same indexes.
Sword is storing their index in a "lucene" subdirectory of the directory
containing the module text.
For example, modules/.../ztext/kjv/lucene.
Should we do the same? I think the answer is dependent upon whether we
think that an index
should exist apart from a module. When we add the "Delete" button back
in the installer, it will
delete the entire module directory (e.g. kjv under ztext). If lucene is
a subdirectory, then that
will be deleted as well.
</pre>
</blockquote>
<pre wrap=""><!---->
This is a big question. Issues include:
- What if Sword want to update their indexes and we can't or the other
way around.
</pre>
</blockquote>
>From my experiments w/ Troy's help, it seems that this is not as big an
issue. As long as we use<br>
the same analyzer, the same "Field" names we probably will be able to
share indexes. The problem<br>
will arise when we or Sword goes to other Analyzers for the sake of
non-English texts. If the<br>
analyzer is not common to both C and Java, then we could not share the
index.<br>
<blockquote cite="mid5dd4742605021508595e10f5cb@mail.gmail.com"
type="cite">
<pre wrap="">- What if indexing becomes customizable so that users can select the
amount of info indexed to trade of speed against search depth. Then
indexes are fairly personal rather than book specific.
</pre>
</blockquote>
Indexing Strong numbers added a half meg to the index size. Compressed
that will be even less.<br>
Having Strong numbers in the index does not require that they be used.<br>
<br>
Turns out that having them in the index does not require any program
changes, if the user can<br>
supply a lucene search request to the search engine.<br>
<br>
This is just an example, of course. It could have been footnotes,
transliterations or other things.<br>
>From what I can tell, the presence of additional Fields in the index
does not affect search performance.<br>
The only things that it seems to affect are download and local disk.<br>
<br>
I do think it makes sense for an index that is constructed on the local
machine to be crafted to<br>
the user's needs, but when one is downloaded, it can be more robust
than their needs.<br>
<br>
<blockquote cite="mid5dd4742605021508595e10f5cb@mail.gmail.com"
type="cite">
<pre wrap="">- Are Sword allowing index downloading? They were against it before,
but it sounds like they are for it now. It would make sense to make
use of indexes that exist already.
</pre>
</blockquote>
I think that Troy has had a change of heart, especially since we can
share them. Nothing definite yet.<br>
<br>
It has dawned on me that the indexes are valuable in and of themselves.<br>
<br>
It would take just a few lines of code to write a program that would
return a list of verses returned<br>
from a search against a lucene index. The only thing that the
programmer would need to know<br>
is the analyzer to use and the Field names.<br>
<br>
I think that this is sufficient reason to provide the indexes for
download. Apart from the module it<br>
indexes.<br>
<blockquote cite="mid5dd4742605021508595e10f5cb@mail.gmail.com"
type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">================================================================
I noticed that JSword has its own search syntax and is doing a word at a
time search and then
merging the answer sets into a result.
From looking at the lucene search syntax, it seems much richer than
what is present in JSword.
Would it make sense to add a pass through mode? For example, if the
search begins with a
particular prefix, say "lucene:" then the rest of the line would be
passed untouched.
Also, if the search did not begin with "lucene:", would it make sense to
translate it to an equivalent
lucene search and let lucene do it's magic?
</pre>
</blockquote>
<pre wrap=""><!---->
I don't have strong feelings about this, however when we (I?) last
considered the issue I felt that things like strongs number searching,
verse blurring, etc. were fairly Bible specific and valuable. The
searcher interface could be implemented to use lucene I think, but the
advanced search dialog probably depends on the search syntax.
</pre>
</blockquote>
<br>
I am thinking of adding a passthrough mechanism, but not otherwise
touching the existing parsing<br>
and searching. We could use it to evaluate whether we want to go with
one or the other.<br>
<br>
I do remember the discussion and it was "we". The one thing that I have
not found in lucene is to<br>
do adjacency searches (blurring).<br>
<blockquote cite="mid5dd4742605021508595e10f5cb@mail.gmail.com"
type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">================================================================
I am in the process of refactoring the installer out of
org.crosswire.bibledesktop.book and
into a package of its own: org.crosswire.bibledesktop.book.install.
I came up with this name to parallel the
org.crosswire.jsword.book.install package name.
</pre>
</blockquote>
<pre wrap=""><!---->
Eh? it is there already!
</pre>
</blockquote>
<br>
Sorry about that. You usually respond right away if you have concerns.<br>
<blockquote cite="mid5dd4742605021508595e10f5cb@mail.gmail.com"
type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">================================================================
I will be working on various installer issues, which I have assigned to
myself in the bug database.
After that, I will be doing a thorough testing of the application to see
what bugs we have introduced
since 0.9.8. I have noticed some refresh problems related to displaying
passages.
</pre>
</blockquote>
<pre wrap=""><!---->
Thanks,
Joe.
_______________________________________________
jsword-devel mailing list
<a class="moz-txt-link-abbreviated" href="mailto:jsword-devel@crosswire.org">jsword-devel@crosswire.org</a>
<a class="moz-txt-link-freetext" href="http://www.crosswire.org/mailman/listinfo/jsword-devel">http://www.crosswire.org/mailman/listinfo/jsword-devel</a>
</pre>
</blockquote>
</body>
</html>