<html><head><base href="x-msg://260/"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Feb 25, 2010, at 8:51 PM, <a href="mailto:trent.jsword@trentonadams.ca">trent.jsword@trentonadams.ca</a> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div><div style="font-family: Courier; font-size: 12pt; color: rgb(0, 0, 0); ">This is not the fastest way I'm sure, but it did take 13 seconds. I don't know much about jsword yet, but I'm going to see if I can load entire books at a time, and go through them verse by verse. Bulk loading would likely be much faster.<br></div></div></span></blockquote><div><br></div>The problem with bulk loading is the assumption that SWORD has stored the verse in order. This is generally true, but not guaranteed.</div><div><br></div><div>We thought about giving the "engine" a passage to get and have it return the whole passage, but we haven't done that. JSword does have an optimization that when a block of compressed data is read, it is cached in core. If the next reference is for the same block, it is read from cache. This is the nature of compressed modules.</div><div><br></div><div>The shortcoming with the following is the setup to get the verse. It would be better to iterate over a bitmap holding a reference to the whole Bible.</div><div><br></div><div>See BookExporter for an example of iterating over the entire KJV efficiently:</div><div><a href="http://www.crosswire.org/jsword/java2html/org/crosswire/jsword/bridge/BookExporter.java.html">http://www.crosswire.org/jsword/java2html/org/crosswire/jsword/bridge/BookExporter.java.html</a></div><div><br></div><div>The other problem with your example is that each call to matches recompile the pattern. Hoist the pattern outside of the loop and it should be even faster.</div><div><br></div><div>A minor nit. OSISUtil.getPlainText(data.getOsisFragment()) will get titles and other non-canonical text. You probably want OSISUtil.getCanonicalText(data.getOsisFragment()); (Note, this will get canonical Psalm titles.)</div><div><br></div><div>Putting it together it would be something like:</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span> Book book = Books.installed().getBook("KJV");</div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "> Key keys = <span style="color: #2600c9">book</span>.getGlobalKeyList();</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; min-height: 15px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "> Iterator iter = keys.iterator();</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "> StringBuffer buf = <span style="color: #8e0667">new</span> StringBuffer();</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; color: rgb(74, 0, 251); "><span style="color: #000000"><span class="Apple-tab-span" style="white-space:pre"> </span>Pattern </span><span style="color: #2600c9">searchPattern</span><span style="color: #000000"> = Pattern.compile(</span>"wash.*word"<span style="color: #000000">);</span></div></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "> <span style="color: #8e0667">while</span> (iter.hasNext()) {</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "> Key key = (Key) iter.next();</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "> BookData data = <span style="color: #8e0667">new</span> BookData(book, key);</div><div><br></div></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "> String plainText = <font class="Apple-style-span" color="#2600C9">OSISUtil.getCanonicalText(data.getOsisFragment())</font>;</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "> <span style="color: #8e0667">if</span> (plainText != <span style="color: #8e0667">null</span> && plainText.trim().length() > 0) {</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "> Matcher m = <span style="color: #2600c9">searchPattern</span>.matcher(plainText);</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "> if (m.find()) {</div></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "> buf.delete(0, buf.length());</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "> buf.append(key).append(<span style="color: #4a00fb">':'</span>).append(plainText); <span style="color: #559173">//$NON-NLS-1$</span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "> System.<span style="color: #2600c9">out</span>.println(buf.toString());</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "> }</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "> }</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Monaco; "> }</div><div><font class="Apple-style-span" face="Monaco" size="3"><span class="Apple-style-span" style="font-size: 11px; "><br></span></font></div><div><font class="Apple-style-span" face="Monaco" size="3"><span class="Apple-style-span" style="font-size: 11px;">Note this should work for all modules. Not just Bibles.</span></font></div></div><div><br></div><div>In Him,</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>DM</div><div><br><blockquote type="cite"><div><div style="font-family: Courier; font-size: 12pt; color: rgb(0, 0, 0); "><br> Book book = Books.installed().getBook("KJV");<br> if (book == null)<br> {<br> System.exit(1);<br> }<br><br> long before;<br> long after;<br><br> before = System.currentTimeMillis();<br> for (int bookIndex = 1; bookIndex <= 66; bookIndex++)<br> { // all books<br> int maxChapters = BibleInfo.chaptersInBook(bookIndex);<br> for (int chapter = 1; chapter < maxChapters; chapter++)<br> {<br> int maxVerses = BibleInfo.versesInChapter(bookIndex, chapter);<br> for (int verse = 1; verse < maxVerses; verse++)<br> {<br> String bookName = BibleInfo.getBookName(<br> bookIndex).getLongName();<br> Passage versePassage = (Passage) book.getKey(bookName + " " + chapter + ":" + verse);<br> BookData data = new BookData(book, versePassage);<br> String plainText = OSISUtil.getPlainText(data.getOsisFragment());<br> if (plainText.matches(".*wash.*word.*"))<br> {<br> System.out.println(plainText);<br> }<br><br> }<br> }<br>// System.out.println(BibleInfo.getBookName(bookIndex).getLongName());<br> }<br> after = System.currentTimeMillis();<br><br><br>John 13:12So after he had washed their feet, and had taken his garments, and was set down again, he said unto them, Know ye what I have done to you?tag 17 - has no greek word tag 12 "and was set down"<br>Ephesians 5:26That he might sanctify and cleanse it with the washing of water by the word,<br>Revelation of John 7:14And I said unto him, Sir, thou knowest. And he said to me, These are they which came out of great tribulation, and have washed their robes, and made them white in the blood of the Lamb.NOTE! Extra word: "stolei"<br>search took: 13280ms<br><br>----- "trent jsword" <<a href="mailto:trent.jsword@trentonadams.ca">trent.jsword@trentonadams.ca</a>> wrote:<span class="Apple-converted-space"> </span><br>> From: "trent jsword" <<a href="mailto:trent.jsword@trentonadams.ca">trent.jsword@trentonadams.ca</a>><br>> To: "J-Sword Developers Mailing List" <<a href="mailto:jsword-devel@crosswire.org">jsword-devel@crosswire.org</a>><br>> Sent: Thursday, February 25, 2010 6:10:41 PM GMT -06:00 US/Canada Central<br>> Subject: Re: [jsword-devel] regex searching<br>><br>><div style="font-family: Courier; font-size: 12pt; color: rgb(0, 0, 0); ">> Well, I don't know how fast loading verses is in jsword, but my app takes .18 seconds, or possibly a little more depending on what you search for, for searching the entire bible. The bible text is on one verse per line, so there's no need to load one verse at a time. An example of a slow search, which is rare, would be "the.*or", which takes 18.32 seconds because it loads almost the entire bible over the internet, being that LARGE portions of scripture are found, lol)<br>><span class="Apple-converted-space"> </span><br>> Try "blood.*sacrifice" on<span class="Apple-converted-space"> </span><a href="http://www.trentonadams.ca/biblesearch">http://www.trentonadams.ca/biblesearch</a><span class="Apple-converted-space"> </span>for an example of a relatively fast search.<br>><span class="Apple-converted-space"> </span><br>> I can't imagine it being a "huge" amount slower with jsword, unless the file types that the data is stored in are particularly slow loading verses. But, jsword seems snappy enough, so I doubt it would be a problem.<br>><span class="Apple-converted-space"> </span><br>> But, if regex was implemented, you could allow the person to specify "within X verses of each other". i.e. "wash" would be in v3 and "word" would be in v4 But it would probably require strange verse concatenations for one attempt, no concatenations for another attempt, shifting one verse at a time through the whole bible, etc, unless you wrote a regular expression parser of your own, that could find certain parts in one verse, and other parts in another. But that gets really complicated. So, on first thought, finding regular expressions between verses would be VERY time consuming. But that's just off the top of my head, so who knows, you might be able to do something fast. ;)<br>><span class="Apple-converted-space"> </span><br>> ----- "Sijo Cherian" <<a href="mailto:sijo.cherian@gmail.com">sijo.cherian@gmail.com</a>> wrote:<span class="Apple-converted-space"> </span><br>> > From: "Sijo Cherian" <<a href="mailto:sijo.cherian@gmail.com">sijo.cherian@gmail.com</a>><br>> > To: "J-Sword Developers Mailing List" <<a href="mailto:jsword-devel@crosswire.org">jsword-devel@crosswire.org</a>><br>> > Sent: Thursday, February 25, 2010 5:46:17 PM GMT -06:00 US/Canada Central<br>> > Subject: Re: [jsword-devel] regex searching<br>> ><br>> > What about finding terms across verse boudaries? As far as I remember the index documents are on per verse basis. So there is no way lucene query can find nearby words in a paragraph (across verse boundaries). Regex can help if we string-together verses or use Passage, but like DM mentioned, it will be slow to run regex (regex can be compiled once, but it has to run on so many verse strings).<div><br>> ></div><div>BTW Is anyone working on getting search result highlighting? What needs to done besides using lucene-highlighter during the search?</div><div><br>> ></div><div>-Sijo</div><div><br>> ><div class="gmail_quote">> > On Wed, Feb 24, 2010 at 11:05 PM, DM Smith<span class="Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:dmsmith@crosswire.org" target="_blank">dmsmith@crosswire.org</a>></span><span class="Apple-converted-space"> </span>wrote:<br>> ><blockquote class="gmail_quote" style="border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); margin-top: 0pt; margin-right: 0pt; margin-bottom: 0pt; margin-left: 0.8ex; padding-left: 1ex; ">JSword search indexes support the full search syntax of Lucene. This has some support for regular expressions. Specifically, it allows for * to mean zero or more characters. I think we've got the Lucene flag to allow prefix wild cards. Note that this is more like what a shell uses: it is not a modifier for the previous character.<br>> ><span class="Apple-converted-space"> </span><br>> > It was correctly noted that wash.*word will not search the verse as a whole. Lucene search is based on words. The correct pattern in Lucene would be wash*word, as '.' does not mean any character but rather it means a punctuation mark. So "wash.*word" would be split into "wash *word" and would find all verses with the word "wash" and any word ending with "word", such as "sword".<br>> ><span class="Apple-converted-space"> </span><br>> > The default connector for JSword is OR. It would be good to add the ability for a user to change it to AND. With that, "wash* word" would produce the expected results as it would be interpreted as "wash* AND word" instead of "wash* OR word". (Choice would be some thing like "Search for ALL words instead of ANY words.") We made OR the default because it more closely matched various familiar search engines, such as Google, and because it is the default of Lucene.<br>> ><span class="Apple-converted-space"> </span><br>> > Lucene does have a regular expression capability, but it is not part of JSword. It would be a good addition. Still, it would be based on words and not on the text of a verse.<br>> ><span class="Apple-converted-space"> </span><br>> > Adding the ability to search an arbitrary regular expression would be a good addition. I don't think it would be too hard to add it. Jsword already has the interface for any search implementation. Java's Regex is a variant of Perl's but has a bit more power. There are some issues: We'll be adding highlighting to Lucene's search. Adding that to Regex would be a separate effort. The regex search would be mutually exclusive from Lucene search, so that would need to be made obvious. (In The SWORD Project for Windows, they have both it is a bit confusing as it is not clear.)<br>> ><span class="Apple-converted-space"> </span><br>> > And yes, it will need to go throw the "plain" text of each verse. It will be about as slow as creating an index. Basically, it will need to take the raw verse and strip the markup. (This is part of JSword already and Lucene indexing uses it.) And perhaps strip out the punctuation. Finally doing the search.<br>> ><span class="Apple-converted-space"> </span><br>> > If you want, add an issue or two to Jira (<a href="http://www.crosswire.org/bugs" target="_blank">www.crosswire.org/bugs</a>) under JSword. That way it won't be forgotten.<br>> ><span class="Apple-converted-space"> </span><br>> > In Christ,<br>> ><span class="Apple-converted-space"> </span><font color="#888888"> DM</font><div><div></div><div class="h5">> ><span class="Apple-converted-space"> </span><br>> ><span class="Apple-converted-space"> </span><br>> ><span class="Apple-converted-space"> </span><br>> > On 02/23/2010 08:33 PM, Trenton D. Adams wrote:<br>> ><blockquote class="gmail_quote" style="border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); margin-top: 0pt; margin-right: 0pt; margin-bottom: 0pt; margin-left: 0.8ex; padding-left: 1ex; ">But then again, I wonder if it's even needed, who knows.<br>> ><span class="Apple-converted-space"> </span><br>> > ----- "Trenton D. Adams"<<a href="mailto:trent.jsword@trentonadams.ca" target="_blank">trent.jsword@trentonadams.ca</a>> wrote:<br>> ><span class="Apple-converted-space"> </span><br>> > <span class="Apple-converted-space"> </span><br>> ><blockquote class="gmail_quote" style="border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); margin-top: 0pt; margin-right: 0pt; margin-bottom: 0pt; margin-left: 0.8ex; padding-left: 1ex; ">From: "Trenton D. Adams"<<a href="mailto:trent.jsword@trentonadams.ca" target="_blank">trent.jsword@trentonadams.ca</a>><br>> > To: "J-Sword Developers Mailing"<<a href="mailto:jsword-devel@crosswire.org" target="_blank">jsword-devel@crosswire.org</a>><br>> > Sent: Tuesday, February 23, 2010 7:29:08 PM GMT -06:00 US/Canada Central<br>> > Subject: [jsword-devel] regex searching<br>> ><span class="Apple-converted-space"> </span><br>> > Hello,<br>> ><span class="Apple-converted-space"> </span><br>> > I'm getting the impression that regular expression searching is not at<br>> > all possible without implementing something that loads the books<br>> > itself, and goes through each verse of the bible. Is this true?<br>> ><span class="Apple-converted-space"> </span><br>> > It seems like the Lucene Index just doesn't support regular expression<br>> > searches, eh? And it also seems like SearcherFactory is currently not<br>> > finished being implemented as a SearcherFactory, correct? Or, perhaps<br>> > other methods need to be added, like createSearcher(Book, Class<br>> > searcherClass)??? Then you could provide another searcher type.<br>> ><span class="Apple-converted-space"> </span><br>> > What would be appropriate for this? adding the new<br>> > SearcherFactory.createSearcher(), adding another "find" method to<br>> > AbstractBook/Book that is something like "find(SearchRequest,<br>> > Searcher)"???<br>> ><span class="Apple-converted-space"> </span><br>> > Are the books and everything abstract enough that I can search them by<br>> > loading the text? Or would I have to restrict it to the book types I<br>> > know how to parse?<br>> ><span class="Apple-converted-space"> </span><br>> > Anything else?<br>> ><span class="Apple-converted-space"> </span><br>> > Thanks.<br>> ><span class="Apple-converted-space"> </span><br>> > _______________________________________________<br>> > jsword-devel mailing list<br>> ><span class="Apple-converted-space"> </span><a href="mailto:jsword-devel@crosswire.org" target="_blank">jsword-devel@crosswire.org</a><br>> ><span class="Apple-converted-space"> </span><a href="http://www.crosswire.org/mailman/listinfo/jsword-devel" target="_blank">http://www.crosswire.org/mailman/listinfo/jsword-devel</a><br>> > <span class="Apple-converted-space"> </span><br>> ></blockquote>_______________________________________________<br>> > jsword-devel mailing list<br>> ><span class="Apple-converted-space"> </span><a href="mailto:jsword-devel@crosswire.org" target="_blank">jsword-devel@crosswire.org</a><br>> ><span class="Apple-converted-space"> </span><a href="http://www.crosswire.org/mailman/listinfo/jsword-devel" target="_blank">http://www.crosswire.org/mailman/listinfo/jsword-devel</a><br>> > <span class="Apple-converted-space"> </span><br>> ></blockquote><br>> ><span class="Apple-converted-space"> </span><br>> > _______________________________________________<br>> > jsword-devel mailing list<br>> ><span class="Apple-converted-space"> </span><a href="mailto:jsword-devel@crosswire.org" target="_blank">jsword-devel@crosswire.org</a><br>> ><span class="Apple-converted-space"> </span><a href="http://www.crosswire.org/mailman/listinfo/jsword-devel" target="_blank">http://www.crosswire.org/mailman/listinfo/jsword-devel</a><br>> ></div></div></blockquote></div><br>> ><span class="Apple-converted-space"> </span><br clear="all"><br>> > --<span class="Apple-converted-space"> </span><br>> > Regards,<br>> > Sijo<br>> ></div><br>> > _______________________________________________ jsword-devel mailing list<span class="Apple-converted-space"> </span><a href="mailto:jsword-devel@crosswire.org">jsword-devel@crosswire.org</a><span class="Apple-converted-space"> </span><a href="http://www.crosswire.org/mailman/listinfo/jsword-devel">http://www.crosswire.org/mailman/listinfo/jsword-devel</a></div><br>> _______________________________________________ jsword-devel mailing list<span class="Apple-converted-space"> </span><a href="mailto:jsword-devel@crosswire.org">jsword-devel@crosswire.org</a><a href="http://www.crosswire.org/mailman/listinfo/jsword-devel">http://www.crosswire.org/mailman/listinfo/jsword-devel</a></div>_______________________________________________<br>jsword-devel mailing list<br><a href="mailto:jsword-devel@crosswire.org">jsword-devel@crosswire.org</a><br><a href="http://www.crosswire.org/mailman/listinfo/jsword-devel">http://www.crosswire.org/mailman/listinfo/jsword-devel</a><br></div></blockquote></div><br></body></html>