[sword-devel] Efficiently extracting section + chapter headers

Tobias Klein contact at tklein.info
Sun May 23 15:38:57 EDT 2021 

Hi Troy,

I am compiling SWORD using CMake and I am setting CMAKE_BUILD_TYPE to "Release", which should automatically result in compiler optimization (-O3) being active (right, Greg?).

The numbers below are when using the NASB, but I guess that’s similar in markup compared to the KJV.

From a pure user perspective I am thinking … getting an outline (the list of section headers from one particular book) shouldn’t take 700ms … 
But I realize that we are essentially dealing with the complexity of a full text search. If you could have a look with your profiler I’d be glad!

I am currently implementing chapter-based loading in Ezra. So far I have always loaded books fully, but this tends to be too much data and too slow for lower performance devices (tablets!), especially with large books like Psalms or Isaiah.
To enhance the navigation experience I would like to still load the full list of section headers even though only one chapter is loaded at a time.
But if loading the section header list takes almost as much time as loading the whole book then I am not winning much here ...

Best regards,
Tobias

> Am 23.05.2021 um 21:08 schrieb Troy A. Griffitts <scribe at crosswire.org>:
> 
> Hi Tobias,
> 
> What's happening when you do an EntryAttributes search, is that each entry is pulled from storage and all tags processed to build the attributes map for that entry, then a pattern match is attempted against the attributes which match your attribute key map.  Loading up each entry and processing all tags on a module which is richly marked up like the KJV module, for example, can take a bit of time.  Compiling SWORD with optimization can greatly reduce the time.  I can also make a profiling pass on that action and see if there is anything I can clean up, but generally, for unindexed searches, we try to hit <1sec on desktops and <3sec on handhelds for a richly marked up Bible like the KJV.
> 
> Troy
> 
> On 5/23/21 5:40 PM, Tobias Klein wrote:
>> When I extract the section headers using the search function this takes ~700ms for Psalms (on my Core i7, measured on JavaScript side). I suppose it is much slower on a tablet or phone.
>> 
>> Is there any way to speed up this operation in the SWORD engine?
>> 
>> Best regards,
>> Tobias
>> 
>> On 5/23/21 2:20 PM, Tobias Klein wrote:
>>> Thanks, Troy.
>>> 
>>> The issue was that I previously hadn't used this:
>>> 
>>> mgr.setGlobalOption("Headings", "On");
>>> My new helper function now works after using that call initially.
>>> 
>>> Best regards,
>>> Tobias
>>> 
>>> On 5/22/21 11:09 PM, Troy A. Griffitts wrote:
>>>> Hi Tobias. Have a look at sword/examples/cmdline/search.cpp. I believe it takes an optional range parameter.
>>>> 
>>>> 
>>>> 
>>>> On May 22, 2021 7:49:48 PM GMT+02:00, Tobias Klein <contact at tklein.info> <mailto:contact at tklein.info> wrote:
>>>> Hi Troy,
>>>> 
>>>> I can't seem to get the scope parameter of the search function right ... 
>>>> I either get crashes or no results.
>>>> 
>>>> How do I have to initialize the scope parameter for the module search so 
>>>> that I get the headings for one particular Bible book as search results?
>>>> 
>>>> Best regards,
>>>> Tobias
>>>> 
>>>> On 4/12/21 9:07 PM, Troy A. Griffitts wrote:
>>>>  Sure Tobias,
>>>> 
>>>>  module->getEntryAttributes() will allow you to pull the headings from a
>>>>  verse, and
>>>> 
>>>>  an Entry Attributes search for '/Heading' will give you only the verses
>>>>  with Heading entries attributes.
>>>> 
>>>>  I've thrown together a quick example here, which you can use with
>>>>  something like:
>>>> 
>>>>  ./showHeadings NASB Matt-John
>>>> 
>>>>  which will give you the headings from the Gospels.
>>>> 
>>>>  https://crosswire.org/svn/sword/trunk/examples/tasks/findHeadings.cpp <https://crosswire.org/svn/sword/trunk/examples/tasks/findHeadings.cpp>
>>>> 
>>>>  Creating this example, I found I needed to fix a bug in the engine.  The
>>>>  engine Entry Attributes search lets you search for values in the entry
>>>>  attributes (e.g., a Strong's number 1234).  In your use case, when
>>>>  searching, you don't care about the value; you only care about
>>>>  presence.  This wasn't working, but simple presence can now be searched
>>>>  for with my latest commit.
>>>> 
>>>>  Hope this helps,
>>>> 
>>>>  Troy
>>>> 
>>>> 
>>>>  On 4/12/21 9:31 AM, Tobias Klein wrote:
>>>>  Hi,
>>>> 
>>>>  I have been getting a request from a user to render section headings independently of the currently opened bible translation module.
>>>> 
>>>>  So for example when opening the KJV still using the section/chapter headers of the NASB.
>>>> 
>>>>  This would be useful in those cases when the respective bible translation module does not come with section/chapter headers, but another one does.
>>>> 
>>>>  A requirement would be to efficiently extract section/chapter headers from a module using the SWORD API. As of now I only see that you can iterate over the verses of a book and individually scan each verse for the headers. Is there a more efficient way of doing that?
>>>> 
>>>>  Another use case could be to render a book outline. This is what I am already doing now in Ezra, but in a way that is not so clean (I am traversing the DOM looking for all section header elements of the current book).
>>>> 
>>>>  Best regards,
>>>>  Tobias
>>>>  sword-devel mailing list: sword-devel at crosswire.org <mailto:sword-devel at crosswire.org>
>>>>  http://crosswire.org/mailman/listinfo/sword-devel <http://crosswire.org/mailman/listinfo/sword-devel>
>>>>  Instructions to unsubscribe/change your settings at above page
>>>> sword-devel mailing list: sword-devel at crosswire.org <mailto:sword-devel at crosswire.org>
>>>> http://crosswire.org/mailman/listinfo/sword-devel <http://crosswire.org/mailman/listinfo/sword-devel>
>>>> Instructions to unsubscribe/change your settings at above page
>>>> sword-devel mailing list: sword-devel at crosswire.org <mailto:sword-devel at crosswire.org>
>>>> http://crosswire.org/mailman/listinfo/sword-devel <http://crosswire.org/mailman/listinfo/sword-devel>
>>>> Instructions to unsubscribe/change your settings at above page
>>>> 
>>>> -- 
>>>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>> 
>>> 
>>> _______________________________________________
>>> sword-devel mailing list: sword-devel at crosswire.org <mailto:sword-devel at crosswire.org>
>>> http://crosswire.org/mailman/listinfo/sword-devel <http://crosswire.org/mailman/listinfo/sword-devel>
>>> Instructions to unsubscribe/change your settings at above page
>> 
>> 
>> _______________________________________________
>> sword-devel mailing list: sword-devel at crosswire.org <mailto:sword-devel at crosswire.org>
>> http://crosswire.org/mailman/listinfo/sword-devel <http://crosswire.org/mailman/listinfo/sword-devel>
>> Instructions to unsubscribe/change your settings at above page
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://crosswire.org/pipermail/sword-devel/attachments/20210523/51ad206c/attachment.html>

More information about the sword-devel mailing list