[jsword-devel] New versification mapping - preload mappings to prevent pauses

Chris Burrell chris at burrell.me.uk
Sat Jan 18 12:42:40 MST 2014 

The reason I went for passages is for less memory consumption. I also
thought it would provide better parsing and we're parsing less.

But I think the way we store passages is to represent every verse by a bit.
means that iterating through ranges, calculating cardinality or
initialising them means is expensive.

I wonder if we could have a passage that just stores boundaries rather than
all its content. That would be very fast to parse and very quick for
operations on it.

Chris
 On 18 Jan 2014 19:17, "Martin Denham" <mjdenham at gmail.com> wrote:

> I think I have found a workaround for this problem (using a background
> thread makes it run very quickly) but it seems to point toward a possible
> point of improvement in the new JSword mapping data structure.
>
> In the old AB v11n mapping code I stored maps of Verses.  However the new
> JSword code seems to store maps of RocketPassage/BitwisePassage.  Can you
> verify this is the best data structure for what will normally be a single
> verse.  I don't know an awful lot about this so I am just asking, but maybe
> a Passage will always consume more memory than a Verse.
>
> Thanks
> Martin
>
>
>
>
> On 18 January 2014 13:30, Martin Denham <mjdenham at gmail.com> wrote:
>
>> I tried your patch and it did noticeably improve things, so it is a
>> worthwhile patch, but did not solve the actual problem.  Still viewing the
>> RusSynodal straight after download takes well over a minute, whereas to
>> boot straight into it takes about 2 secs.
>>
>> One or 2 things I have noticed:
>>
>>    - a huge amount of memory allocation and GC occurs when trying to
>>    view RusSynodal straight after download
>>    - I am not great with the profiler but here are my tentative
>>    observations
>>       - Seems to be spending about half the time in
>>       java.util.BitSet.cardinality()
>>          - called by
>>          BitwisePassage.countVerses()<-RocketPassage.countVerses()<-AbstractPassage.getCardinality()<-VersificationToKJVMapper.add...
>>       - I tried caching AbstractPassage.getCardinality but that had no
>>       affect which would be the case if there were lots of different keys
>>    - Could it be that a module is in an unusual state after download?  I
>>    have noticed that a module cannot be deleted directly after download, until
>>    the app is restarted due to:
>>       - sbmd.getConfigFile()==null when called by
>>       SwordBookDriver.isDeletable directly after module download
>>
>> Any suggestions are welcome.
>>
>> Cheers
>> Martin
>>
>>
>> On 16 January 2014 23:13, Martin Denham <mjdenham at gmail.com> wrote:
>>
>>> I think I will use the old AB v11n mapping for the next release of AB
>>> and come back to this after the release which has quite a lot of changes in
>>> it already anyway.
>>>
>>> The problem could be AB, Android, or JSword related and it is not
>>> trivial to reproduce involving repeated re-installs of RusSynodal.  I had a
>>> similar problem a year or two ago, it took a long time to investigate and
>>> ended up being a flaw in the Android VM which required a change to JSword.
>>>
>>> I keep trying to think what could cause these symptoms and drawing a
>>> blank.
>>>
>>> Cheers
>>> Martin
>>>
>>>
>>> On 16 January 2014 21:35, Chris Burrell <chris at burrell.me.uk> wrote:
>>>
>>>> I'll try profiling it! Nothing comes to mind though.
>>>>  On 16 Jan 2014 20:04, "Martin Denham" <mjdenham at gmail.com> wrote:
>>>>
>>>>> There is something unusual happening with the v11n mapping creation
>>>>> straight after downloading a file requiring mapping e.g. It takes over 2
>>>>> mins to load the mapping (a single call to mapVerse) for RusSynodal
>>>>> straight after download but a few seconds to load the file normally.
>>>>>
>>>>> 1. View ESV and anything else in split screen
>>>>> 2. Download RusSynodal
>>>>> 3. Exit download screen after download
>>>>> 4. select RusSynodal for display (with ESV) in 1 half of split screen
>>>>> 5. Mapping code now takes over 2 minutes to load mapping
>>>>> 6. But there are no errors and then everything seems fine
>>>>>
>>>>> 7. Exit and kill And Bible
>>>>> 8. Restart with ESV/RST in split screens
>>>>> 9. This time everything is initialised in a few seconds
>>>>>
>>>>> Any ideas?
>>>>>
>>>>> Martin
>>>>>
>>>>>
>>>>> On 16 January 2014 19:06, Chris Burrell <chris at burrell.me.uk> wrote:
>>>>>
>>>>>> Hi Martin
>>>>>>
>>>>>> Fine by me. Think it should be opt in if possible.
>>>>>>
>>>>>> A couple of thoughts on that note. Do you know of it s the io or the
>>>>>> cpu time that's the issue?
>>>>>>
>>>>>> We probably want to opt in with a list as well as all as there's no
>>>>>> point in loading the v11n if not required.
>>>>>>
>>>>>> I guess especially if we end up with mappings per module eventually.
>>>>>>
>>>>>> On a separate note, I also found the books.installed call is very
>>>>>> expensive. Thinking it may be worth partially loading these. With around
>>>>>> 200 modules we spend almost 15 seconds loading them.
>>>>>>
>>>>>> Finally you can apply for a open source license of jprofiler which
>>>>>> helps massively to work out what's going on. Have got a couple of
>>>>>> uncommitted fixes for books. Installed find with that.
>>>>>>
>>>>>> Chris
>>>>>> On 16 Jan 2014 18:10, "Martin Denham" <mjdenham at gmail.com> wrote:
>>>>>>
>>>>>>>  Hi,
>>>>>>>
>>>>>>> I have integrated the new v11n mapping code and find I am getting a
>>>>>>> pause when doing the initial mapping between any 2 v11ns.
>>>>>>>
>>>>>>> I originally had the same problem with the AB mapping code so
>>>>>>> pre-loaded all mapping required for the installed set of documents at the
>>>>>>> start in a background thread to prevent delays.  The code I used is in
>>>>>>> VersificationMappingFactory.initialiseRequiredMappings()<https://github.com/mjdenham/and-bible/blob/master/AndBible/src/net/bible/android/control/versification/mapping/VersificationMappingFactory.java>
>>>>>>> .
>>>>>>>
>>>>>>> I am trying to think of the best way to do this in the new code.
>>>>>>>  Either i) you could add a method to do it which could be called or ii) you
>>>>>>> could proeload all required mappings automatically or iii) add a new method
>>>>>>> to allow AB (or other frontend) to trigger preload of all required mappings
>>>>>>> by adding a public method a bit like the new
>>>>>>> VersificationsMapper.ensure(v11ntopreload).
>>>>>>>
>>>>>>> This is quite an issue for mobile users.  I have a fast mobile and
>>>>>>> loading a mapping causes a noticeable delay the first time a verse changes,
>>>>>>> but the preload fix is fairly simple and worked well.
>>>>>>>
>>>>>>> Cheers
>>>>>>> Martin
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> jsword-devel mailing list
>>>>>>> jsword-devel at crosswire.org
>>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> jsword-devel mailing list
>>>>>> jsword-devel at crosswire.org
>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>
>>>>>>
>>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/jsword-devel/attachments/20140118/c6b80434/attachment-0001.html>

More information about the jsword-devel mailing list