[jsword-devel] New versification mapping - preload mappings to prevent pauses

Chris Burrell chris at burrell.me.uk
Sat Jan 18 13:44:58 MST 2014 

I think there are two things here. The versifications mappings should
probably hard-code the use RangedPassage as it's the most appropriate data
structure for reading those. Mappings will typically be single verses &
verse ranges and we want to limit the memory consumption to the minimum as
well.

On the other side, I think DM is suggesting that by setting the default
passage type to RangedPassage you may experience general speed ups on And
Bible anyway.

Chris



On 18 January 2014 20:39, Martin Denham <mjdenham at gmail.com> wrote:

> Should I specify RangedPassage as the default passage type for And Bible
> or should it be specified in the v11n mapping code?
>
> RangedPassage seems to be missing from the PassageType enum.  Is that a
> mistake?
>
> Martin
>
>
> On 18 January 2014 20:19, Chris Burrell <chris at burrell.me.uk> wrote:
>
>> Great stuff - helps really explain things. So I think for the
>> versification mappings, seen as they don't change much as they are mostly
>> contiguous ranges a RangePassage might be the best bet.
>>
>> I think the other thing that's quite expensive
>> is: PassageKeyFactory.instance().getKey(targetVersification,
>> qualifiedKey.getKey().getOsisRef())
>>
>> I'm not sure what's the best way of dealing with this but I suspect that
>> we may want to have specific cases for Verses as I believe the above is
>> quite heavy as well.
>>
>> I think I also often
>> use: PassageKeyFactory.instance().createEmptyKeyList(versification) which
>> probably should use a RangePassage as well
>>
>> Would that make sense?
>> Chris
>>
>>
>>
>> On 18 January 2014 20:11, DM Smith <dmsmith at crosswire.org> wrote:
>>
>>> There are several different implementations for Passage. They have
>>> trade-offs.
>>> BitwisePassage is strictly a BitMap, with one bit for each verse in the
>>> v11n. It is very space heavy but fast for most operations. Especially
>>> adding or removing Verses. Iteration is fast as it uses NextSetBit (or
>>> whatever it is called to find the next set bit).
>>> RocketPassage is a very heavy implementation. Not appropriate for
>>> mobile. It occupies lots more space than a BitwisePassage
>>> PassageTally is basically a weighted set of verses such that you can
>>> order by weights. This is used to return weighted search results. It is
>>> probably not much use otherwise. It is used to return a ranked Lucene
>>> search.
>>> RangedPassage keeps ranges (start verse and length) ordered in a
>>> TreeSet. It probably is a good candidate for mobile. Maybe the best
>>> regarding storage. Works well if the RangedPassage is not modified much. If
>>> you add a verse (not in the RangedPassage) it may cause two VerseRanges to
>>> be merged into one, otherwise it'll extend one or create a new one. That
>>> computation takes time. Removing a verse is also complex.
>>> DistinctPassage keeps verses ordered in a TreeSet. This can be a good
>>> candidate for mobile, if there are not many verses in the Passage. If there
>>> are lots it gets bigger than BitwisePassage.
>>>
>>> The PassageKeyFactory is used throughout JSword to create the preferred
>>> type of Passage. You can set it to one of the above via PassageType
>>> typically at program startup.
>>>
>>> I wrote BitwisePassage; Joe wrote the others.
>>>
>>> I've been thinking that there might be another variation of
>>> BitwisePassage that might make sense. One with testament or book bitmaps.
>>>
>>> Hope this helps.
>>>
>>>  In Him,
>>> DM
>>>
>>>
>>> On Jan 18, 2014, at 2:42 PM, Chris Burrell <chris at burrell.me.uk> wrote:
>>>
>>> The reason I went for passages is for less memory consumption. I also
>>> thought it would provide better parsing and we're parsing less.
>>>
>>> But I think the way we store passages is to represent every verse by a
>>> bit. means that iterating through ranges, calculating cardinality or
>>> initialising them means is expensive.
>>>
>>> I wonder if we could have a passage that just stores boundaries rather
>>> than all its content. That would be very fast to parse and very quick for
>>> operations on it.
>>>
>>> Chris
>>>  On 18 Jan 2014 19:17, "Martin Denham" <mjdenham at gmail.com> wrote:
>>>
>>>> I think I have found a workaround for this problem (using a background
>>>> thread makes it run very quickly) but it seems to point toward a possible
>>>> point of improvement in the new JSword mapping data structure.
>>>>
>>>> In the old AB v11n mapping code I stored maps of Verses.  However the
>>>> new JSword code seems to store maps of RocketPassage/BitwisePassage.  Can
>>>> you verify this is the best data structure for what will normally be a
>>>> single verse.  I don't know an awful lot about this so I am just asking,
>>>> but maybe a Passage will always consume more memory than a Verse.
>>>>
>>>> Thanks
>>>> Martin
>>>>
>>>>
>>>>
>>>>
>>>> On 18 January 2014 13:30, Martin Denham <mjdenham at gmail.com> wrote:
>>>>
>>>>> I tried your patch and it did noticeably improve things, so it is a
>>>>> worthwhile patch, but did not solve the actual problem.  Still viewing the
>>>>> RusSynodal straight after download takes well over a minute, whereas to
>>>>> boot straight into it takes about 2 secs.
>>>>>
>>>>> One or 2 things I have noticed:
>>>>>
>>>>>    - a huge amount of memory allocation and GC occurs when trying to
>>>>>    view RusSynodal straight after download
>>>>>    - I am not great with the profiler but here are my tentative
>>>>>    observations
>>>>>       - Seems to be spending about half the time in
>>>>>       java.util.BitSet.cardinality()
>>>>>          - called by
>>>>>          BitwisePassage.countVerses()<-RocketPassage.countVerses()<-AbstractPassage.getCardinality()<-VersificationToKJVMapper.add...
>>>>>       - I tried caching AbstractPassage.getCardinality but that had
>>>>>       no affect which would be the case if there were lots of different keys
>>>>>    - Could it be that a module is in an unusual state after download?
>>>>>     I have noticed that a module cannot be deleted directly after download,
>>>>>    until the app is restarted due to:
>>>>>       - sbmd.getConfigFile()==null when called by
>>>>>       SwordBookDriver.isDeletable directly after module download
>>>>>
>>>>> Any suggestions are welcome.
>>>>>
>>>>> Cheers
>>>>> Martin
>>>>>
>>>>>
>>>>> On 16 January 2014 23:13, Martin Denham <mjdenham at gmail.com> wrote:
>>>>>
>>>>>> I think I will use the old AB v11n mapping for the next release of AB
>>>>>> and come back to this after the release which has quite a lot of changes in
>>>>>> it already anyway.
>>>>>>
>>>>>> The problem could be AB, Android, or JSword related and it is not
>>>>>> trivial to reproduce involving repeated re-installs of RusSynodal.  I had a
>>>>>> similar problem a year or two ago, it took a long time to investigate and
>>>>>> ended up being a flaw in the Android VM which required a change to JSword.
>>>>>>
>>>>>> I keep trying to think what could cause these symptoms and drawing a
>>>>>> blank.
>>>>>>
>>>>>> Cheers
>>>>>> Martin
>>>>>>
>>>>>>
>>>>>> On 16 January 2014 21:35, Chris Burrell <chris at burrell.me.uk> wrote:
>>>>>>
>>>>>>> I'll try profiling it! Nothing comes to mind though.
>>>>>>>  On 16 Jan 2014 20:04, "Martin Denham" <mjdenham at gmail.com> wrote:
>>>>>>>
>>>>>>>> There is something unusual happening with the v11n mapping creation
>>>>>>>> straight after downloading a file requiring mapping e.g. It takes over 2
>>>>>>>> mins to load the mapping (a single call to mapVerse) for RusSynodal
>>>>>>>> straight after download but a few seconds to load the file normally.
>>>>>>>>
>>>>>>>> 1. View ESV and anything else in split screen
>>>>>>>> 2. Download RusSynodal
>>>>>>>> 3. Exit download screen after download
>>>>>>>> 4. select RusSynodal for display (with ESV) in 1 half of split
>>>>>>>> screen
>>>>>>>> 5. Mapping code now takes over 2 minutes to load mapping
>>>>>>>> 6. But there are no errors and then everything seems fine
>>>>>>>>
>>>>>>>> 7. Exit and kill And Bible
>>>>>>>> 8. Restart with ESV/RST in split screens
>>>>>>>> 9. This time everything is initialised in a few seconds
>>>>>>>>
>>>>>>>> Any ideas?
>>>>>>>>
>>>>>>>> Martin
>>>>>>>>
>>>>>>>>
>>>>>>>> On 16 January 2014 19:06, Chris Burrell <chris at burrell.me.uk>wrote:
>>>>>>>>
>>>>>>>>> Hi Martin
>>>>>>>>>
>>>>>>>>> Fine by me. Think it should be opt in if possible.
>>>>>>>>>
>>>>>>>>> A couple of thoughts on that note. Do you know of it s the io or
>>>>>>>>> the cpu time that's the issue?
>>>>>>>>>
>>>>>>>>> We probably want to opt in with a list as well as all as there's
>>>>>>>>> no point in loading the v11n if not required.
>>>>>>>>>
>>>>>>>>> I guess especially if we end up with mappings per module
>>>>>>>>> eventually.
>>>>>>>>>
>>>>>>>>> On a separate note, I also found the books.installed call is very
>>>>>>>>> expensive. Thinking it may be worth partially loading these. With around
>>>>>>>>> 200 modules we spend almost 15 seconds loading them.
>>>>>>>>>
>>>>>>>>> Finally you can apply for a open source license of jprofiler which
>>>>>>>>> helps massively to work out what's going on. Have got a couple of
>>>>>>>>> uncommitted fixes for books. Installed find with that.
>>>>>>>>>
>>>>>>>>> Chris
>>>>>>>>> On 16 Jan 2014 18:10, "Martin Denham" <mjdenham at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>>  Hi,
>>>>>>>>>>
>>>>>>>>>> I have integrated the new v11n mapping code and find I am getting
>>>>>>>>>> a pause when doing the initial mapping between any 2 v11ns.
>>>>>>>>>>
>>>>>>>>>> I originally had the same problem with the AB mapping code so
>>>>>>>>>> pre-loaded all mapping required for the installed set of documents at the
>>>>>>>>>> start in a background thread to prevent delays.  The code I used is in
>>>>>>>>>> VersificationMappingFactory.initialiseRequiredMappings()<https://github.com/mjdenham/and-bible/blob/master/AndBible/src/net/bible/android/control/versification/mapping/VersificationMappingFactory.java>
>>>>>>>>>> .
>>>>>>>>>>
>>>>>>>>>> I am trying to think of the best way to do this in the new code.
>>>>>>>>>>  Either i) you could add a method to do it which could be called or ii) you
>>>>>>>>>> could proeload all required mappings automatically or iii) add a new method
>>>>>>>>>> to allow AB (or other frontend) to trigger preload of all required mappings
>>>>>>>>>> by adding a public method a bit like the new
>>>>>>>>>> VersificationsMapper.ensure(v11ntopreload).
>>>>>>>>>>
>>>>>>>>>> This is quite an issue for mobile users.  I have a fast mobile
>>>>>>>>>> and loading a mapping causes a noticeable delay the first time a verse
>>>>>>>>>> changes, but the preload fix is fairly simple and worked well.
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>> Martin
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> jsword-devel mailing list
>>>>>>>>>> jsword-devel at crosswire.org
>>>>>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> jsword-devel mailing list
>>>>>>>>> jsword-devel at crosswire.org
>>>>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>  _______________________________________________
>>> jsword-devel mailing list
>>> jsword-devel at crosswire.org
>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/jsword-devel/attachments/20140118/cdff4551/attachment-0001.html>

More information about the jsword-devel mailing list