[jsword-devel] New versification mapping - preload mappings to prevent pauses

Sat Jan 18 13:11:56 MST 2014

There are several different implementations for Passage. They have trade-offs.
BitwisePassage is strictly a BitMap, with one bit for each verse in the v11n. It is very space heavy but fast for most operations. Especially adding or removing Verses. Iteration is fast as it uses NextSetBit (or whatever it is called to find the next set bit).
RocketPassage is a very heavy implementation. Not appropriate for mobile. It occupies lots more space than a BitwisePassage
PassageTally is basically a weighted set of verses such that you can order by weights. This is used to return weighted search results. It is probably not much use otherwise. It is used to return a ranked Lucene search.
RangedPassage keeps ranges (start verse and length) ordered in a TreeSet. It probably is a good candidate for mobile. Maybe the best regarding storage. Works well if the RangedPassage is not modified much. If you add a verse (not in the RangedPassage) it may cause two VerseRanges to be merged into one, otherwise it'll extend one or create a new one. That computation takes time. Removing a verse is also complex.
DistinctPassage keeps verses ordered in a TreeSet. This can be a good candidate for mobile, if there are not many verses in the Passage. If there are lots it gets bigger than BitwisePassage.

The PassageKeyFactory is used throughout JSword to create the preferred type of Passage. You can set it to one of the above via PassageType typically at program startup.

I wrote BitwisePassage; Joe wrote the others.

I've been thinking that there might be another variation of BitwisePassage that might make sense. One with testament or book bitmaps.

Hope this helps.

In Him,
	DM

On Jan 18, 2014, at 2:42 PM, Chris Burrell <chris at burrell.me.uk> wrote:

> The reason I went for passages is for less memory consumption. I also thought it would provide better parsing and we're parsing less.
> 
> But I think the way we store passages is to represent every verse by a bit. means that iterating through ranges, calculating cardinality or initialising them means is expensive.
> 
> I wonder if we could have a passage that just stores boundaries rather than all its content. That would be very fast to parse and very quick for operations on it.
> 
> Chris
> On 18 Jan 2014 19:17, "Martin Denham" <mjdenham at gmail.com> wrote:
> I think I have found a workaround for this problem (using a background thread makes it run very quickly) but it seems to point toward a possible point of improvement in the new JSword mapping data structure.
> 
> In the old AB v11n mapping code I stored maps of Verses.  However the new JSword code seems to store maps of RocketPassage/BitwisePassage.  Can you verify this is the best data structure for what will normally be a single verse.  I don't know an awful lot about this so I am just asking, but maybe a Passage will always consume more memory than a Verse.
> 
> Thanks
> Martin
> 
> 
> 
> 
> On 18 January 2014 13:30, Martin Denham <mjdenham at gmail.com> wrote:
> I tried your patch and it did noticeably improve things, so it is a worthwhile patch, but did not solve the actual problem.  Still viewing the RusSynodal straight after download takes well over a minute, whereas to boot straight into it takes about 2 secs.
> 
> One or 2 things I have noticed:
> a huge amount of memory allocation and GC occurs when trying to view RusSynodal straight after download
> I am not great with the profiler but here are my tentative observations
> Seems to be spending about half the time in java.util.BitSet.cardinality() 
> called by BitwisePassage.countVerses()<-RocketPassage.countVerses()<-AbstractPassage.getCardinality()<-VersificationToKJVMapper.add...
> I tried caching AbstractPassage.getCardinality but that had no affect which would be the case if there were lots of different keys
> Could it be that a module is in an unusual state after download?  I have noticed that a module cannot be deleted directly after download, until the app is restarted due to:
> sbmd.getConfigFile()==null when called by SwordBookDriver.isDeletable directly after module download
> Any suggestions are welcome.
> 
> Cheers
> Martin
> 
> 
> On 16 January 2014 23:13, Martin Denham <mjdenham at gmail.com> wrote:
> I think I will use the old AB v11n mapping for the next release of AB and come back to this after the release which has quite a lot of changes in it already anyway.
> 
> The problem could be AB, Android, or JSword related and it is not trivial to reproduce involving repeated re-installs of RusSynodal.  I had a similar problem a year or two ago, it took a long time to investigate and ended up being a flaw in the Android VM which required a change to JSword.  
> 
> I keep trying to think what could cause these symptoms and drawing a blank.
> 
> Cheers
> Martin
> 
> 
> On 16 January 2014 21:35, Chris Burrell <chris at burrell.me.uk> wrote:
> I'll try profiling it! Nothing comes to mind though.
> 
> On 16 Jan 2014 20:04, "Martin Denham" <mjdenham at gmail.com> wrote:
> There is something unusual happening with the v11n mapping creation straight after downloading a file requiring mapping e.g. It takes over 2 mins to load the mapping (a single call to mapVerse) for RusSynodal straight after download but a few seconds to load the file normally.
> 
> 1. View ESV and anything else in split screen
> 2. Download RusSynodal
> 3. Exit download screen after download
> 4. select RusSynodal for display (with ESV) in 1 half of split screen
> 5. Mapping code now takes over 2 minutes to load mapping
> 6. But there are no errors and then everything seems fine 
> 
> 7. Exit and kill And Bible
> 8. Restart with ESV/RST in split screens
> 9. This time everything is initialised in a few seconds
> 
> Any ideas?
> 
> Martin
> 
> 
> On 16 January 2014 19:06, Chris Burrell <chris at burrell.me.uk> wrote:
> Hi Martin
> 
> Fine by me. Think it should be opt in if possible.
> 
> A couple of thoughts on that note. Do you know of it s the io or the cpu time that's the issue?
> 
> We probably want to opt in with a list as well as all as there's no point in loading the v11n if not required.
> 
> I guess especially if we end up with mappings per module eventually.
> 
> On a separate note, I also found the books.installed call is very expensive. Thinking it may be worth partially loading these. With around 200 modules we spend almost 15 seconds loading them. 
> 
> Finally you can apply for a open source license of jprofiler which helps massively to work out what's going on. Have got a couple of uncommitted fixes for books. Installed find with that.
> 
> Chris
> 
> On 16 Jan 2014 18:10, "Martin Denham" <mjdenham at gmail.com> wrote:
> Hi,
> 
> I have integrated the new v11n mapping code and find I am getting a pause when doing the initial mapping between any 2 v11ns.
> 
> I originally had the same problem with the AB mapping code so pre-loaded all mapping required for the installed set of documents at the start in a background thread to prevent delays.  The code I used is in VersificationMappingFactory.initialiseRequiredMappings().
> 
> I am trying to think of the best way to do this in the new code.  Either i) you could add a method to do it which could be called or ii) you could proeload all required mappings automatically or iii) add a new method to allow AB (or other frontend) to trigger preload of all required mappings by adding a public method a bit like the new VersificationsMapper.ensure(v11ntopreload).
> 
> This is quite an issue for mobile users.  I have a fast mobile and loading a mapping causes a noticeable delay the first time a verse changes, but the preload fix is fairly simple and worked well.
> 
> Cheers
> Martin
> 
> 
> _______________________________________________
> jsword-devel mailing list
> jsword-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/jsword-devel
> 
> 
> _______________________________________________
> jsword-devel mailing list
> jsword-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/jsword-devel
> 
> 
> 
> 
> 
> _______________________________________________
> jsword-devel mailing list
> jsword-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/jsword-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/jsword-devel/attachments/20140118/b4b0b658/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4145 bytes
Desc: not available
URL: <http://www.crosswire.org/pipermail/jsword-devel/attachments/20140118/b4b0b658/attachment-0001.p7s>