<div dir="ltr">Should I specify RangedPassage as the default passage type for And Bible or should it be specified in the v11n mapping code?<div><br></div><div>RangedPassage seems to be missing from the PassageType enum. Is that a mistake?</div>
<div><br></div><div>Martin</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 18 January 2014 20:19, Chris Burrell <span dir="ltr"><<a href="mailto:chris@burrell.me.uk" target="_blank">chris@burrell.me.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Great stuff - helps really explain things. So I think for the versification mappings, seen as they don't change much as they are mostly contiguous ranges a RangePassage might be the best bet.<div>
<br></div>
<div>I think the other thing that's quite expensive is: PassageKeyFactory.instance().getKey(targetVersification, qualifiedKey.getKey().getOsisRef())</div><div><br></div><div>I'm not sure what's the best way of dealing with this but I suspect that we may want to have specific cases for Verses as I believe the above is quite heavy as well.</div>
<div><br></div><div>I think I also often use: PassageKeyFactory.instance().createEmptyKeyList(versification) which probably should use a RangePassage as well</div><div><br></div><div>Would that make sense?</div><span class="HOEnZb"><font color="#888888"><div>
Chris</div>
<div><br></div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On 18 January 2014 20:11, DM Smith <span dir="ltr"><<a href="mailto:dmsmith@crosswire.org" target="_blank">dmsmith@crosswire.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">There are several different implementations for Passage. They have trade-offs.<div>BitwisePassage is strictly a BitMap, with one bit for each verse in the v11n. It is very space heavy but fast for most operations. Especially adding or removing Verses. Iteration is fast as it uses NextSetBit (or whatever it is called to find the next set bit).</div>
<div>RocketPassage is a very heavy implementation. Not appropriate for mobile. It occupies lots more space than a BitwisePassage</div><div>PassageTally is basically a weighted set of verses such that you can order by weights. This is used to return weighted search results. It is probably not much use otherwise. It is used to return a ranked Lucene search.</div>
<div>RangedPassage keeps ranges (start verse and length) ordered in a TreeSet. It probably is a good candidate for mobile. Maybe the best regarding storage. Works well if the RangedPassage is not modified much. If you add a verse (not in the RangedPassage) it may cause two VerseRanges to be merged into one, otherwise it'll extend one or create a new one. That computation takes time. Removing a verse is also complex.</div>
<div>DistinctPassage keeps verses ordered in a TreeSet. This can be a good candidate for mobile, if there are not many verses in the Passage. If there are lots it gets bigger than BitwisePassage.</div><div><br></div><div>
The PassageKeyFactory is used throughout JSword to create the preferred type of Passage. You can set it to one of the above via PassageType typically at program startup.</div><div><br></div><div>I wrote BitwisePassage; Joe wrote the others.</div>
<div><br></div><div>I've been thinking that there might be another variation of BitwisePassage that might make sense. One with testament or book bitmaps.</div><div><br></div><div>Hope this helps.</div><div><br></div>
<div>
In Him,</div><div><span style="white-space:pre-wrap"> </span>DM</div><div><div><div><br></div><div><br></div><div><div><div>On Jan 18, 2014, at 2:42 PM, Chris Burrell <<a href="mailto:chris@burrell.me.uk" target="_blank">chris@burrell.me.uk</a>> wrote:</div>
<br><blockquote type="cite"><p dir="ltr">The reason I went for passages is for less memory consumption. I also thought it would provide better parsing and we're parsing less. </p><p dir="ltr">But I think the way we store passages is to represent every verse by a bit. means that iterating through ranges, calculating cardinality or initialising them means is expensive. </p>
<p dir="ltr">I wonder if we could have a passage that just stores boundaries rather than all its content. That would be very fast to parse and very quick for operations on it. </p><p dir="ltr">Chris<br>
</p>
<div class="gmail_quote">On 18 Jan 2014 19:17, "Martin Denham" <<a href="mailto:mjdenham@gmail.com" target="_blank">mjdenham@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I think I have found a workaround for this problem (using a background thread makes it run very quickly) but it seems to point toward a possible point of improvement in the new JSword mapping data structure.<div>
<br></div><div>In the old AB v11n mapping code I stored maps of Verses. However the new JSword code seems to store maps of RocketPassage/BitwisePassage. Can you verify this is the best data structure for what will normally be a single verse. I don't know an awful lot about this so I am just asking, but maybe a Passage will always consume more memory than a Verse.</div>
<div><br></div><div>Thanks</div><div>Martin</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 18 January 2014 13:30, Martin Denham <span dir="ltr"><<a href="mailto:mjdenham@gmail.com" target="_blank">mjdenham@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I tried your patch and it did noticeably improve things, so it is a worthwhile patch, but did not solve the actual problem. Still viewing the RusSynodal straight after download takes well over a minute, whereas to boot straight into it takes about 2 secs.<div>
<br></div><div>One or 2 things I have noticed:</div><div><ul><li>a huge amount of memory allocation and GC occurs when trying to view RusSynodal straight after download<br></li><li>I am not great with the profiler but here are my tentative observations</li>
<ul><li>Seems to be spending about half the time in java.util.BitSet.cardinality() </li><ul><li>called by BitwisePassage.countVerses()<-RocketPassage.countVerses()<-AbstractPassage.getCardinality()<-VersificationToKJVMapper.add...</li>
</ul><li>I tried caching AbstractPassage.getCardinality but that had no affect which would be the case if there were lots of different keys</li></ul><li>Could it be that a module is in an unusual state after download? I have noticed that a module cannot be deleted directly after download, until the app is restarted due to:</li>
<ul><li>sbmd.getConfigFile()==null when called by SwordBookDriver.isDeletable directly after module download</li></ul>
</ul><div>Any suggestions are welcome.</div></div><div><br></div><div>Cheers</div><span><font color="#888888">Martin</font></span><div><div class="gmail_extra"><br><br><div class="gmail_quote">
On 16 January 2014 23:13, Martin Denham <span dir="ltr"><<a href="mailto:mjdenham@gmail.com" target="_blank">mjdenham@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">I think I will use the old AB v11n mapping for the next release of AB and come back to this after the release which has quite a lot of changes in it already anyway.<div>
<br></div><div>The problem could be AB, Android, or JSword related and it is not trivial to reproduce involving repeated re-installs of RusSynodal. I had a similar problem a year or two ago, it took a long time to investigate and ended up being a flaw in the Android VM which required a change to JSword. </div>
<div><br></div><div>I keep trying to think what could cause these symptoms and drawing a blank.</div><div><br></div><div>Cheers</div><span><font color="#888888">Martin</font></span></div><div>
<div><div class="gmail_extra"><br><br><div class="gmail_quote">On 16 January 2014 21:35, Chris Burrell <span dir="ltr"><<a href="mailto:chris@burrell.me.uk" target="_blank">chris@burrell.me.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><p dir="ltr">I'll try profiling it! Nothing comes to mind though. </p>
<div>
<div class="gmail_quote">On 16 Jan 2014 20:04, "Martin Denham" <<a href="mailto:mjdenham@gmail.com" target="_blank">mjdenham@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">There is something unusual happening with the v11n mapping creation straight after downloading a file requiring mapping e.g. It takes over 2 mins to load the mapping (a single call to mapVerse) for RusSynodal straight after download but a few seconds to load the file normally.<div>
<br></div><div>1. View ESV and anything else in split screen</div><div>2. Download RusSynodal</div><div>3. Exit download screen after download</div><div>4. select RusSynodal for display (with ESV) in 1 half of split screen</div>
<div>5. Mapping code now takes over 2 minutes to load mapping</div><div>6. But there are no errors and then everything seems fine </div><div><br></div><div>7. Exit and kill And Bible</div><div>8. Restart with ESV/RST in split screens</div>
<div>9. This time everything is initialised in a few seconds</div><div><br></div><div>Any ideas?</div><div><br></div><div>Martin</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 16 January 2014 19:06, Chris Burrell <span dir="ltr"><<a href="mailto:chris@burrell.me.uk" target="_blank">chris@burrell.me.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><p dir="ltr">Hi Martin</p><p dir="ltr">Fine by me. Think it should be opt in if possible. </p>
<p dir="ltr">A couple of thoughts on that note. Do you know of it s the io or the cpu time that's the issue? </p><p dir="ltr">We probably want to opt in with a list as well as all as there's no point in loading the v11n if not required. </p>
<p dir="ltr">I guess especially if we end up with mappings per module eventually. </p><p dir="ltr">On a separate note, I also found the books.installed call is very expensive. Thinking it may be worth partially loading these. With around 200 modules we spend almost 15 seconds loading them. </p>
<p dir="ltr">Finally you can apply for a open source license of jprofiler which helps massively to work out what's going on. Have got a couple of uncommitted fixes for books. Installed find with that. </p><p dir="ltr">
Chris</p>
<div class="gmail_quote"><div>On 16 Jan 2014 18:10, "Martin Denham" <<a href="mailto:mjdenham@gmail.com" target="_blank">mjdenham@gmail.com</a>> wrote:<br type="attribution"></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div>
<div dir="ltr">Hi,<div><br></div><div>I have integrated the new v11n mapping code and find I am getting a pause when doing the initial mapping between any 2 v11ns.</div><div><br></div><div>I originally had the same problem with the AB mapping code so pre-loaded all mapping required for the installed set of documents at the start in a background thread to prevent delays. The code I used is in <a href="https://github.com/mjdenham/and-bible/blob/master/AndBible/src/net/bible/android/control/versification/mapping/VersificationMappingFactory.java" target="_blank">VersificationMappingFactory.<span style="color:rgb(153,0,0);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12px;font-weight:bold;line-height:18px;white-space:pre-wrap">initialiseRequiredMappings()</span></a>.</div>
<div><br></div><div>I am trying to think of the best way to do this in the new code. Either i) you could add a method to do it which could be called or ii) you could proeload all required mappings automatically or iii) add a new method to allow AB (or other frontend) to trigger preload of all required mappings by adding a public method a bit like the new VersificationsMapper.ensure(v11ntopreload).</div>
<div><br></div><div>This is quite an issue for mobile users. I have a fast mobile and loading a mapping causes a noticeable delay the first time a verse changes, but the preload fix is fairly simple and worked well.</div>
<div><br></div><div>Cheers</div><div>Martin</div><div><br></div></div>
<br></div>_______________________________________________<br>
jsword-devel mailing list<br>
<a href="mailto:jsword-devel@crosswire.org" target="_blank">jsword-devel@crosswire.org</a><br>
<a href="http://www.crosswire.org/mailman/listinfo/jsword-devel" target="_blank">http://www.crosswire.org/mailman/listinfo/jsword-devel</a><br>
<br></blockquote></div>
<br>_______________________________________________<br>
jsword-devel mailing list<br>
<a href="mailto:jsword-devel@crosswire.org" target="_blank">jsword-devel@crosswire.org</a><br>
<a href="http://www.crosswire.org/mailman/listinfo/jsword-devel" target="_blank">http://www.crosswire.org/mailman/listinfo/jsword-devel</a><br>
<br></blockquote></div><br></div>
</blockquote></div>
</div></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div>
</blockquote></div>
_______________________________________________<br>jsword-devel mailing list<br><a href="mailto:jsword-devel@crosswire.org" target="_blank">jsword-devel@crosswire.org</a><br><a href="http://www.crosswire.org/mailman/listinfo/jsword-devel" target="_blank">http://www.crosswire.org/mailman/listinfo/jsword-devel</a><br>
</blockquote></div><br></div></div></div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>