<div dir="ltr">While debugging I was looking for inefficiencies leading to unintended storage of data but really struggled to analyse the memory use. A couple of things I noted:<div><ul><li>many values had long sequences of \u0000 at the end. I don't know why but they disappeared after adding buf.trimToSize() at the end of IniSection.more()<br></li><li>IniSections.warnings can get long and is never cleared</li><li>Inisection.MAX_BUFF_SIZE is probably too large for conf files</li><li>Can buffer memory creation be moved outside the loop in AbstractSwordInstaller.loadCachedIndex; and just cleared at the start of each loop; to relieve some stress on the gc</li><li>SwordBookMetaData spends most of it's time doing pattern matching in report(). Not sure if there is any quicker alternative<br></li><li>In memory analysis I saw many key strings and value strings (as expected) but I also saw many complete config line strings and could not work out where they were being stored e.g. "History_2.18=Automatically generated on 2015-12-09 from source files dated 2014-09-16 by the Digital Bible Society (<a href="http://dbs.org">http://dbs.org</a>)". If we store keys, and values, and the whole key=value string then that would double data storage, but try as I might I could not find where or how the whole line was being stored.</li></ul><div>However, I tried most of the above and still could not load the repositories. I just listed the above suggestions for you to consider.</div></div><div><br></div><div>Removing configSword sounds like a good idea. It could potentially save half the storage. However, it might not save much if it is just storing references to data that is also stored in configJSword - I got confused about this when scanning the code.</div><div><br></div><div>Yes, like you I have thought of streamlining conf loading for repo lists. One idea I had was to enable specification of a filter to SwordBookMetaData to limit the conf values that are stored.</div><div><br></div><div>Martin</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 9 January 2016 at 20:39, DM Smith <span dir="ltr"><<a href="mailto:dmsmith@crosswire.org" target="_blank">dmsmith@crosswire.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hmm. Not sure where the problem could be. Here is an overview of the design, which is quite different than before.<br>
<br>
The SWORD conf is a Windows-style INI file. From a technical perspective it is a collection of named multi-maps. Each key can exist more than once in the file. This is represented by the class o.c.c.utils.Ini. While SWORD allows for all the confs to be concatenated into a single file, which Ini supports. Each conf is a named multi-map. This is represented by o.c.c.utils.IniSection. JSword only uses IniSection.<br>
<br>
The class IniSection maintains the order of the keys as seen in the module and for the order of values for a key that is seen multiple times. As long as all repeated keys are adjacent, the reading and writing of the conf will yield the same lines in the same order. Internally it uses a TreeMap<String, ListSet<String>> to hold the keys and values. It uses List<String> to maintain the insertion order of keys. Rather than spitting out errors as they are seen, they are stored in a StringBuilder. It also remembers the File to which it was loaded.<br>
<br>
This implementation may be too heavy for memory constrained devices.<br>
<br>
The SwordBookMetaData maintains multiple IniSections for each conf on an as needed basis. This is to support the basic idea that the SWORD conf should be kept pristine and any modifications to it should either be in memory or be stored on a shared basis or a private basis. There are 4 IniSection objects that support this:<br>
configSword — The pristine SWORD conf for the module<br>
configJSword — The changes to the SWORD conf that should be persistent and shared by all front-ends<br>
configFrontend — The changes to the SWORD conf that should be persistent and private to the current frontend.<br>
configAll — The merge of the above such that the configFrontend takes precedence over configJSword and configJSword over configFrontend. It only maintains that precedence during loading. putProperty, which targets either configAll, configJSword or configFrontend, does not honor that precedence at this time. (It could.)<br>
<br>
The obvious optimization I see is that since configSword is only used during construction it doesn’t need to exist as a member variable, but as a local variable. That would cut memory by half. I’ll make that change.<br>
<br>
With the increase in the number of modules, we may need to re-think the Books class. There’s no need to have a full blown SwordBookMetaData for each book in a remote repository. People have no interest beyond curiosity in most of the modules. Unless they are showing the content of a conf, the vast majority of the conf is not seen. Unfortunately, the SWORD conf does not have a defined order and necessary elements can be at the end. So reading the entire conf and then analyzing it is necessary.<br>
<br>
Hope this helps you target what else needs to be done.<br>
<br>
DM<br>
<div><div class="h5"><br>
> On Jan 9, 2016, at 2:19 PM, Martin Denham <<a href="mailto:mjdenham@gmail.com">mjdenham@gmail.com</a>> wrote:<br>
><br>
> Testing And Bible on a couple of smaller devices I have noticed OutOfMemory errors during the creation of SwordBookMetaData when attempting to get a list of downloadable modules from a repo. I can just about load the eBible repo or all the other repos but get out of memory errors when loading all repos. I can no longer even load just the Crosswire repo alone on my old G1.<br>
><br>
> All the memory consumed by the list of downloadable modules is consumed by the BookMetaData i.e. bmd.configAll and bmd.configSword.<br>
><br>
> I have been struggling to analyse memory usage by SwordBookMetaData and so I thought I would raise the issue here for comments.<br>
><br>
> My small test Android devices only have about 64Mb of heap, much of which is used by other system classes.<br>
><br>
> Sizes for the list of Books:<br>
> Crosswire 318 modules 10Mb<br>
> eBible 680 modules 41Mb<br>
><br>
> The size of Crosswire conf files is normally less than 1k so 318*1k = 318k<br>
> The size of eBible conf files is up to about 6k so 680*6k = 4megs<br>
><br>
> So I suspect an inefficiency in JSword but have been pulling my hair out trying to find the issue. I tweaked a couple of things but nothing made a lot of difference.<br>
><br>
> Has anybody else noticed anything similar?<br>
><br>
> Martin<br>
</div></div>> _______________________________________________<br>
> jsword-devel mailing list<br>
> <a href="mailto:jsword-devel@crosswire.org">jsword-devel@crosswire.org</a><br>
> <a href="http://www.crosswire.org/mailman/listinfo/jsword-devel" rel="noreferrer" target="_blank">http://www.crosswire.org/mailman/listinfo/jsword-devel</a><br>
<br>
<br>
_______________________________________________<br>
jsword-devel mailing list<br>
<a href="mailto:jsword-devel@crosswire.org">jsword-devel@crosswire.org</a><br>
<a href="http://www.crosswire.org/mailman/listinfo/jsword-devel" rel="noreferrer" target="_blank">http://www.crosswire.org/mailman/listinfo/jsword-devel</a><br>
</blockquote></div><br></div>