[jsword-devel] Out of Memory Issues Loading repo module lists

[DM Smith]

Hmm. Not sure where the problem could be. Here is an overview of the design, which is quite different than before.

The SWORD conf is a Windows-style INI file. From a technical perspective it is a collection of named multi-maps. Each key can exist more than once in the file. This is represented by the class o.c.c.utils.Ini. While SWORD allows for all the confs to be concatenated into a single file, which Ini supports. Each conf is a named multi-map. This is represented by o.c.c.utils.IniSection. JSword only uses IniSection.

The class IniSection maintains the order of the keys as seen in the module and for the order of values for a key that is seen multiple times. As long as all repeated keys are adjacent, the reading and writing of the conf will yield the same lines in the same order. Internally it uses a TreeMap<String, ListSet<String>> to hold the keys and values. It uses List<String> to maintain the insertion order of keys. Rather than spitting out errors as they are seen, they are stored in a StringBuilder. It also remembers the File to which it was loaded.

This implementation may be too heavy for memory constrained devices.

The SwordBookMetaData maintains multiple IniSections for each conf on an as needed basis. This is to support the basic idea that the SWORD conf should be kept pristine and any modifications to it should either be in memory or be stored on a shared basis or a private basis. There are 4 IniSection objects that support this:
configSword — The pristine SWORD conf for the module
configJSword — The changes to the SWORD conf that should be persistent and shared by all front-ends
configFrontend — The changes to the SWORD conf that should be persistent and private to the current frontend.
configAll — The merge of the above such that the configFrontend takes precedence over configJSword and configJSword over configFrontend. It only maintains that precedence during loading. putProperty, which targets either configAll, configJSword or configFrontend, does not honor that precedence at this time. (It could.)

The obvious optimization I see is that since configSword is only used during construction it doesn’t need to exist as a member variable, but as a local variable. That would cut memory by half. I’ll make that change.

With the increase in the number of modules, we may need to re-think the Books class. There’s no need to have a full blown SwordBookMetaData for each book in a remote repository. People have no interest beyond curiosity in most of the modules. Unless they are showing the content of a conf, the vast majority of the conf is not seen. Unfortunately, the SWORD conf does not have a defined order and necessary elements can be at the end. So reading the entire conf and then analyzing it is necessary.

Hope this helps you target what else needs to be done.

DM

> On Jan 9, 2016, at 2:19 PM, Martin Denham <mjdenham at gmail.com> wrote:
> 
> Testing And Bible on a couple of smaller devices I have noticed OutOfMemory errors during the creation of SwordBookMetaData when attempting to get a list of downloadable modules from a repo.  I can just about load the eBible repo or all the other repos but get out of memory errors when loading all repos.  I can no longer even load just the Crosswire repo alone on my old G1.
> 
> All the memory consumed by the list of downloadable modules is consumed by the BookMetaData i.e. bmd.configAll and bmd.configSword.
> 
> I have been struggling to analyse memory usage by SwordBookMetaData and so I thought I would raise the issue here for comments.
> 
> My small test Android devices only have about 64Mb of heap, much of which is used by other system classes.  
> 
> Sizes for the list of Books:
> Crosswire 318 modules 10Mb
> eBible 680 modules 41Mb
> 
> The size of Crosswire conf files is normally less than 1k so 318*1k = 318k
> The size of eBible conf files is up to about 6k so 680*6k = 4megs
> 
> So I suspect an inefficiency in JSword but have been pulling my hair out trying to find the issue.  I tweaked a couple of things but nothing made a lot of difference.
> 
> Has anybody else noticed anything similar?
> 
> Martin
> _______________________________________________
> jsword-devel mailing list
> jsword-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/jsword-devel