[jsword-devel] Out of Memory Issues Loading repo module lists

[Martin Denham]

Thanks for the quick response.  I have had a brief look at the new commits.

A lot of the attributes aren't being returned now so it is tricky to test
and there are various errors but running the current tip 'Partial load of
conf file.
<https://github.com/crosswire/jsword/commit/80020f51c6a762d458ce8ae70007b78eadee1fb3>'
the SBMD for eBible is now only a quarter of the original size at 10Mb
which is fine but I still don't understand why it is so large for the
minimal attribute set now being returned.

I get a lot of errors like:
SwordBookMetaData(492): Book not supported: malformed conf file for [BBE]
no ModDrv found.
SwordBookMetaData(492): Malformed conf file: missing [BBE]Description=.
Using BBE

and peculiarly the eBible repo seems to be the only repo I can use because
all the others error.

I also tried the previous commit Cut the memory requirements of a
SwordBookMetaData in half.
<https://github.com/crosswire/jsword/commit/cc32ba8f1bb245932a747390d03874b2be70e9a1>
but
it did not work because basic attributes like language were not being
returned.

I still don't understand why removing configSword should reduce memory by
half because it should just be removing references to data that is also
referenced from configAll, so it would reduce memory slightly but not much.

Martin



On 10 January 2016 at 04:14, DM Smith <dmsmith at crosswire.org> wrote:

> OK. That’s done. Also accidentally introduced a bug with the last commit.
> It is noticeably fast.
>
> Next up, allow for *a* SwordBookMetaData to be reloaded fully. This is
> needed to bring in all the other elements which are information only, such
> as About, in order to display info to the end user. Since the user will
> only look at one modules info at a time, it will load that one. You may
> need to change your code (hope not) to force that one to reload.
>
> Give the code a try to see if it solves your out of memory error.
>
> DM
>
>
> On Jan 9, 2016, at 9:06 PM, DM Smith <dmsmith at crosswire.org> wrote:
>
> I’ll be adding a filter to IniSection. Something like:
> if  (filter.test(key)) {
> use the key
> } else {
> do nothing
> }
>
> SwordBookMetaData will be responsible for building the filter. At least
> for a first go around. A single object should do.
>
> DM
>
> On Jan 9, 2016, at 6:29 PM, DM Smith <dmsmith at crosswire.org> wrote:
>
>
> Yes, like you I have thought of streamlining conf loading for repo lists.
> One idea I had was to enable specification of a filter to SwordBookMetaData
> to limit the conf values that are stored.
>
>
> I was thinking of something similar. My ideas aren’t good enough to be put
> into practice, but some kind of flag indicating empty, partially or fully
> loaded. Empty would mean that it hasn’t gone to disk to get the conf.
> Partial means that it read everything, but threw away most as not
> interesting (since the conf does not have order you have to read and parse
> it all). Full would mean that nothing was pitched.
> SwordBookMetaData.getProperty would need to be changed to determine whether
> the key is in memory or might be on disk and do the right thing. Or we
> could keep getProperty as it is and if you want one of the fields that is
> not stored (e.g. About) you have to call reload().
>
> Maybe we could also cache that info into a separate file(s)? When
> mods.d.tar.gz is updated then the cache would be recomputed. In doing the
> computation, each conf would be read then pitched. Basically, the storage
> would be o.c.c.utils.Ini, if one file or IniSection, if many files.
>
> What do you think?
>
>
> _______________________________________________
> jsword-devel mailing list
> jsword-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/jsword-devel
>
>
>
> _______________________________________________
> jsword-devel mailing list
> jsword-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/jsword-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/jsword-devel/attachments/20160110/1da6ba90/attachment-0001.html>