[jsword-devel] Out of Memory Issues Loading repo module lists

Martin Denham mjdenham at gmail.com
Sat Jan 16 09:48:35 MST 2016 

Could you add Version to the core list of attributes initially loaded.

Martin

On 16 January 2016 at 15:13, Martin Denham <mjdenham at gmail.com> wrote:

> Hi DM,
>
> I think it is a good time to do another alpha build of AB.  Could you
> commit Ini.java before I do a build?
>
> Martin
>
> On 15 January 2016 at 21:45, DM Smith <dmsmith at crosswire.org> wrote:
>
>> Glad you like it. I missed Ini.java in the last commit. I had the change
>> already in my system.
>>
>> On Jan 15, 2016, at 4:15 PM, Martin Denham <mjdenham at gmail.com> wrote:
>>
>> Wow, just tried Reduced the footprint of IniSection, used by
>> SwordBookMetaData.
>> <https://github.com/crosswire/jsword/commit/24343c32bfd25586a3b85a32df454525890ed51f> and
>> it uses even less memory than my experiments: 3Mb for eBible and 4.5Mb for
>> all repos.
>>
>> There seems to be a minor dependency error in Ini.java as a result of
>> changes in IniSection.  I just changed List to Collection to make it
>> compile.
>>     public List<String> getKeys()
>>
>> Martin
>>
>> On 15 January 2016 at 18:36, DM Smith <dmsmith at crosswire.org> wrote:
>>
>>> I haven’t had a moment to look at the internal data structures. One of
>>> the natures of IniSection is that it is a multi-map. That means a key can
>>> be repeated one or more times. Though the typical case is that it is not
>>> repeated.
>>>
>>> So get(String key, int index) is needed.
>>>
>>> The order of values for a repeated key needs to be retained. So for a
>>> given key that has n values, they can be logically accessed as 0 .. n-1.
>>> Being able to iterate over the list to get them is just fine. It doesn’t
>>> need to be indexed access. Aside: The reason for keeping them ordered is
>>> that a SWORD app expects GlobalOptionFilter to be executed in the order it
>>> occurs. It is not readily obvious but each of the values for a
>>> GlobalOptionFilter is the name of a C++ class. The collection of them forms
>>> a chain of modification on the raw text from a module. While we are
>>> independent of SWORD apps and library, we try to coexist and support the
>>> same underlying abilities.
>>> So HashMap<String, HashSet<String>> doesn’t retain the order of the
>>> values.
>>>
>>> The current implementation has the keys being ordered by insertion
>>> order. (TreeMap and List). This is no longer necessary. We’ve added the
>>> ability to request the keys in a particular order. SWORD treats the conf
>>> file as a database of key, value pairs where order (other than for repeated
>>> keys) is unimportant.
>>>
>>> If a key/value is repeated, it is ignored. The code does this by using a
>>> set. But it could have used a list and call contains on it. It is a
>>> time/space trade-off.
>>>
>>> Couple of options:
>>> 1) Have two data structures:
>>> One to hold key, value pairs.
>>> Another to hold key, value lists.
>>> The invariant is that a stored key can be in at most one of these.
>>> The trick is that when adding a key,value pair an entry may need to
>>> migrate from the one data structure to the other.
>>> Also, removing a key/value means looking at both data structures and
>>> possibly migrating from one to the other.
>>> The old code did something like this, but I opted for something simpler.
>>> 2) Use lighterweight data structures that meet the minimum requirements
>>> of order.
>>>
>>> I’ll give it a go.
>>>
>>> — DM
>>>
>>> On Jan 15, 2016, at 12:35 PM, Martin Denham <mjdenham at gmail.com> wrote:
>>>
>>> I have done a few experiments using different data structures in
>>> IniSection and they seem to occupy a lot less memory but some IniSection
>>> functions are no longer well supported e.g. get(String key, int index), so
>>> I am not sure if this is worth pursuing.
>>>
>>> HashMap<String, String> 3Mb for all repos (forgot to check just eBible)
>>> HashMap<String, HashSet<String>> 3.6Mb for eBible and 5Mb for all repos
>>>
>>> This compares to 10Mb just for eBible with the original data structure.
>>>
>>> Martin
>>>
>>> On 15 January 2016 at 15:05, Martin Denham <mjdenham at gmail.com> wrote:
>>>
>>>> I was also thinking that maybe the default should be to return a full
>>>> sbmd and another method or parameter will cause a partial sbmd to be
>>>> returned.  This would prevent surprises for other JSword users who don't
>>>> have tight memory constraints.
>>>>
>>>> Martin
>>>>
>>>> On 15 January 2016 at 14:11, DM Smith <dmsmith at crosswire.org> wrote:
>>>>
>>>>> I’ll do that soon. And get back to you. I hadn’t meant to move
>>>>> GatherAllReferences and ReadEverything from tests to examples at this time,
>>>>> but later.
>>>>>
>>>>> — DM
>>>>>
>>>>> On Jan 15, 2016, at 9:09 AM, Martin Denham <mjdenham at gmail.com> wrote:
>>>>>
>>>>> Hi DM,
>>>>>
>>>>> Could you verify that you have checked in the Category fix as I still
>>>>> have the problem after fetching the latest commit?  The last commit says
>>>>> 'Fixed bug in category' but only includes changes in the examples package.
>>>>> Also, the changes in the examples package, GatherAllReferences and
>>>>> ReadEverything, do not compile as they are in the wrong package.
>>>>>
>>>>> Martin
>>>>>
>>>>> On 14 January 2016 at 02:16, DM Smith <dmsmith at crosswire.org> wrote:
>>>>>
>>>>>> Glad it is a good, workable solution.
>>>>>>
>>>>>> I saw that category problem earlier today and checked in a fix for
>>>>>> it. I was addressing a problem with Bible Desktop’s display of a red ? over
>>>>>> the cult bibles in the installer. Tried breaking it out into it’s own
>>>>>> category. Tried putting it into “Other”. Not satisfied quite yet. But all
>>>>>> the others are appropriately classified.
>>>>>> — DM
>>>>>>
>>>>>> On Jan 13, 2016, at 5:30 PM, Martin Denham <mjdenham at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> The latest code seems to be running quite smoothly.
>>>>>> eBible 680 modules 10Mb ram
>>>>>>
>>>>>> I could not notice any new major pauses.
>>>>>> sbmd.reload works well and AB can show the About dialog.  I can see a
>>>>>> slight but insignificant pause as the full attributes are loaded.
>>>>>> I have installed various modules from different repositories.
>>>>>> The above tests were done on a fairly low spec Android 2.2 AVD with
>>>>>> 64Mb heap.
>>>>>>
>>>>>> The only issue I have noticed is that non-Bibles are appearing in the
>>>>>> list of Bibles so I think there may be an issue with Category.  I can see
>>>>>> some commentaries and GenBooks in a list that should just contain Bibles.
>>>>>> If the problem is not obvious I can investigate further later.
>>>>>>
>>>>>> Thanks
>>>>>> Martin
>>>>>>
>>>>>> On 12 January 2016 at 13:58, DM Smith <dmsmith at crosswire.org> wrote:
>>>>>>
>>>>>>> I’m working on transforming the tar.gz to a zip. The zip has much
>>>>>>> faster access to files in it. The same amount of time per file. The tar.gz
>>>>>>> is Tape ARchive and is fast to get the first and slow to get the last.
>>>>>>>
>>>>>>> I just did some computations and unpacking the tar.gz is not good
>>>>>>> for your app. But you told me that…. :)
>>>>>>>
>>>>>>> In Him,
>>>>>>> DM
>>>>>>>
>>>>>>> On Jan 12, 2016, at 8:16 AM, Martin Denham <mjdenham at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> I like this idea: "A file in a jar has an URL that is something
>>>>>>> like …/fred.jar!file"
>>>>>>>
>>>>>>> Martin
>>>>>>>
>>>>>>> On 11 January 2016 at 23:29, DM Smith <dmsmith at crosswire.org> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> On Jan 11, 2016, at 6:07 PM, Martin Denham <mjdenham at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> My estimate of file size might be too low because I forgot to take
>>>>>>>> into account block size.  Quickly playing around with my android adds about
>>>>>>>> 40% making it at least 7Mb for the conf files.
>>>>>>>>
>>>>>>>>
>>>>>>>> Understand.
>>>>>>>>
>>>>>>>>
>>>>>>>> By 'fluff' do you mean extract all the files from mods.d.tar.gz and
>>>>>>>> write them all to disk.  I am a little concerned about writing and deleting
>>>>>>>> hundreds of small files to the sd card repeatedly.  SD cards are not as
>>>>>>>> good at high r/w as normal disks or flash drives.  That is the reason I do
>>>>>>>> not store the AB database on the SD card.
>>>>>>>>
>>>>>>>>
>>>>>>>> By fluff, I meant that the conf file would be re-read without a
>>>>>>>> filter, thus getting everything.
>>>>>>>>
>>>>>>>>
>>>>>>>> The process described would also make viewing a description (in AB
>>>>>>>> right-click About) an unexpectedly expensive operation involving writing
>>>>>>>> hundreds of files to the sd card.
>>>>>>>>
>>>>>>>>
>>>>>>>> It would involve re-reading the one file without a filter. It
>>>>>>>> should happen fast.
>>>>>>>>
>>>>>>>>
>>>>>>>> I did not know about sbmd.toOSIS() and have not used it.  AB just
>>>>>>>> pops up a little dialog with a few fields like About, copyright, licence,
>>>>>>>> version, versification.
>>>>>>>>
>>>>>>>>
>>>>>>>> Ok. Then you’ll need to call “fluff” before retreiving those
>>>>>>>> fields. The code for fluff (or whatever we call it) would be something like:
>>>>>>>> public void fluff() {
>>>>>>>>   if (partiallyLoaded) {
>>>>>>>> re-read and process the conf without a filter
>>>>>>>> partiallyLoaded = false;
>>>>>>>>    }
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> For the 2 reasons above my preference would be to avoid writing
>>>>>>>> hundreds of files to the SD card but I can't think of a perfect solution.
>>>>>>>> While grappling with this last week I was just trying to get the original
>>>>>>>> code to work more efficiently (but failed).  I am not very experienced in
>>>>>>>> Memory Analysis but suspected the memory use was higher than it might have
>>>>>>>> been.
>>>>>>>>
>>>>>>>> By design, which files do you write to SD card? If they are only
>>>>>>>> written when the mods.d.tar.gz is downloaded, would that help?
>>>>>>>>
>>>>>>>>
>>>>>>>> If the tar.gz was searched each time for the conf it would be more
>>>>>>>> expensive to process the tar.gz every time a Description is requested but
>>>>>>>> the first time it would be quicker than writing hundreds of .conf files and
>>>>>>>> to be honest I think a lot of people do not know about the long-press menu
>>>>>>>> in AB so probably just the initial list of modules would be used most of
>>>>>>>> the time.
>>>>>>>>
>>>>>>>>
>>>>>>>> I don’t know about the long-press menu. In BibleDesktop, it is easy
>>>>>>>> to navigate from one available to the next and each time it shows the full
>>>>>>>> conf.
>>>>>>>>
>>>>>>>>
>>>>>>>> Coincidentally my android slowed to a crawl when I tried to copy
>>>>>>>> all of eBible's .conf files to it just now - initially fast then 1 file per
>>>>>>>> 3 secs, after 10 minutes I unplugged it, although that probably is not a
>>>>>>>> realistic test and there is probably an explanation for the issue.
>>>>>>>>
>>>>>>>>
>>>>>>>> I’ve nearly got the code written to unpack the conf. Let me zip up
>>>>>>>> the files that have changed and send them to you.
>>>>>>>>
>>>>>>>> Basically, if you delete mods.d.tar.gz, it will fetch a new one
>>>>>>>> (current behavior). If you delete mods.d/ it will unpack mods.d.tar.gz into
>>>>>>>> it. If you fetch mods.d.tar.gz it will unpack it into mods.d. All of this
>>>>>>>> takes place in the folder that mods.d.tar.gz is present.
>>>>>>>>
>>>>>>>> I tried adding new confs to mods.d that weren’t in mods.d.tar.gz to
>>>>>>>> simulate a takedown and that works as well.
>>>>>>>>
>>>>>>>> If this code is no good for you, I’ve another thought. A file in a
>>>>>>>> jar has an URL that is something like …/fred.jar!file. Maybe we can
>>>>>>>> transform the mods.d.tar.gz into mods.d.tar and use that addressing
>>>>>>>> mechanism to fetch the file? I’ll take a look at how the JRE does that.
>>>>>>>> Maybe, I’ll roll the same for JSword over a tar.gz file.
>>>>>>>>
>>>>>>>> DM
>>>>>>>>
>>>>>>>>
>>>>>>>> Martin
>>>>>>>>
>>>>>>>> On 11 January 2016 at 19:28, DM Smith <dmsmith at crosswire.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I have been thinking about this a bit more. I was knew there was a
>>>>>>>>> need to prevent stale confs. The time performance is something that I’m not
>>>>>>>>> able to test. My machine has an SSD, a fast 4 core CPU and gobs of RAM. So
>>>>>>>>> I need you to keep me in line. ;)
>>>>>>>>>
>>>>>>>>> The easiest way to keep it pristine is to unpack it into a
>>>>>>>>> temporary folder, rename the old folder and then rename the new folder.
>>>>>>>>> Finally deleting the old folder. By doing it in this order it minimizes the
>>>>>>>>> time that mods.d is unavailable. Important for multi-threaded apps and
>>>>>>>>> multiple apps that share the same machine simultaneously.
>>>>>>>>>
>>>>>>>>> Right now the SwordBookMetaData remembers the File for the conf of
>>>>>>>>> installed modules and is able to re-read it easily. But it does not store
>>>>>>>>> anything about a conf’s location when it is from mods.d.tar.gz. I suppose I
>>>>>>>>> could have it remember the location of mods.d.tar.gz and the name of the
>>>>>>>>> conf entry and create a method to extract a that conf out of the compressed
>>>>>>>>> archive. This would need to be done for each module that the user requests
>>>>>>>>> info. To do this is quite expensive as it means inflating the file then
>>>>>>>>> iterating over the contents until the desired conf is found.
>>>>>>>>>
>>>>>>>>> I think that it would be better to see how much time it adds to
>>>>>>>>> extract the files and store them on disk. The fluffing of them would only
>>>>>>>>> be when the user wants to browse a description of the module.
>>>>>>>>>
>>>>>>>>> I’d like to modify sbmd.toOSIS to check if the sbmd is partial or
>>>>>>>>> full and if not full re-read the conf fully and then continue as before. I
>>>>>>>>> think that is how JSword is designed to retreive the conf for presentation
>>>>>>>>> to the end user. Does AndBible use that or some other mechanism to get what
>>>>>>>>> it wants for presentation?
>>>>>>>>>
>>>>>>>>> I think I’ll add a “fluff” method to BookMetaData that will do
>>>>>>>>> this. This could be called to get it to fluff at another time.
>>>>>>>>>
>>>>>>>>> DM
>>>>>>>>>
>>>>>>>>> On Jan 11, 2016, at 1:00 PM, Martin Denham <mjdenham at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> My rough estimates have the total size of conf files in all repos
>>>>>>>>> at about 5Mb which is not too different to the size of a module like ESV so
>>>>>>>>> the impact should not be significant and it should not be a problem if this
>>>>>>>>> is required.
>>>>>>>>>
>>>>>>>>> Other things to consider that come to mind i) would need to remove
>>>>>>>>> conf files no longer in mods.d.tar.gz or delete and re-extract everything
>>>>>>>>> after a refresh ii) Time taken to save files - loading the list is already
>>>>>>>>> slow.
>>>>>>>>>
>>>>>>>>> I can't think of any major reason not to do as you describe.
>>>>>>>>>
>>>>>>>>> However, would an easier approach be to find files in the zip a
>>>>>>>>> bit like this
>>>>>>>>> <http://stackoverflow.com/questions/11123528/finding-a-file-in-zipentry-java>.
>>>>>>>>> Speed would not be an issue because it would only be done once or twice
>>>>>>>>> after fetching the list e.g. to view About or to actually download.  The
>>>>>>>>> mod.conf file name/path could be saved in SBMD if required.
>>>>>>>>>
>>>>>>>>> Martin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 11 January 2016 at 01:39, DM Smith <dmsmith at crosswire.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I’m trying to figure out how to reload a conf from a remote
>>>>>>>>>> source (to go from a partial load to a full load).  The problem is that the
>>>>>>>>>> AbstractSwordInstaller sits over top of mods.d.tar.gz, which it does not
>>>>>>>>>> unpack. Instead, it iterates over all the entries in that binary file and
>>>>>>>>>> handles each entry (i.e. a conf) in core. It doesn’t hit the disk. I’m
>>>>>>>>>> wondering whether it would be alright to unpack the file in the same
>>>>>>>>>> folder? That would allow a SwordBookMetaData to reload the file. It would
>>>>>>>>>> also mean that SwordBookMetaData would only need one means of reading a
>>>>>>>>>> conf as it’d be a file and not a byte array.
>>>>>>>>>>
>>>>>>>>>> It isn’t a problem with desktop or server apps, but it might be
>>>>>>>>>> for AndBible.
>>>>>>>>>>
>>>>>>>>>> — DM
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Jan 10, 2016, at 3:31 PM, DM Smith <dmsmith at crosswire.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> The problem you encountered was 2 bugs:
>>>>>>>>>> When the module is not UTF-8 the remote repository’s conf is
>>>>>>>>>> re-read, but the filter wasn’t passed.
>>>>>>>>>> Not intended, but IniSection required a filter, rather than
>>>>>>>>>> saying a null filter meant everything passed.
>>>>>>>>>>
>>>>>>>>>> I’ve checked in that fix. Still trying to make the memory less….
>>>>>>>>>>
>>>>>>>>>> — DM
>>>>>>>>>>
>>>>>>>>>> On Jan 10, 2016, at 1:18 PM, DM Smith <dmsmith at crosswire.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> The “Partial load of conf file.’ was to load all of the things in
>>>>>>>>>> a conf that the JSword engine needs to work with a module. I don’t know why
>>>>>>>>>> the CrossWire repo is working for me but not for you. I’ll keep working on
>>>>>>>>>> it today. The problem with the previous commit was fixed with the last
>>>>>>>>>> commit. I wasn’t “adjusting” the module after loading to fill in things
>>>>>>>>>> like BookDriver and BookCategory.
>>>>>>>>>>
>>>>>>>>>> I’m wondering whether getting the list of Books from the
>>>>>>>>>> installer creates a deep rather than a shallow copy of them.
>>>>>>>>>>
>>>>>>>>>> Today I hope to make SwordBookMetaData even more lazy. It has a
>>>>>>>>>> BookDriver and validates its storage when the repo is loaded. I plan to
>>>>>>>>>> break one of my modules by renaming one of the files and see the impact.
>>>>>>>>>> Chris and I have noticed that the FileState objects are not fully released.
>>>>>>>>>> This actually is part of the design.
>>>>>>>>>>
>>>>>>>>>> Anyway, I think it is going in the right direction. Reducing the
>>>>>>>>>> memory 4x is a  good thing. The data structures within the IniSection may
>>>>>>>>>> be too heavy. I may relax the requirement that it maintains the SWORD confs
>>>>>>>>>> order. The idea was to be able to modify the provided conf, retaining its
>>>>>>>>>> order. However, now we never modify that conf.
>>>>>>>>>>
>>>>>>>>>> configAll was a deep clone of configSword. configAll adds in the
>>>>>>>>>> contents of configJSword and then configFrontend. These last two are
>>>>>>>>>> created even if not needed. We could make them lazy as well.
>>>>>>>>>>
>>>>>>>>>> DM
>>>>>>>>>>
>>>>>>>>>> On Jan 10, 2016, at 11:07 AM, Martin Denham <mjdenham at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Thanks for the quick response.  I have had a brief look at the
>>>>>>>>>> new commits.
>>>>>>>>>>
>>>>>>>>>> A lot of the attributes aren't being returned now so it is tricky
>>>>>>>>>> to test and there are various errors but running the current tip 'Partial
>>>>>>>>>> load of conf file.
>>>>>>>>>> <https://github.com/crosswire/jsword/commit/80020f51c6a762d458ce8ae70007b78eadee1fb3>'
>>>>>>>>>> the SBMD for eBible is now only a quarter of the original size at 10Mb
>>>>>>>>>> which is fine but I still don't understand why it is so large for the
>>>>>>>>>> minimal attribute set now being returned.
>>>>>>>>>>
>>>>>>>>>> I get a lot of errors like:
>>>>>>>>>> SwordBookMetaData(492): Book not supported: malformed conf file
>>>>>>>>>> for [BBE] no ModDrv found.
>>>>>>>>>> SwordBookMetaData(492): Malformed conf file: missing
>>>>>>>>>> [BBE]Description=. Using BBE
>>>>>>>>>>
>>>>>>>>>> and peculiarly the eBible repo seems to be the only repo I can
>>>>>>>>>> use because all the others error.
>>>>>>>>>>
>>>>>>>>>> I also tried the previous commit Cut the memory requirements of
>>>>>>>>>> a SwordBookMetaData in half.
>>>>>>>>>> <https://github.com/crosswire/jsword/commit/cc32ba8f1bb245932a747390d03874b2be70e9a1> but
>>>>>>>>>> it did not work because basic attributes like language were not being
>>>>>>>>>> returned.
>>>>>>>>>>
>>>>>>>>>> I still don't understand why removing configSword should reduce
>>>>>>>>>> memory by half because it should just be removing references to data that
>>>>>>>>>> is also referenced from configAll, so it would reduce memory slightly but
>>>>>>>>>> not much.
>>>>>>>>>>
>>>>>>>>>> Martin
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 10 January 2016 at 04:14, DM Smith <dmsmith at crosswire.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> OK. That’s done. Also accidentally introduced a bug with the
>>>>>>>>>>> last commit. It is noticeably fast.
>>>>>>>>>>>
>>>>>>>>>>> Next up, allow for *a* SwordBookMetaData to be reloaded fully.
>>>>>>>>>>> This is needed to bring in all the other elements which are information
>>>>>>>>>>> only, such as About, in order to display info to the end user. Since the
>>>>>>>>>>> user will only look at one modules info at a time, it will load that one.
>>>>>>>>>>> You may need to change your code (hope not) to force that one to reload.
>>>>>>>>>>>
>>>>>>>>>>> Give the code a try to see if it solves your out of memory error.
>>>>>>>>>>>
>>>>>>>>>>> DM
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Jan 9, 2016, at 9:06 PM, DM Smith <dmsmith at crosswire.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> I’ll be adding a filter to IniSection. Something like:
>>>>>>>>>>> if  (filter.test(key)) {
>>>>>>>>>>> use the key
>>>>>>>>>>> } else {
>>>>>>>>>>> do nothing
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> SwordBookMetaData will be responsible for building the filter.
>>>>>>>>>>> At least for a first go around. A single object should do.
>>>>>>>>>>>
>>>>>>>>>>> DM
>>>>>>>>>>>
>>>>>>>>>>> On Jan 9, 2016, at 6:29 PM, DM Smith <dmsmith at crosswire.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Yes, like you I have thought of streamlining conf loading for
>>>>>>>>>>> repo lists.  One idea I had was to enable specification of a filter to
>>>>>>>>>>> SwordBookMetaData to limit the conf values that are stored.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I was thinking of something similar. My ideas aren’t good enough
>>>>>>>>>>> to be put into practice, but some kind of flag indicating empty, partially
>>>>>>>>>>> or fully loaded. Empty would mean that it hasn’t gone to disk to get the
>>>>>>>>>>> conf. Partial means that it read everything, but threw away most as not
>>>>>>>>>>> interesting (since the conf does not have order you have to read and parse
>>>>>>>>>>> it all). Full would mean that nothing was pitched.
>>>>>>>>>>> SwordBookMetaData.getProperty would need to be changed to determine whether
>>>>>>>>>>> the key is in memory or might be on disk and do the right thing. Or we
>>>>>>>>>>> could keep getProperty as it is and if you want one of the fields that is
>>>>>>>>>>> not stored (e.g. About) you have to call reload().
>>>>>>>>>>>
>>>>>>>>>>> Maybe we could also cache that info into a separate file(s)?
>>>>>>>>>>> When mods.d.tar.gz is updated then the cache would be recomputed. In doing
>>>>>>>>>>> the computation, each conf would be read then pitched. Basically, the
>>>>>>>>>>> storage would be o.c.c.utils.Ini, if one file or IniSection, if many files.
>>>>>>>>>>>
>>>>>>>>>>> What do you think?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> jsword-devel mailing list
>>>>>>>>>>> jsword-devel at crosswire.org
>>>>>>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> jsword-devel mailing list
>>>>>>>>>>> jsword-devel at crosswire.org
>>>>>>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> jsword-devel mailing list
>>>>>>>>>> jsword-devel at crosswire.org
>>>>>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> jsword-devel mailing list
>>>>>>>>>> jsword-devel at crosswire.org
>>>>>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> jsword-devel mailing list
>>>>>>>>>> jsword-devel at crosswire.org
>>>>>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> jsword-devel mailing list
>>>>>>>>>> jsword-devel at crosswire.org
>>>>>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> jsword-devel mailing list
>>>>>>>>> jsword-devel at crosswire.org
>>>>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> jsword-devel mailing list
>>>>>>>>> jsword-devel at crosswire.org
>>>>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>>>>
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> jsword-devel mailing list
>>>>>>>> jsword-devel at crosswire.org
>>>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> jsword-devel mailing list
>>>>>>>> jsword-devel at crosswire.org
>>>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> jsword-devel mailing list
>>>>>>> jsword-devel at crosswire.org
>>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> jsword-devel mailing list
>>>>>>> jsword-devel at crosswire.org
>>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> jsword-devel mailing list
>>>>>> jsword-devel at crosswire.org
>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> jsword-devel mailing list
>>>>>> jsword-devel at crosswire.org
>>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> jsword-devel mailing list
>>>>> jsword-devel at crosswire.org
>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> jsword-devel mailing list
>>>>> jsword-devel at crosswire.org
>>>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>>>
>>>>>
>>>>
>>> _______________________________________________
>>> jsword-devel mailing list
>>> jsword-devel at crosswire.org
>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>
>>>
>>>
>>> _______________________________________________
>>> jsword-devel mailing list
>>> jsword-devel at crosswire.org
>>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>>
>>>
>> _______________________________________________
>> jsword-devel mailing list
>> jsword-devel at crosswire.org
>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>
>>
>>
>> _______________________________________________
>> jsword-devel mailing list
>> jsword-devel at crosswire.org
>> http://www.crosswire.org/mailman/listinfo/jsword-devel
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/jsword-devel/attachments/20160116/f6605af1/attachment-0001.html>

More information about the jsword-devel mailing list