[sword-devel] InstallMgr::getModuleStatus / Performance optimization possible?

Sat Nov 5 12:35:31 EDT 2022

Dear Tobias,

Thank you for taking the time to test. Are you sure you have built with 
usrinst.sh options adjusted to switch to release optimizations?

Also, I wanted to mention that the cache is built upon the first scan of 
the files in the folder. I am not sure what your test program does, but 
have a look for a modules-conf.cache file in your mods.d/ folder to see 
if your next SWMgr construction will benefit from the cache. For the 
remote repositories on your desktop, these should live under your home 
folder/.sword/installMgr/...

I'll do a few more tests today to see if there is anything else I can 
speed up. It is good to hear things improved from SVN before my recent 
changes. We should do a profiling pass before a new stable branch to be 
sure we haven't done anything silly over the past year which affects 
performance.

Let me know if you discover any new info,

Troy

On November 4, 2022 10:25:14 AM MST, Tobias Klein <contact at tklein.info> 
wrote:

    Hi Troy,

    Thanks for looking into this!

    Just running this on my Dev PC (Linux, SSD disk, Intel i7) I am
    actually noticing a degradation with this change ... but only at
    first sight and compared to the SWORD version I have been using before.

    I am running this function within a test binary.
    https://github.com/ezra-bible-app/node-sword-interface/blob/1.0.0/src/sword_backend/repository_interface.cpp#L300

    I previously used SVN Rev. 3873 (which is from Nov. 2021).
    With that revision my test binary takes 1.65s to execute.

    After switching to SVN Rev. 3887 (HEAD) the test binary takes 1.95
    seconds to execute, so roughly 300ms slower.

    To be sure, I also tested with the SVN Rev. preceding your latest
    changes (3883) and this was a surprise: With that the test binary
    takes 2.9s to execute.

    So ... compared to SVN Rev. 3883 your latest changes were actually a
    significant performance improvement ... but compared to the version
    from one year ago (which is my production version) it is slower.

    So something else must have changed in the last 12 months that
    impacts performance?!

    Best regards,
    Tobias

    On 11/4/22 12:13 AM, Troy A. Griffitts wrote:
>
>     OK Tobias,
>
>     I have a first cut of a caching mechanism for a mods.d/ folder
>     coded up and pushed to trunk.  I have attempted to remove the
>     cache when the folder is changed (= new module is installed or
>     removed), but I possibly have forgotten something.  Give it a go
>     and let me know if it improves things for you on your slower
>     devices and we'll go from there.
>
>     Troy
>
>
>     On 11/3/22 12:05, Tobias Klein wrote:
>>
>>     Thank you, Troy!
>>
>>     I'd be glad to test any changes you come up with!
>>
>>     Best regards,
>>     Tobias
>>
>>     On 11/3/22 5:57 PM, Troy A. Griffitts wrote:
>>>
>>>     Dear Tobias,
>>>
>>>     I can have a look at optimizing getModuleStatus.  I can
>>>     certainly see that only checking locally installed modules might
>>>     be a speed improvement, but I suspect most of the time is
>>>     loading all the .conf files for each module, and that is done
>>>     when constructing the SWMgr (remote and local).  SWORD
>>>     previously worked with a single modules.conf file where the
>>>     section was appended each time a module was loaded.  It still
>>>     will work in this manner.  I might try dumping all the .conf
>>>     files into a single cache.conf file and using that for reading
>>>     the info of a remote repository-- which can include thousands of
>>>     individual .conf files.  We can see if that speeds up this
>>>     operation.
>>>
>>>     On 10/30/22 08:49, Tobias Klein wrote:
>>>>
>>>>     Hi Troy,
>>>>
>>>>     When integrating the module update functionality in Ezra Bible
>>>>     App, I noticed a performance issue in the function
>>>>     InstallMgr::getModuleStatus.
>>>>
>>>>     On my laptop, it takes almost two seconds to run this against
>>>>     all repositories from the master repo list. On my slower
>>>>     surface tablet, it takes even longer and I haven’t tested it on
>>>>     my even slower Android devices, yet. This generates a bit of an
>>>>     issue in my JavaScript based application (longer interrupts of
>>>>     the JavaScript event loop lead to some freezing in the UI).
>>>>
>>>>     Considering the parameters constSWMgr &base, constSWMgr &other,
>>>>     I saw that the function loops through all modules within other.
>>>>     If one just wants to see which local modules are outdated, it
>>>>     would be enough to go through the ones that are also present
>>>>     within base, right? Could that be a way of optimizing the
>>>>     performance?
>>>>
>>>>     Best regards,
>>>>     Tobias
>>>>
>>>>
>>>>     _______________________________________________
>>>>     sword-devel mailing list:sword-devel at crosswire.org
>>>>     http://crosswire.org/mailman/listinfo/sword-devel
>>>>     Instructions to unsubscribe/change your settings at above page
>>>
>>>     _______________________________________________
>>>     sword-devel mailing list:sword-devel at crosswire.org
>>>     http://crosswire.org/mailman/listinfo/sword-devel
>>>     Instructions to unsubscribe/change your settings at above page
>>
>>     _______________________________________________
>>     sword-devel mailing list:sword-devel at crosswire.org
>>     http://crosswire.org/mailman/listinfo/sword-devel
>>     Instructions to unsubscribe/change your settings at above page
>
>     _______________________________________________
>     sword-devel mailing list:sword-devel at crosswire.org
>     http://crosswire.org/mailman/listinfo/sword-devel
>     Instructions to unsubscribe/change your settings at above page

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://crosswire.org/pipermail/sword-devel/attachments/20221105/e113fe54/attachment.htm>