[sword-devel] Determining the repository a SWModule belongs to

[Troy A. Griffitts]

Hi Tobias,
Yes, thank you for all this. We certainly could optimize SWMgr c-tor for large repos. My guess is that the creation of every SWModule for large repos is taking the time. Many impls of SWModule check their data files on c-tor. SWMgr also adds all the filters to each SWModule. All of this could be delayed until use. That's my suspicion for the speed difference. I'll keep this in mind for the next release and have a look at your code for a new method to InstallMgr. Thank you!

Troy

On September 30, 2019 5:13:23 AM MST, contact at tklein.info wrote:
>Hi,
>
>in my Sword backend (node-sword-interface) I've now implemented a new  
>method isModuleAvailableInRepo:
>
>bool isModuleAvailableInRepo(std::string moduleName, std::string  
>repoName="all");
>
>This method avoids the creation of a SWMgr instance (which is slow for 
>
>large repos), but just quickly goes through the .conf files of  
>sword::InstallSource.localShadow. It then extracts the module ids  
>(from the first line of each .conf file) and checks whether the given  
>module id is found in the overall list.
>
>On my Windows tablet this operation completes in 0.5s, whereas  
>creation of SWMgr + going through the ModMap takes 5.8s.
>When executed for a bunch of modules at start-up this change  
>significantly reduces start-up time of Ezra Project (especially on  
>slower computers).
>
>The question remains whether a similar function would make sense  
>directly in the SWORD engine.
>
>Best regards,
>Tobias
>
>Some links for you to see how this was implemented:
>
>bool isModuleAvailableInRepo(std::string moduleName, std::string  
>repoName="all");
>https://github.com/tobias-klein/node-sword-interface/blob/0.68.0/sword_facade.cpp#L483
>
>std::vector<std::string> getRepoModuleIds(std::string repoName);
>https://github.com/tobias-klein/node-sword-interface/blob/0.68.0/sword_facade.cpp#L301
>
>std::string getModuleIdFromFile(std::string moduleFileName);
>https://github.com/tobias-klein/node-sword-interface/blob/0.68.0/sword_facade.cpp#L276
>
>Quoting contact at tklein.info:
>
>> To correct what I wrote earlier ...
>> It seems to be the construction of SWMgr that takes some time, in my
>> case InstallSource::getMgr().
>> The actual iteration over the ModMap is fast.
>>
>> Best regards,
>> Tobias
>>
>> Quoting contact at tklein.info:
>>
>>> Hi,
>>>
>>> For now I'm thinking to store this information (Repository of a
>>> module) in my own application-level persistence layer.
>>>
>>> However, I'm still wondering about the performance of
>>> InstallMgr::getModuleStatus.
>>> It's not a big issue on current desktop PCs with fast SSDs. On my
>>> Windows tablet, though - it takes ~3s to call this function using
>>> the "eBible.org" repository. When calling it for all repositories to
>>> gather a complete list of modules, it takes ~6s on my Windows
>tablet.
>>>
>>> I guess the function walks through all the module files and a long
>>> list of files just takes its toll ... Is there a possibility for a
>>> performance improvement, though?
>>>
>>> Best regards,
>>> Tobias
>>>
>>> Quoting contact at tklein.info:
>>>
>>>> Hi Troy!
>>>>
>>>>> I guess InstallMgr could add a line into the .conf when
>>>>> installing: InstalledFrom, but that could be a side load, a local
>>>>> repo, another SWORD application install on a local lan, or one of
>>>>> our remote sources which might not still host the module.
>>>>
>>>> That would be helpful!
>>>>
>>>> I have 2 use cases:
>>>>
>>>> 1) Show the repository that a module was installed from as part of
>>>> other module meta information ("Module About Dialog")
>>>> 2) Check, which repository a certain module is available from in
>>>> the context of a "sync feature". In Ezra Project, I'm storing a
>>>> list of installed modules and this list can be synced between
>>>> different computers. My expectation as a user is that I want to
>>>> work with the same set of modules whichever of my computers I'm
>>>> using. The "sync feature" in Ezra Project basically checks the
>>>> "list of installed modules" (from Ezra Project synced
>>>> configuration, synced via Dropbox or similar service) and then
>>>> automatically installs missing modules if there are any modules in
>>>> the list that have not been installed locally yet. For this to work
>>>> efficiently it would be good to know the source repository of a
>>>> module, because otherwise I have to search all repositories for it
>>>> (That's what I'm currently doing and the performance is the issue).
>>>>
>>>> Best regards,
>>>> Tobias
>>>>
>>>>
>>>> _______________________________________________
>>>> sword-devel mailing list: sword-devel at crosswire.org
>>>> http://www.crosswire.org/mailman/listinfo/sword-devel
>>>> Instructions to unsubscribe/change your settings at above page
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> sword-devel mailing list: sword-devel at crosswire.org
>>> http://www.crosswire.org/mailman/listinfo/sword-devel
>>> Instructions to unsubscribe/change your settings at above page
>>
>>
>>
>>
>> _______________________________________________
>> sword-devel mailing list: sword-devel at crosswire.org
>> http://www.crosswire.org/mailman/listinfo/sword-devel
>> Instructions to unsubscribe/change your settings at above page
>
>
>_______________________________________________
>sword-devel mailing list: sword-devel at crosswire.org
>http://www.crosswire.org/mailman/listinfo/sword-devel
>Instructions to unsubscribe/change your settings at above page

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20190930/871a97b7/attachment.html>