[sword-devel] Determining the repository a SWModule belongs to

Mon Sep 30 16:04:50 MST 2019

Hi Troy,

Thanks for the feedback! 😊
Yes, lazy loading in SWMgr would probably help. Looking forward to getting this improvement in place.

Regarding my original question:
With my newly developed method getRepoModuleIds I can now also rather quickly (< 1s) determine the repository a module belongs to (dynamically). The performance could still be optimized further, but it’s already noticably better than by going through SWMgr construction for all remote repositories.

I’m still planning to add the repo information for installed modules in my own persistence layer to further improve performance for the usecases mentioned below.

I have one more question.
Is there an example / easy way on how a SWModule object can be created independently of SWMgr? As of now SWMgr acts as a factory for SWModules, right? When working with specific modules, it may be enough to just create the objects directly, rather going through SWMgr?

Best regards,
Tobias

From: Troy A. Griffitts
Sent: Montag, 30. September 2019 16:29
To: SWORD Developers' Collaboration Forum
Subject: Re: [sword-devel] Determining the repository a SWModule belongs to

Hi Tobias,
Yes, thank you for all this. We certainly could optimize SWMgr c-tor for large repos. My guess is that the creation of every SWModule for large repos is taking the time. Many impls of SWModule check their data files on c-tor. SWMgr also adds all the filters to each SWModule. All of this could be delayed until use. That's my suspicion for the speed difference. I'll keep this in mind for the next release and have a look at your code for a new method to InstallMgr. Thank you!

Troy
On September 30, 2019 5:13:23 AM MST, contact at tklein.info wrote:
Hi,

in my Sword backend (node-sword-interface) I've now implemented a new  
method isModuleAvailableInRepo:

bool isModuleAvailableInRepo(std::string moduleName, std::string  
repoName="all");

This method avoids the creation of a SWMgr instance (which is slow for  
large repos), but just quickly goes through the .conf files of  
sword::InstallSource.localShadow. It then extracts the module ids  
(from the first line of each .conf file) and checks whether the given  
module id is found in the overall list.

On my Windows tablet this operation completes in 0.5s, whereas  
creation of SWMgr + going through the ModMap takes 5.8s.
When executed for a bunch of modules at start-up this change  
significantly reduces start-up time of Ezra Project (especially on  
slower computers).

The question remains whether a similar function would make sense  
directly in the SWORD engine.

Best regards,
Tobias

Some links for you to see how this was implemented:

bool isModuleAvailableInRepo(std::string moduleName, std::string  
repoName="all");
https://github.com/tobias-klein/node-sword-interface/blob/0.68.0/sword_facade.cpp#L483

std::vector<std::string> getRepoModuleIds(std::string repoName);
https://github.com/tobias-klein/node-sword-interface/blob/0.68.0/sword_facade.cpp#L301

std::string getModuleIdFromFile(std::string moduleFileName);
https://github.com/tobias-klein/node-sword-interface/blob/0.68.0/sword_facade.cpp#L276

Quoting contact at tklein.info:
 To correct what I wrote earlier ...
 It seems to be the construction of SWMgr that takes some time, in my
 case InstallSource::getMgr().
 The actual iteration over the ModMap is fast.

 Best regards,
 Tobias

 Quoting contact at tklein.info:
 Hi,

 For now I'm thinking to store this information (Repository of a
 module) in my own application-level persistence layer.

 However, I'm still wondering about the performance of
 InstallMgr::getModuleStatus.
 It's not a big issue on current desktop PCs with fast SSDs. On my
 Windows tablet, though - it takes ~3s to call this function using
 the "eBible.org" repository. When calling it for all repositories to
 gather a complete list of modules, it takes ~6s on my Windows tablet.

 I guess the function walks through all the module files and a long
 list of files just takes its toll ... Is there a possibility for a
 performance improvement, though?

 Best regards,
 Tobias

 Quoting contact at tklein.info:
 Hi Troy!
I guess InstallMgr could add a line into the .conf when
installing: InstalledFrom, but that could be a side load, a local
repo, another SWORD application install on a local lan, or one of
our remote sources which might not still host the module.

 That would be helpful!

 I have 2 use cases:

 1) Show the repository that a module was installed from as part of
 other module meta information ("Module About Dialog")
 2) Check, which repository a certain module is available from in
 the context of a "sync feature". In Ezra Project, I'm storing a
 list of installed modules and this list can be synced between
 different computers. My expectation as a user is that I want to
 work with the same set of modules whichever of my computers I'm
 using. The "sync feature" in Ezra Project basically checks the
 "list of installed modules" (from Ezra Project synced
 configuration, synced via Dropbox or similar service) and then
 automatically installs missing modules if there are any modules in
 the list that have not been installed locally yet. For this to work
 efficiently it would be good to know the source repository of a
 module, because otherwise I have to search all repositories for it
 (That's what I'm currently doing and the performance is the issue).

 Best regards,
 Tobias

 sword-devel mailing list: sword-devel at crosswire.org
 http://www.crosswire.org/mailman/listinfo/sword-devel
 Instructions to unsubscribe/change your settings at above page

 sword-devel mailing list: sword-devel at crosswire.org
 http://www.crosswire.org/mailman/listinfo/sword-devel
 Instructions to unsubscribe/change your settings at above page

 sword-devel mailing list: sword-devel at crosswire.org
 http://www.crosswire.org/mailman/listinfo/sword-devel
 Instructions to unsubscribe/change your settings at above page

sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20191001/b53688c0/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 962C6AFDCE97477A83BB9FCDFC9BDD59.png
Type: image/png
Size: 141 bytes
Desc: not available
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20191001/b53688c0/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: DFBBA2B03D5443D68B7FAA3FE0BF69E7.png
Type: image/png
Size: 141 bytes
Desc: not available
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20191001/b53688c0/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 57CD9E9A3839432AB9C159AC22B3154A.png
Type: image/png
Size: 141 bytes
Desc: not available
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20191001/b53688c0/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 3FDF8E96DDBE492AB246B95A390BDC3E.png
Type: image/png
Size: 140 bytes
Desc: not available
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20191001/b53688c0/attachment-0003.png>