[sword-devel] InstallMgr

[DM Smith]

Chris Little wrote:
>
>
> Troy A. Griffitts wrote:
>> I would like to add new officially supported methods, like adding:
>>
>> o  Installation of modules from another existing SWORD library 
>> installation via HTTP  ( e.g., 
>> http://crosswire.org/ftpmirror/pub/sword/raw )
>>
>> A skeleton class for this method was already added in 1.6.0 so we can 
>> shoot to implement this in the 1.6.x branch without changing the API 
>> interface.
>>
>> ======================
>>
>> Aside from the officially supported methods of installation, there 
>> are caching methods we implement to assist in optimizing 
>> installation, e.g. if we find a mods.d.tar.gz file at the root of the 
>> 'other' SWORD library installation from where we're installing, we'll 
>> copy and untargz this to discover which modules are available, 
>> instead of copying each mods.d/*.conf file one at a time.
>>
>> **** BUT THE IMPORTANT THING HERE in my mind, is that, while we have 
>> caching mechanisms in place, and while we might even add more 
>> (hypothetically, a zips/ folder at the root of the 'other' SWORD 
>> library installation where we can check for a pre-zipped module 
>> archive so we can avoid downloading module data files and images one 
>> at a time, and other methods)...
>>
>> .... THE VERY IMPORTANT THING NOT TO MISS HERE....  (have I focused 
>> the attention of skimmers yet? :) )
>>
>>     ... is that these are EXACTLY THAT: CACHING/OPTIMIZATION 
>> features.  *** NOT *** required features to make a valid repository 
>> SWORD module repository.
>>
>> A frontend should NOT depend on these features being available.  If 
>> they exist, use them-- great!  If not, please support the minimum 
>> common installation features outlined above.
>
> Specifically with reference to the way JSword deals with module 
> downloading: I really look forward to seeing a shift from use of the 
> cached ZIPs to using the FTP site (via HTTP if necessary).
Maybe someone can help a bit. The problem we ran into was that there was 
no reliable way to download a folder via http and there was no way to 
reliably know the content of the folder.

If there were a manifest of the files in a module that would work and 
then http of all the parts would be just fine. I have a script in my bin 
directory (~dmsmith/bin/listing.pl) on the server that builds such a 
manifest. I would love it if we can make it a part of the creation of 
mods.d.tar.gz by building it daily into the mods.d folder on the server.
>
> Greater than 99% of the time, the ZIPs will be fine, but there will be 
> occasional problems. One is simply that their use requires an extra 
> step, which very occasionally fails. We had some problems after the 
> server upgrade, where new ZIPs couldn't be created, which would have 
> affected JSword users and went unnoticed until someone downloading one 
> of the affected ZIPs from the webpage noticed and alerted us.
>
> The (to my mind) greater issue is simply that it this system requires 
> someone else to have already downloaded the most recent version of a 
> module. The ZIPs are only created by the download servlet. Assuming 
> our hope that people migrate to InstallMgrs comes true, fewer and 
> fewer people will download ZIPs from the webpage, making it 
> progressively less likely that the correct ZIP exists. 

Yes that is exactly the problem that JSword has.

One of the things that Troy suggested is that FTP of the zip, can 
generate the zip if it does not exist. Such a mechanism would be useful 
to SWORD as it would increase the reliability of transfer. If you look 
at the logs you'll see that sometimes a part fails to download probably 
leaving a partial or corrupt module.

Such a mechanism would leave the problem the same as it is today, but it 
would be minimized.

DM