[sword-devel] InstallMgr

[Gregory Hellings]

On May 14, 2009, at 15:39, DM Smith <dmsmith at crosswire.org> wrote:

> Chris Little wrote:
>>
>>
>> Troy A. Griffitts wrote:
>>> I would like to add new officially supported methods, like adding:
>>>
>>> o  Installation of modules from another existing SWORD library  
>>> installation via HTTP  ( e.g., http://crosswire.org/ftpmirror/pub/sword/raw 
>>>  )
>>>
>>> A skeleton class for this method was already added in 1.6.0 so we  
>>> can shoot to implement this in the 1.6.x branch without changing  
>>> the API interface.
>>>
>>> ======================
>>>
>>> Aside from the officially supported methods of installation, there  
>>> are caching methods we implement to assist in optimizing  
>>> installation, e.g. if we find a mods.d.tar.gz file at the root of  
>>> the 'other' SWORD library installation from where we're  
>>> installing, we'll copy and untargz this to discover which modules  
>>> are available, instead of copying each mods.d/*.conf file one at a  
>>> time.
>>>
>>> **** BUT THE IMPORTANT THING HERE in my mind, is that, while we  
>>> have caching mechanisms in place, and while we might even add more  
>>> (hypothetically, a zips/ folder at the root of the 'other' SWORD  
>>> library installation where we can check for a pre-zipped module  
>>> archive so we can avoid downloading module data files and images  
>>> one at a time, and other methods)...
>>>
>>> .... THE VERY IMPORTANT THING NOT TO MISS HERE....  (have I  
>>> focused the attention of skimmers yet? :) )
>>>
>>>    ... is that these are EXACTLY THAT: CACHING/OPTIMIZATION  
>>> features.  *** NOT *** required features to make a valid  
>>> repository SWORD module repository.
>>>
>>> A frontend should NOT depend on these features being available.   
>>> If they exist, use them-- great!  If not, please support the  
>>> minimum common installation features outlined above.
>>
>> Specifically with reference to the way JSword deals with module  
>> downloading: I really look forward to seeing a shift from use of  
>> the cached ZIPs to using the FTP site (via HTTP if necessary).
> Maybe someone can help a bit. The problem we ran into was that there  
> was no reliable way to download a folder via http and there was no  
> way to reliably know the content of the folder.
>
> If there were a manifest of the files in a module that would work  
> and then http of all the parts would be just fine. I have a script  
> in my bin directory (~dmsmith/bin/listing.pl) on the server that  
> builds such a manifest. I would love it if we can make it a part of  
> the creation of mods.d.tar.gz by building it daily into the mods.d  
> folder on the server.

I've seen sites that had a file named ls-lR that's a recursive long  
file list of all the directories under it. Might that be used in a  
case like this?

--Greg

>
>>
>> Greater than 99% of the time, the ZIPs will be fine, but there will  
>> be occasional problems. One is simply that their use requires an  
>> extra step, which very occasionally fails. We had some problems  
>> after the server upgrade, where new ZIPs couldn't be created, which  
>> would have affected JSword users and went unnoticed until someone  
>> downloading one of the affected ZIPs from the webpage noticed and  
>> alerted us.
>>
>> The (to my mind) greater issue is simply that it this system  
>> requires someone else to have already downloaded the most recent  
>> version of a module. The ZIPs are only created by the download  
>> servlet. Assuming our hope that people migrate to InstallMgrs comes  
>> true, fewer and fewer people will download ZIPs from the webpage,  
>> making it progressively less likely that the correct ZIP exists.
>
> Yes that is exactly the problem that JSword has.
>
> One of the things that Troy suggested is that FTP of the zip, can  
> generate the zip if it does not exist. Such a mechanism would be  
> useful to SWORD as it would increase the reliability of transfer. If  
> you look at the logs you'll see that sometimes a part fails to  
> download probably leaving a partial or corrupt module.
>
> Such a mechanism would leave the problem the same as it is today,  
> but it would be minimized.
>
> DM
>
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page