[sword-devel] HTTP Module Install Preference

[Troy A. Griffitts]

Hey guys,

I have some pretty experimental code in trunk which I'd love for you to 
try out.
In general, it is an attempt to update InstallMgr to prefer HTTPS zip 
package downloads if HTTPS is a supported protocol on the platform and 
they exists on the remote source.  Details follow:

1) new unzip expansion support in the engine:

we've wanted, for a while now, to support zip archive expansion; 
historically, we've supported tar.gz expansion.  My issue in the past is 
that zlib only supports the decompression action, not actually reading 
an archive of files and expanding them. zlib does include a contrib/ 
folder which has enough functionality to provide this.  I've finally 
waded through it all and it appears it might be cross-platform enough 
for us to compile on our targets. The new files are:

src/modules/common/ioapi.c
src/modules/common/unzip.c
include/internal/unzip/ioapi.h
include/internal/unzip/unzip.h

with a new static method on our class ZipCompress::unZip(const char 
*sourceZipPath, const char *destPath);

I've done my best to try to add the proper lines to our cmake system.  
I've only tested our autotools build system (autotools users, be sure to 
./autogen.sh  and ./usrinst.sh to refresh and pick up the new files)

2) InstallMgr will now look for a cached copy of the module at 
[Source]/packages/ModName.zip

Just like InstallMgr has looked for a cached copy of mods.d.tar.gz at 
the source root, it now looks for a cached copy of the moduled zipped up 
under packages/.  If it finds the zip file, it will download that and 
unzip instead of trying to download file by file.
To test, I've added a packages/ symlink for the CrossWire repo pointing 
to our JSword zip bundles.  You can see the packages folder here:

https://crosswire.org/ftpmirror/pub/sword/raw/

The idea is that any repo which wants to make these package zips 
available (many do anyway because JSword has always required it) can now 
place them in a standard location at the root of their repo under a 
folder called packages/

3) InstallMgr now supports a concept called "chained sources"

This allows a repo to specify multiple transport configurations (e.g., 
FTP and HTTP). Sources are chained if multiple configuration entries 
exist for the same repo name.  This isn't fully implemented yes, but it 
is finished enough to support the next item, #4, below.


4) InstallMgr now supports a new configuration line in the [Repos] 
section HTTPSPackagePreference.

This allows a repo to say that it prefers HTTPS Packages, if possible.  
This will cause an HTTPS install source to be created as the first 
attempted installation route, but only for caches packages, if the 
platform doesn't have HTTPS support, or the cached package doesn't 
exist, it will move on to the next chained source (e.g., fall back to FTP).

To test, I've added an entry for CrossWire which you can see in our 
master repo list:

https://crosswire.org/ftpmirror/pub/sword/masterRepoList.conf


5) InstallMgr now first tries to sync its configuration from our master 
repo source list via HTTPS and then falls back to FTP if unsuccessful.


This all let's us use our HTTP remote transport we've had in the engine 
for a while, but avoids the problem of parsing various http folder 
listing formats.  It also speeds up FTP downloads if the cached zip 
archive file exists on the server.
The new HTTPSPackagePreference configuration line will be ignored by old 
SWORD software, and if the packages/ folder with zips doesn't exist on 
the remote source, all falls back to the previous method of installation.


The bottom line is that I believe one can now:

cd sword/utilities/
./installmgr -sc
./installmgr -r CrossWire
./installmgr -ri CrossWire KJV

completely over HTTPS and without downloading individual files.


Let me know if you have a chance to try it out.

Happy Memorial Day!  Our Lord's best for you,

Troy