[sword-devel] HTTPS Transport

[Troy A. Griffitts]

Hey guys,

For a while now, we've been working on a new mechanism for allowing a 
remote module installation repository to use only HTTPS traffic to 
supply their modules, if they wish.  A little history and how things 
work in the released SWORD engine and what we have in SVN trunk now... 
(skip to === NEW === if you don't care about the history and why)

 From the beginning of SWORD we have had as a core value the simple 
enabling of Bible distribution.  The very first versions of our 
installer could use as an installation source any working installation 
of a SWORD library.  E.g., user 1 sets up Xiphos and manually unzips 100 
Bibles, commentaries, lexicons, dictionaries, etc. for use with Xiphos.  
User 1 can then share (network drive, USB stick, FTP) their installation 
folder where they have unzipped all the data for their library, and user 
2 can come along and install Bibletime or Xiphos or any other SWORD 
application and point their installer to this shared location and 
install from there any Bible, commentary, lexicon, dictionary, etc., 
from user 1's working installation.  Then user 2 can travel to their 
school in Zimbabwe, plug into their school's network and share their 
data folder from their working SWORD application and students on that 
network can install Bibles from them.

None of this has changed.  This is still a core value and still works 
with all the same mechanisms.

Over the years, we have added on top of this behavior optional 
optimizations for remote repositories.  For example, instead of looking 
for the mods.d/ folder and downloading individually all the .conf files 
found there to present to a user a list of which Bibles, commentaries, 
etc. are available, we first look for a mods.d.tar.gz file with all the 
.conf files bundled into a single download.  This saves a lot of time 
working with large remote repositories.  If we don't find this file, we 
still fallback to downloading the individual files.  We don't want a 
failure to happen when passing along Bibles if this optimization is not 
in place, but we do want to speed things up if the manager of the remote 
repository knows how to manage their repository optimally and is willing 
to do this extra work to keep this file in place and up to date.

Over the years, the FTP protocol, which SWORD has primarily used for 
remote module installation over the internet, has seen data providers 
block traffic due to its unencrypted nature.  Being Bible distributors, 
in most cases we don't care if anyone snoops on our data packets.  
Generally, again in most cases, we WANT people to snoop.  We don't 
require user / password for distribution so the security issues involved 
in sending those in plain text don't apply to our applications, 
generally.  Now, of course there are scenarios which people may wish to 
distribute Bibles without public knowledge, and sometimes users may wish 
to protect their modules with username / password credentials, and for 
this we have historically also supported SFTP.

One driving factor for the latest improvement described below is that we 
have found some mobile data providers blocking FTP traffic on their 
network, requiring our users to get to a WiFi connection before they can 
install Bibles, etc.


=== NEW ===

In SVN trunk there is code to handle a new facility for remote module 
installation.  Like the optional optimization with the mods.d.tar.gz 
file, this new mechanism is optional.  All will work as before if 
nothing is changed.

Fully enabling the new mechanism consists of 4 steps:

1. assuring https access to the root folder of your module repository.

2. mods.d.tar.gz is required for this mechanism to be successful.

3. module.zip files must be available from a packages/ folder at the 
root of your module repository folder.  These .zip files have been 
historically required for JSword-based apps because JSword does not yet 
know how to install from an working installation of modules, as 
described at the beginning of this email.  So because many of our 
repository maintainers support JSword, this step might be as simple as 
creating a packages/ -> symbolic link to your JSword .zip module files 
folder, if you are already maintaining zip files.

4. adding an HTTPSPackagePreference entry into our master repository 
list telling us the server, and path on that server, to find your 
repository with https


The main CrossWire repository now has this mechanism enabled and can be 
used as a reference to test frontends and can be used as an example for 
remote module installation repository maintainers.

For CrossWire main, #1 is available here, and at the root of this 
location you can also see #2 mods.d.tar.gz and #3 packages/ :

https://crosswire.org/ftpmirror/pub/sword/raw/

Step 4 can be seen in our master repo list, the first entry under 
[Repos] here:

https://crosswire.org/ftpmirror/pub/sword/masterRepoList.conf


Any SWORD app compiled against SVN trunk should now only use HTTPS when 
installing modules from CrossWire main.

May I ask to please test and give feedback?  Thank you for all the 
advice and encouragement to add this functionality.  I pray this 
enhances our ability to distribute more Bibles to those who have yet to 
hear the Good New of Jesus Christ and to be used by Him to build up His 
church,

Troy