[sword-devel] HTTPS Transport

[Tobias Klein]

Hi Troy,

I have just done some testing with the latest changes in sword (SVN Rev. 
3894).

I have not observed any issues on Linux. However, on Windows 10 the 
following happens:

Repository data does not seem to be returned in the same way as before 
(I was previously on SVN Rev. 3873, which is from November 7 2021).
E.g. a helper function in node-sword-interface that returns the number 
of modules per repository suddenly returns 0 modules for the CrossWire 
repository.

Is there any explanation for that?
I did not change anything in node-sword-interface, but only updated the 
sword dependency to the latest version 3894.

Best regards,
Tobias

On 12/16/23 11:11 PM, Troy A. Griffitts wrote:
> Hey guys,
>
> For a while now, we've been working on a new mechanism for allowing a 
> remote module installation repository to use only HTTPS traffic to 
> supply their modules, if they wish.  A little history and how things 
> work in the released SWORD engine and what we have in SVN trunk now... 
> (skip to === NEW === if you don't care about the history and why)
>
> From the beginning of SWORD we have had as a core value the simple 
> enabling of Bible distribution.  The very first versions of our 
> installer could use as an installation source any working installation 
> of a SWORD library.  E.g., user 1 sets up Xiphos and manually unzips 
> 100 Bibles, commentaries, lexicons, dictionaries, etc. for use with 
> Xiphos.  User 1 can then share (network drive, USB stick, FTP) their 
> installation folder where they have unzipped all the data for their 
> library, and user 2 can come along and install Bibletime or Xiphos or 
> any other SWORD application and point their installer to this shared 
> location and install from there any Bible, commentary, lexicon, 
> dictionary, etc., from user 1's working installation.  Then user 2 can 
> travel to their school in Zimbabwe, plug into their school's network 
> and share their data folder from their working SWORD application and 
> students on that network can install Bibles from them.
>
> None of this has changed.  This is still a core value and still works 
> with all the same mechanisms.
>
> Over the years, we have added on top of this behavior optional 
> optimizations for remote repositories.  For example, instead of 
> looking for the mods.d/ folder and downloading individually all the 
> .conf files found there to present to a user a list of which Bibles, 
> commentaries, etc. are available, we first look for a mods.d.tar.gz 
> file with all the .conf files bundled into a single download.  This 
> saves a lot of time working with large remote repositories.  If we 
> don't find this file, we still fallback to downloading the individual 
> files.  We don't want a failure to happen when passing along Bibles if 
> this optimization is not in place, but we do want to speed things up 
> if the manager of the remote repository knows how to manage their 
> repository optimally and is willing to do this extra work to keep this 
> file in place and up to date.
>
> Over the years, the FTP protocol, which SWORD has primarily used for 
> remote module installation over the internet, has seen data providers 
> block traffic due to its unencrypted nature.  Being Bible 
> distributors, in most cases we don't care if anyone snoops on our data 
> packets.  Generally, again in most cases, we WANT people to snoop.  We 
> don't require user / password for distribution so the security issues 
> involved in sending those in plain text don't apply to our 
> applications, generally.  Now, of course there are scenarios which 
> people may wish to distribute Bibles without public knowledge, and 
> sometimes users may wish to protect their modules with username / 
> password credentials, and for this we have historically also supported 
> SFTP.
>
> One driving factor for the latest improvement described below is that 
> we have found some mobile data providers blocking FTP traffic on their 
> network, requiring our users to get to a WiFi connection before they 
> can install Bibles, etc.
>
>
> === NEW ===
>
> In SVN trunk there is code to handle a new facility for remote module 
> installation.  Like the optional optimization with the mods.d.tar.gz 
> file, this new mechanism is optional.  All will work as before if 
> nothing is changed.
>
> Fully enabling the new mechanism consists of 4 steps:
>
> 1. assuring https access to the root folder of your module repository.
>
> 2. mods.d.tar.gz is required for this mechanism to be successful.
>
> 3. module.zip files must be available from a packages/ folder at the 
> root of your module repository folder.  These .zip files have been 
> historically required for JSword-based apps because JSword does not 
> yet know how to install from an working installation of modules, as 
> described at the beginning of this email.  So because many of our 
> repository maintainers support JSword, this step might be as simple as 
> creating a packages/ -> symbolic link to your JSword .zip module files 
> folder, if you are already maintaining zip files.
>
> 4. adding an HTTPSPackagePreference entry into our master repository 
> list telling us the server, and path on that server, to find your 
> repository with https
>
>
> The main CrossWire repository now has this mechanism enabled and can 
> be used as a reference to test frontends and can be used as an example 
> for remote module installation repository maintainers.
>
> For CrossWire main, #1 is available here, and at the root of this 
> location you can also see #2 mods.d.tar.gz and #3 packages/ :
>
> https://crosswire.org/ftpmirror/pub/sword/raw/
>
> Step 4 can be seen in our master repo list, the first entry under 
> [Repos] here:
>
> https://crosswire.org/ftpmirror/pub/sword/masterRepoList.conf
>
>
> Any SWORD app compiled against SVN trunk should now only use HTTPS 
> when installing modules from CrossWire main.
>
> May I ask to please test and give feedback?  Thank you for all the 
> advice and encouragement to add this functionality.  I pray this 
> enhances our ability to distribute more Bibles to those who have yet 
> to hear the Good New of Jesus Christ and to be used by Him to build up 
> His church,
>
> Troy
>
>
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page