[sword-devel] HTTPS Transport

Sun Jan 21 15:28:11 EST 2024

Hi Troy,

my Windows build of SWORD has already been including CURL (currently used tag: curl-7_68_0).

Here is what CMake outputs (extract from build log on GitHub).

2024-01-21T14:39:13.3799672Z -- CONFIGURING SOURCE LIST
2024-01-21T14:39:13.3800942Z -- ZLib: system D:\a\sword-build-win32\sword-build-win32\dist\lib\zlib.lib
2024-01-21T14:39:13.3802219Z -- bzip2: system D:/a/sword-build-win32/sword-build-win32/dist/lib/libbz2.lib
2024-01-21T14:39:13.3803606Z -- xz: system D:/a/sword-build-win32/sword-build-win32/dist/lib/liblzma.lib
2024-01-21T14:39:13.3804732Z -- cURL: system D:\a\sword-build-win32\sword-build-win32\dist\lib\libcurl_imp.lib and D:/a/sword-build-win32/sword-build-win32/curl/include
2024-01-21T14:39:13.3805695Z -- CLucene: no
2024-01-21T14:39:13.3806800Z -- PkgConfig: yes
2024-01-21T14:39:13.3807142Z -- ICU: no
2024-01-21T14:39:13.3807647Z -- Regex.h: internal
2024-01-21T14:39:13.3814178Z -- Building Shared library.
2024-01-21T14:39:13.3818932Z -- Building Static library.
2024-01-21T14:39:13.3821940Z -- Setting SOVERSION to 1.9.0
2024-01-21T14:39:13.3972004Z -- Setting link libraries to D:\a\sword-build-win32\sword-build-win32\dist\lib\zlib.lib;D:/a/sword-build-win32/sword-build-win32/dist/lib/libbz2.lib;D:/a/sword-build-win32/sword-build-win32/dist/lib/liblzma.lib;D:\a\sword-build-win32\sword-build-win32\dist\lib\libcurl_imp.lib

I am attaching the full build log from GitHub for reference, not sure if it helps.

In a moment I will share screenshots from Ezra Bible App (before and after) with the available repository list – maybe that helps.

Best regards,
Tobias

From: Troy A. Griffitts
Sent: Sonntag, 21. Januar 2024 19:57
To: SWORD Developers' Collaboration Forum; Tobias Klein
Subject: Re: [sword-devel] HTTPS Transport

Hi Tobias,

Thank you so much for testing. My guess is that the difference is likely due to the way SWORD is compiled-- which dependencies SWORD is using. My speculation is that libcurl is probably available on Linux and your Windows build probably isn't using this library.

Can you let me know if SWORD is compiled with libcurl on your Windows build? I will try to test a Linux build without libcurl. I tried to make the new feature fallback to previous behavior if it couldn't use the new feature successfully. I obviously have failed in my backward compatibility effort. 😊 I will try to get it right.

Troy

On January 21, 2024 09:44:12 MST, Tobias Klein <contact at tklein.info> wrote:
Hi Troy,

I have just done some testing with the latest changes in sword (SVN Rev. 3894).

I have not observed any issues on Linux. However, on Windows 10 the following happens:

Repository data does not seem to be returned in the same way as before (I was previously on SVN Rev. 3873, which is from November 7 2021).
E.g. a helper function in node-sword-interface that returns the number of modules per repository suddenly returns 0 modules for the CrossWire repository.

Is there any explanation for that?
I did not change anything in node-sword-interface, but only updated the sword dependency to the latest version 3894.

Best regards,
Tobias

On 12/16/23 11:11 PM, Troy A. Griffitts wrote:
Hey guys,

For a while now, we've been working on a new mechanism for allowing a remote module installation repository to use only HTTPS traffic to supply their modules, if they wish.  A little history and how things work in the released SWORD engine and what we have in SVN trunk now... (skip to === NEW === if you don't care about the history and why)

>From the beginning of SWORD we have had as a core value the simple enabling of Bible distribution.  The very first versions of our installer could use as an installation source any working installation of a SWORD library.  E.g., user 1 sets up Xiphos and manually unzips 100 Bibles, commentaries, lexicons, dictionaries, etc. for use with Xiphos.  User 1 can then share (network drive, USB stick, FTP) their installation folder where they have unzipped all the data for their library, and user 2 can come along and install Bibletime or Xiphos or any other SWORD application and point their installer to this shared location and install from there any Bible, commentary, lexicon, dictionary, etc., from user 1's working installation.  Then user 2 can travel to their school in Zimbabwe, plug into their school's network and share their data folder from their working SWORD application and students on that network can install Bibles from them.

None of this has changed.  This is still a core value and still works with all the same mechanisms.

Over the years, we have added on top of this behavior optional optimizations for remote repositories.  For example, instead of looking for the mods.d/ folder and downloading individually all the .conf files found there to present to a user a list of which Bibles, commentaries, etc. are available, we first look for a mods.d.tar.gz file with all the .conf files bundled into a single download.  This saves a lot of time working with large remote repositories.  If we don't find this file, we still fallback to downloading the individual files.  We don't want a failure to happen when passing along Bibles if this optimization is not in place, but we do want to speed things up if the manager of the remote repository knows how to manage their repository optimally and is willing to do this extra work to keep this file in place and up to date.

Over the years, the FTP protocol, which SWORD has primarily used for remote module installation over the internet, has seen data providers block traffic due to its unencrypted nature.  Being Bible distributors, in most cases we don't care if anyone snoops on our data packets.  Generally, again in most cases, we WANT people to snoop.  We don't require user / password for distribution so the security issues involved in sending those in plain text don't apply to our applications, generally.  Now, of course there are scenarios which people may wish to distribute Bibles without public knowledge, and sometimes users may wish to protect their modules with username / password credentials, and for this we have historically also supported SFTP.

One driving factor for the latest improvement described below is that we have found some mobile data providers blocking FTP traffic on their network, requiring our users to get to a WiFi connection before they can install Bibles, etc.

=== NEW ===

In SVN trunk there is code to handle a new facility for remote module installation.  Like the optional optimization with the mods.d.tar.gz file, this new mechanism is optional.  All will work as before if nothing is changed.

Fully enabling the new mechanism consists of 4 steps:

1. assuring https access to the root folder of your module repository.

2. mods.d.tar.gz is required for this mechanism to be successful.

3. module.zip files must be available from a packages/ folder at the root of your module repository folder.  These .zip files have been historically required for JSword-based apps because JSword does not yet know how to install from an working installation of modules, as described at the beginning of this email.  So because many of our repository maintainers support JSword, this step might be as simple as creating a packages/ -> symbolic link to your JSword .zip module files folder, if you are already maintaining zip files.

4. adding an HTTPSPackagePreference entry into our master repository list telling us the server, and path on that server, to find your repository with https

The main CrossWire repository now has this mechanism enabled and can be used as a reference to test frontends and can be used as an example for remote module installation repository maintainers.

For CrossWire main, #1 is available here, and at the root of this location you can also see #2 mods.d.tar.gz and #3 packages/ :

https://crosswire.org/ftpmirror/pub/sword/raw/

Step 4 can be seen in our master repo list, the first entry under [Repos] here:

https://crosswire.org/ftpmirror/pub/sword/masterRepoList.conf

Any SWORD app compiled against SVN trunk should now only use HTTPS when installing modules from CrossWire main.

May I ask to please test and give feedback?  Thank you for all the advice and encouragement to add this functionality.  I pray this enhances our ability to distribute more Bibles to those who have yet to hear the Good New of Jesus Christ and to be used by Him to build up His church,

Troy

sword-devel mailing list: sword-devel at crosswire.org
http://crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page

sword-devel mailing list: sword-devel at crosswire.org
http://crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://crosswire.org/pipermail/sword-devel/attachments/20240121/02feee14/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build sword.zip
Type: application/zip
Size: 33568 bytes
Desc: not available
URL: <http://crosswire.org/pipermail/sword-devel/attachments/20240121/02feee14/attachment-0001.zip>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 14E0BCBB704A483E81FC2BC9A5D14A52.png
Type: image/png
Size: 135 bytes
Desc: not available
URL: <http://crosswire.org/pipermail/sword-devel/attachments/20240121/02feee14/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2330AFC246494E17881FC1D84DA79D86.png
Type: image/png
Size: 135 bytes
Desc: not available
URL: <http://crosswire.org/pipermail/sword-devel/attachments/20240121/02feee14/attachment-0003.png>