[sword-devel] HTTPS remote SWORD repo access

Sat Mar 1 11:09:39 EST 2025

I'm fairly sure the Plaform doesn't provide it. You would need to build the
library and include it with your app. I was under the impression most apps
already did that.

--Greg

On Sat, Mar 1, 2025, 1:35 AM Tobias Klein <contact at tklein.info> wrote:

> Hi Greg,
>
> when configuring SWORD for my Android build (based on CMake) I currently
> do not see CURL being detected automatically and therefore there is no CURL
> in my Android SWORD build:
>
> -- Configuring your system to build libsword.
> -- SWORD Version 1009000000
> --
> -- SEARCHING FOR SYTEM PACKAGES
> -- System regex.h: Yes
> --
> -- CONFIGURING SOURCE LIST
> -- ZLib: system
> /opt/Android/SDK/ndk/r27c/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android/21/libz.so
> -- bzip2: no
> -- xz: no
> *-- cURL: no*
> -- CLucene: no
> -- PkgConfig: yes
> -- ICU: no
> -- Regex.h: system
> /opt/Android/SDK/ndk/r27c/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include
> -- Building Static library.
> -- Setting link libraries to
> /opt/Android/SDK/ndk/r27c/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android/21/libz.so
>
> Best regards,
> Tobias
> On 2/28/25 18:25, Greg Hellings wrote:
>
> This is all wonderful, but it's really just efficiency improvements. I've
> been testing locally with nginx, and the only setting I need is to set
> `autoindex on;` in the location that points to the Sword repo. Everything
> works fine, then. I have tested in the past with Apache (although it has
> been a very long time since I've touched it) and it works fine there.
>
> There's nothing currently stopping us from swapping to HTTP or HTTPS
> client-based access anywhere that is built against curl. Which is every
> Linux distro I've seen, MacOS CLI distributions, and the Xiphos Windows
> builds. Also, I'm pretty sure all the known mobile apps support it, as well.
>
> --Greg
>
> On Fri, Feb 28, 2025 at 10:05 AM Troy A. Griffitts <scribe at crosswire.org>
> wrote:
>
>> Hey guys,
>>
>> We did some work in the past to make this transition begin to happen
>> automatically without user actions.  I think there was an outline in here a
>> year or so back.  Basically, the path forward was:
>>
>> 1) to standardize a location in a SWORD repository for .zip package.
>> This has many benefits beyond helping us get to HTTPS everywhere.  It
>> conceptually allows reading directly from the zip file by the engine.  It
>> could facilitate the engine itself building the zip packages from an
>> exploded tree.  All transports, including the dominant FTP transport right
>> now could look to see if there is an available single .zip package for
>> download instead of downloading each individual file for a module during a
>> remote install, and I am sure more ideas people will imagine in the
>> future... but simply this step was to standardize the location of .zip
>> package which each represent a zip of a single package based at the root of
>> a sword module repo, e.g.,
>>
>> KJV.zip:
>>   mods.d/kjv.conf
>>   modules/texts/ztext/kjv/nt.bzs
>>   modules/texts/ztext/kjv/ot.bzs
>>   modules/texts/ztext/kjv/ot.bzv
>>   modules/texts/ztext/kjv/nt.bzv
>>   modules/texts/ztext/kjv/ot.bzz
>>   modules/texts/ztext/kjv/nt.bzz
>>
>> This location decided on was pretty straighforward, at the root of the
>> sword module repo, packages/
>>
>> You'll notice this if you have a look at the crosswire repo here:
>>
>> https://crosswire.org/ftpmirror/pub/sword/raw/
>>
>> The latest version of the engine does use this now and tries to downoad a
>> package if available, before it falls back to the previous behavior of
>> downloading each individual file.
>>
>> 2) Support 'chained' transport providers in the engine, allowing multiple
>> transports to the same repository.  This has been completed and the first
>> use of this concept is the recognition of a new HTTPSPackagePreference
>> option in installmgr's configuration file.
>>
>> The value of the property is:  existing repo name|hostname|repository_path
>>
>> You can see this added for the CrossWire repository in the master repo
>> list registry at:
>>
>> https://crosswire.org/ftpmirror/pub/sword/masterRepoList.conf
>>
>> HTTPSPackagePreference=CrossWire|crosswire.org|/ftpmirror/pub/sword/raw
>> FTPSource=CrossWire|ftp.crosswire.org|/pub/sword/raw
>>
>> This tells installmgr that if a user requests a package install from
>> "CrossWire" then chain 2 transport provider attempts together and prefer
>> the HTTPS transport first.
>>
>>
>> The hope is that this path forward will see the vast amount of SWORD
>> frontends begin to start using HTTPS instead of using FTP, to facilitate a
>> smooth transition from FTP to HTTPS.  The idea is that if a repo owner sets
>> everything up correctly, and a user has the latest software and latest
>> configuration information from our registry of known SWORD repos (
>> https://crosswire.org/ftpmirror/pub/sword/masterRepoList.conf ),
>> everything will flow over HTTPS instead of FTP and once we see FTP traffic
>> drop to very low levels, a repo owner could choose to discontinue support
>> for FTP if they desire.
>> Michael, I am glad you have your server back up and running well again.
>> Thank you for all that you do to distribute the Word of God to a needing
>> world.
>>
>> Troy
>>
>>
>> On 2/27/25 9:45 PM, Michael Johnson wrote:
>>
>> Hi David,
>>
>> Yes, under eBible.org, where it says "HTTP access", it should say "HTTPS
>> and HTTP access". Also, any repository that also supports HTTPS and/or HTTP
>> access should explicitly say so. As Greg mentioned, mobile often blocks
>> FTP. FTP may not be dead, yet, but it is in hospice care. HTTP likewise can
>> cause problems with mobile platforms, where the major app stores now
>> require use of HTTPS instead of insecure HTTP in most cases. I still
>> support insecure HTTP connections to eBible.org except for at
>> shop.eBible.org and https://eBible.org/cgi-bin/contact.cgi but there may
>> come a day when HTTP goes away, too.
>> On 2/26/25 22:03, David Haslam wrote:
>>
>> Hi Michael,
>>
>> *Does anything need to be changed here?*
>> Official and Affiliated Module Repositories - CrossWire Bible Society
>> <https://wiki.crosswire.org/Official_and_Affiliated_Module_Repositories#eBible.org>
>>
>> *Everyone*: Except for the note under *AndBibleExtra*, there's no
>> mention of *https* !
>> *Do we need to make any wider changes to be future proof?*
>>
>> Best regards,
>>
>> David
>>
>> Sent with Proton Mail <https://proton.me/mail/home> secure email.
>>
>> On Thursday, February 27th, 2025 at 7:13 AM, Michael Johnson
>> <kahunapule at eBible.org> <kahunapule at eBible.org> wrote:
>>
>>
>> On 2/26/25 17:12, Greg Hellings wrote:
>>
>>
>>
>> On Wed, Feb 26, 2025, 7:33 PM Kahunapule Michael Johnson <
>> kahunapule at ebible.org> wrote:
>>
>>> Greetings from Maui!
>>>
>>> tldr: upgrade your Sword apps to always use https instead of http or ftp
>>> to access repositories ASAP.
>>>
>>
>> While technically any network acess other than anonymous FTP support is
>> optionally supported only with a build dep, in reality there is no need to
>> support anything other than HTTPS. Every Linux distribution, and Windows
>> build of note has libcurl, the Brew version is also built against it, and
>> the HTTP(S) support was added because mobile often blocks FTP.
>>
>> So you're basically completely safe.
>>
>> Awesome!
>>
>>
>>
>>
>>> As many of you are probably aware, the last week was not a model of
>>> reliability for the eBible.org repository, or for the rest of the
>>> eBible.org site. On the 19th of February, the eBible.org server hardware
>>> failed. Exactly what failure, I don't know, because it was in a data center
>>> over 4,000 miles from my house. I just knew that it wouldn't talk to me in
>>> any of the 3 ways I can normally access the leased dedicated server. No
>>> worries, because I have a fast backup, right? I allocated a new dedicated
>>> server
>>> from the same company (Ionos) and attempted to restore from a backup.
>>> That failed with about 80 error messages. Next plan: restore from a mirror
>>> image of the server in my home office. That actually worked, but it took
>>> more than 3 days to get all of the data there (about 300 GBytes), plus time
>>> to get all of the configuration right. In the mean time, my other leased
>>> server (the one that didn't crash, hosting 24 other sites) gave early
>>> warning signs that it was not going to be in service much longer. Then
>>> everything worked except that I forgot a couple of tweaks I had to do to
>>> make the ftp server compatible with Sword. I fixed that, and things were
>>> still not OK. EBible.org availability kept going up and down like a yo-yo,
>>> mostly because the remote control software I was using was not designed to
>>> handle multiple IP addresses per server and anonymous ftp sites. Also, the
>>> cost of allocating multiple IP v4 addresses has gone up. Anonymous ftp is
>>> pretty much obsolete. I will be dropping it, but slowly.
>>>
>>
>> A Herculean effort, but I'm glad for you that your recovery was
>> successful! I'm curious why you need 4 separate addresses? What is the
>> need, there?
>>
>> So far, I have been using Plesk to set up virtual hosts. I have 25 sites
>> (and some aliases for those), some of which are much more important than
>> others. Plesk lets me share one IP address with all sites except any site
>> that has an anonymous ftp service associated with it. The only site I have
>> that has an anonymous ftp service associated with it, of course, is the
>> ftp.eBible.org Sword repository. So I had to assign 2 IP version 4
>> addresses to the server. For a long time, I was running 2 servers with
>> every site on them for redundancy. I had stopped doing that because the
>> sites grew too large for one of the servers I was renting, and I thought I
>> had a workable fast backup/restore plan, unlike when I had extremely slow
>> and expensive Internet in Papua New Guinea. (I have some serious space in
>> audio and video Bibles.) So that is 2 servers x 2 IP addresses = 4 IP
>> addresses. But that configuration was unstable, so I went to just one IP
>> address per server by fighting my old ally, Plesk, using manual ProFTP
>> configuration (and a cron job to slap my configuration back whenever Plesk
>> rewrites it). That is not a really good long-term solution, though.
>>
>> ...
>> Would you like a hand building up some DR or deployment automation so you
>> can avoid needing to remember settings? IT automation is one of my primary
>> skillsets, so if you'd like any sort of help setting it up, let me know.
>> For instance, it's not too hard to put together automation scripts to run
>> on a provisioned box to stand up the web server, ftp server, etc so that
>> you don't need to manually edit files and the like.
>>
>> That would be useful. That could be a way to escape my dependence on and
>> fight with Plesk.
>>
>>
>> Alternatively, have you considered an alternative way to host the data?
>> You could probably build a Container image with all the files in it and
>> host that on something like Amazon Container Service or any of the many
>> cloud Kubernetes hosts around. A container image would also make it easy
>> for someone to grab the whole collection and make it available in an
>> offline context the way they can with the old CD images Troy used to
>> distribute.
>>
>> I have looked at alternatives in the past, but it may be worth looking
>> again. When I last looked, AWS was more expensive at my traffic levels and
>> site counts than using a rented dedicated server. Another alternative might
>> be hosting at my house when (if?) Hawaiian Telephone makes good on its
>> promise to bring fiber Internet to my neighborhood. (It is actually
>> available about a half mile away, right now, but I haven't seen them
>> working on it around here.)
>>
>>
>> Or even put the files into an object storage container if you're
>> dedicated to eliminating FTP access eventually. With just a small shell
>> script you can push the needed files and their indexes into an S3, Ceph,
>> etc object storage service and then you wouldn't need to run a dedicated
>> server with them to manage uptime. All of those services offer ways to
>> expose the files over HTTPS.
>>
>> As I said on Facebook, I'm happy to lend a hand if there's anything I can
>> do to help smooth your infrastructure! I can even host an emergency mirror
>> if need be, as I have pretty reliable Internet and electric when my
>> neighbors don't drive into the electric poles. This year I'm dedicating
>> some of my time to working on home electric backups!
>>
>> Thank you, Greg. I may take you up on that...
>> --
>>
>> Peace,
>> *Michael Johnson*
>> * 26 HIWALANI LOOP • MAKAWAO HI 96768-8747
>> <https://www.google.com/maps/search/26+HIWALANI+LOOP+%E2%80%A2%0D%0A++++++++++++++++++++++++++++MAKAWAO+HI+96768-8747+%E2%80%A2+USA?entry=gmail&source=g>* •
>> USA
>> <https://www.google.com/maps/search/26+HIWALANI+LOOP+%E2%80%A2%0D%0A++++++++++++++++++++++++++++MAKAWAO+HI+96768-8747+%E2%80%A2+USA?entry=gmail&source=g>
>> mljohnson.org • eBible.org • WorldEnglish.Bible • PNG.Bible
>> Signal/Telegram/WhatsApp/Telephone: +1 808-333-6921
>> Skype: kahunapule • Telegram: @kahunapule • Facebook: fb.me/kahunapule
>> <https://www.facebook.com/kahunapule>
>>
>>
>>
>> _______________________________________________
>> sword-devel mailing list: sword-devel at crosswire.orghttp://crosswire.org/mailman/listinfo/sword-devel
>> Instructions to unsubscribe/change your settings at above page
>>
>> --
>>
>> Peace,
>> *Michael Johnson*
>> * 26 HIWALANI LOOP • MAKAWAO HI 96768-8747
>> <https://www.google.com/maps/search/26+HIWALANI+LOOP+%E2%80%A2+MAKAWAO+HI%0D%0A++++++++++++++++++++++96768-8747+%E2%80%A2+USA?entry=gmail&source=g>* •
>> USA
>> <https://www.google.com/maps/search/26+HIWALANI+LOOP+%E2%80%A2+MAKAWAO+HI%0D%0A++++++++++++++++++++++96768-8747+%E2%80%A2+USA?entry=gmail&source=g>
>> mljohnson.org • eBible.org • WorldEnglish.Bible • PNG.Bible
>> Signal/Telegram/WhatsApp/Telephone: +1 808-333-6921
>> Skype: kahunapule • Telegram: @kahunapule • Facebook: fb.me/kahunapule
>> <https://www.facebook.com/kahunapule>
>>
>> _______________________________________________
>> sword-devel mailing list: sword-devel at crosswire.orghttp://crosswire.org/mailman/listinfo/sword-devel
>> Instructions to unsubscribe/change your settings at above page
>>
>> _______________________________________________
>> sword-devel mailing list: sword-devel at crosswire.org
>> http://crosswire.org/mailman/listinfo/sword-devel
>> Instructions to unsubscribe/change your settings at above page
>>
>
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.orghttp://crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page
>
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://crosswire.org/pipermail/sword-devel/attachments/20250301/0948dc00/attachment-0001.htm>