[sword-devel] HTTPS remote SWORD repo access
Tobias Klein
contact at tklein.info
Sat Mar 1 02:34:34 EST 2025
Hi Greg,
when configuring SWORD for my Android build (based on CMake) I currently
do not see CURL being detected automatically and therefore there is no
CURL in my Android SWORD build:
-- Configuring your system to build libsword.
-- SWORD Version 1009000000
--
-- SEARCHING FOR SYTEM PACKAGES
-- System regex.h: Yes
--
-- CONFIGURING SOURCE LIST
-- ZLib: system
/opt/Android/SDK/ndk/r27c/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android/21/libz.so
-- bzip2: no
-- xz: no
*-- cURL: no*
-- CLucene: no
-- PkgConfig: yes
-- ICU: no
-- Regex.h: system
/opt/Android/SDK/ndk/r27c/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include
-- Building Static library.
-- Setting link libraries to
/opt/Android/SDK/ndk/r27c/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android/21/libz.so
Best regards,
Tobias
On 2/28/25 18:25, Greg Hellings wrote:
> This is all wonderful, but it's really just efficiency improvements.
> I've been testing locally with nginx, and the only setting I need is
> to set `autoindex on;` in the location that points to the Sword repo.
> Everything works fine, then. I have tested in the past with Apache
> (although it has been a very long time since I've touched it) and it
> works fine there.
>
> There's nothing currently stopping us from swapping to HTTP or HTTPS
> client-based access anywhere that is built against curl. Which is
> every Linux distro I've seen, MacOS CLI distributions, and the Xiphos
> Windows builds. Also, I'm pretty sure all the known mobile apps
> support it, as well.
>
> --Greg
>
> On Fri, Feb 28, 2025 at 10:05 AM Troy A. Griffitts
> <scribe at crosswire.org> wrote:
>
> Hey guys,
>
> We did some work in the past to make this transition begin to
> happen automatically without user actions. I think there was an
> outline in here a year or so back. Basically, the path forward was:
>
> 1) to standardize a location in a SWORD repository for .zip
> package. This has many benefits beyond helping us get to HTTPS
> everywhere. It conceptually allows reading directly from the zip
> file by the engine. It could facilitate the engine itself
> building the zip packages from an exploded tree. All transports,
> including the dominant FTP transport right now could look to see
> if there is an available single .zip package for download instead
> of downloading each individual file for a module during a remote
> install, and I am sure more ideas people will imagine in the
> future... but simply this step was to standardize the location of
> .zip package which each represent a zip of a single package based
> at the root of a sword module repo, e.g.,
>
> KJV.zip:
> mods.d/kjv.conf
> modules/texts/ztext/kjv/nt.bzs
> modules/texts/ztext/kjv/ot.bzs
> modules/texts/ztext/kjv/ot.bzv
> modules/texts/ztext/kjv/nt.bzv
> modules/texts/ztext/kjv/ot.bzz
> modules/texts/ztext/kjv/nt.bzz
>
> This location decided on was pretty straighforward, at the root of
> the sword module repo, packages/
>
> You'll notice this if you have a look at the crosswire repo here:
>
> https://crosswire.org/ftpmirror/pub/sword/raw/
>
> The latest version of the engine does use this now and tries to
> downoad a package if available, before it falls back to the
> previous behavior of downloading each individual file.
>
> 2) Support 'chained' transport providers in the engine, allowing
> multiple transports to the same repository. This has been
> completed and the first use of this concept is the recognition of
> a new HTTPSPackagePreference option in installmgr's configuration
> file.
>
> The value of the property is: existing repo
> name|hostname|repository_path
>
> You can see this added for the CrossWire repository in the master
> repo list registry at:
>
> https://crosswire.org/ftpmirror/pub/sword/masterRepoList.conf
>
> HTTPSPackagePreference=CrossWire|crosswire.org
> <http://crosswire.org>|/ftpmirror/pub/sword/raw
> FTPSource=CrossWire|ftp.crosswire.org
> <http://ftp.crosswire.org>|/pub/sword/raw
>
> This tells installmgr that if a user requests a package install
> from "CrossWire" then chain 2 transport provider attempts together
> and prefer the HTTPS transport first.
>
>
> The hope is that this path forward will see the vast amount of
> SWORD frontends begin to start using HTTPS instead of using FTP,
> to facilitate a smooth transition from FTP to HTTPS. The idea is
> that if a repo owner sets everything up correctly, and a user has
> the latest software and latest configuration information from our
> registry of known SWORD repos (
> https://crosswire.org/ftpmirror/pub/sword/masterRepoList.conf ),
> everything will flow over HTTPS instead of FTP and once we see FTP
> traffic drop to very low levels, a repo owner could choose to
> discontinue support for FTP if they desire.
>
> Michael, I am glad you have your server back up and running well
> again. Thank you for all that you do to distribute the Word of
> God to a needing world.
>
> Troy
>
>
> On 2/27/25 9:45 PM, Michael Johnson wrote:
>>
>> Hi David,
>>
>> Yes, under eBible.org, where it says "HTTP access", it should say
>> "HTTPS and HTTP access". Also, any repository that also supports
>> HTTPS and/or HTTP access should explicitly say so. As Greg
>> mentioned, mobile often blocks FTP. FTP may not be dead, yet, but
>> it is in hospice care. HTTP likewise can cause problems with
>> mobile platforms, where the major app stores now require use of
>> HTTPS instead of insecure HTTP in most cases. I still support
>> insecure HTTP connections to eBible.org except for at
>> shop.eBible.org <http://shop.eBible.org> and
>> https://eBible.org/cgi-bin/contact.cgi
>> <https://eBible.org/cgi-bin/contact.cgi> but there may come a day
>> when HTTP goes away, too.
>>
>> On 2/26/25 22:03, David Haslam wrote:
>>> Hi Michael,
>>>
>>> /Does anything need to be changed here?/
>>> Official and Affiliated Module Repositories - CrossWire Bible
>>> Society
>>> <https://wiki.crosswire.org/Official_and_Affiliated_Module_Repositories#eBible.org>
>>>
>>> _Everyone_: Except for the note under *AndBibleExtra*, there's
>>> no mention of *https* !
>>> /Do we need to make any wider changes to be future proof?/
>>>
>>> Best regards,
>>>
>>> David
>>>
>>> Sent with Proton Mail <https://proton.me/mail/home> secure email.
>>>
>>> On Thursday, February 27th, 2025 at 7:13 AM, Michael Johnson
>>> <kahunapule at eBible.org> <mailto:kahunapule at eBible.org> wrote:
>>>>
>>>>
>>>> On 2/26/25 17:12, Greg Hellings wrote:
>>>>>
>>>>>
>>>>> On Wed, Feb 26, 2025, 7:33 PM Kahunapule Michael Johnson
>>>>> <kahunapule at ebible.org> wrote:
>>>>>
>>>>> Greetings from Maui!
>>>>>
>>>>> tldr: upgrade your Sword apps to always use https instead
>>>>> of http or ftp to access repositories ASAP.
>>>>>
>>>>>
>>>>> While technically any network acess other than anonymous FTP
>>>>> support is optionally supported only with a build dep, in
>>>>> reality there is no need to support anything other than HTTPS.
>>>>> Every Linux distribution, and Windows build of note has
>>>>> libcurl, the Brew version is also built against it, and the
>>>>> HTTP(S) support was added because mobile often blocks FTP.
>>>>>
>>>>> So you're basically completely safe.
>>>>
>>>> Awesome!
>>>>
>>>>
>>>>>
>>>>>
>>>>> As many of you are probably aware, the last week was not a
>>>>> model of reliability for the eBible.org repository, or for
>>>>> the rest of the eBible.org site. On the 19th of February,
>>>>> the eBible.org server hardware failed. Exactly what
>>>>> failure, I don't know, because it was in a data center
>>>>> over 4,000 miles from my house. I just knew that it
>>>>> wouldn't talk to me in any of the 3 ways I can normally
>>>>> access the leased dedicated server. No worries, because I
>>>>> have a fast backup, right? I allocated a new dedicated server
>>>>> from the same company (Ionos) and attempted to restore
>>>>> from a backup. That failed with about 80 error messages.
>>>>> Next plan: restore from a mirror image of the server in my
>>>>> home office. That actually worked, but it took more than 3
>>>>> days to get all of the data there (about 300 GBytes), plus
>>>>> time to get all of the configuration right. In the mean
>>>>> time, my other leased server (the one that didn't crash,
>>>>> hosting 24 other sites) gave early warning signs that it
>>>>> was not going to be in service much longer. Then
>>>>> everything worked except that I forgot a couple of tweaks
>>>>> I had to do to make the ftp server compatible with Sword.
>>>>> I fixed that, and things were still not OK. EBible.org
>>>>> availability kept going up and down like a yo-yo, mostly
>>>>> because the remote control software I was using was not
>>>>> designed to handle multiple IP addresses per server and
>>>>> anonymous ftp sites. Also, the cost of allocating multiple
>>>>> IP v4 addresses has gone up. Anonymous ftp is pretty much
>>>>> obsolete. I will be dropping it, but slowly.
>>>>>
>>>>>
>>>>> A Herculean effort, but I'm glad for you that your recovery
>>>>> was successful! I'm curious why you need 4 separate addresses?
>>>>> What is the need, there?
>>>>
>>>> So far, I have been using Plesk to set up virtual hosts. I have
>>>> 25 sites (and some aliases for those), some of which are much
>>>> more important than others. Plesk lets me share one IP address
>>>> with all sites except any site that has an anonymous ftp
>>>> service associated with it. The only site I have that has an
>>>> anonymous ftp service associated with it, of course, is the
>>>> ftp.eBible.org <http://ftp.eBible.org> Sword repository. So I
>>>> had to assign 2 IP version 4 addresses to the server. For a
>>>> long time, I was running 2 servers with every site on them for
>>>> redundancy. I had stopped doing that because the sites grew too
>>>> large for one of the servers I was renting, and I thought I had
>>>> a workable fast backup/restore plan, unlike when I had
>>>> extremely slow and expensive Internet in Papua New Guinea. (I
>>>> have some serious space in audio and video Bibles.) So that is
>>>> 2 servers x 2 IP addresses = 4 IP addresses. But that
>>>> configuration was unstable, so I went to just one IP address
>>>> per server by fighting my old ally, Plesk, using manual ProFTP
>>>> configuration (and a cron job to slap my configuration back
>>>> whenever Plesk rewrites it). That is not a really good
>>>> long-term solution, though.
>>>>
>>>>> ...
>>>>> Would you like a hand building up some DR or deployment
>>>>> automation so you can avoid needing to remember settings? IT
>>>>> automation is one of my primary skillsets, so if you'd like
>>>>> any sort of help setting it up, let me know. For instance,
>>>>> it's not too hard to put together automation scripts to run on
>>>>> a provisioned box to stand up the web server, ftp server, etc
>>>>> so that you don't need to manually edit files and the like.
>>>> That would be useful. That could be a way to escape my
>>>> dependence on and fight with Plesk.
>>>>>
>>>>> Alternatively, have you considered an alternative way to host
>>>>> the data? You could probably build a Container image with all
>>>>> the files in it and host that on something like Amazon
>>>>> Container Service or any of the many cloud Kubernetes hosts
>>>>> around. A container image would also make it easy for someone
>>>>> to grab the whole collection and make it available in an
>>>>> offline context the way they can with the old CD images Troy
>>>>> used to distribute.
>>>> I have looked at alternatives in the past, but it may be worth
>>>> looking again. When I last looked, AWS was more expensive at my
>>>> traffic levels and site counts than using a rented dedicated
>>>> server. Another alternative might be hosting at my house when
>>>> (if?) Hawaiian Telephone makes good on its promise to bring
>>>> fiber Internet to my neighborhood. (It is actually available
>>>> about a half mile away, right now, but I haven't seen them
>>>> working on it around here.)
>>>>>
>>>>> Or even put the files into an object storage container if
>>>>> you're dedicated to eliminating FTP access eventually. With
>>>>> just a small shell script you can push the needed files and
>>>>> their indexes into an S3, Ceph, etc object storage service and
>>>>> then you wouldn't need to run a dedicated server with them to
>>>>> manage uptime. All of those services offer ways to expose the
>>>>> files over HTTPS.
>>>>>
>>>>> As I said on Facebook, I'm happy to lend a hand if there's
>>>>> anything I can do to help smooth your infrastructure! I can
>>>>> even host an emergency mirror if need be, as I have pretty
>>>>> reliable Internet and electric when my neighbors don't drive
>>>>> into the electric poles. This year I'm dedicating some of my
>>>>> time to working on home electric backups!
>>>>
>>>> Thank you, Greg. I may take you up on that...
>>>>
>>>> --
>>>>
>>>> Peace,
>>>> */Michael Johnson/**
>>>> 26 HIWALANI LOOP • MAKAWAO HI 96768-8747*• USA
>>>> mljohnson.org <https://mljohnson.org/> • eBible.org
>>>> <https://eBible.org> • WorldEnglish.Bible
>>>> <https://WorldEnglish.Bible> • PNG.Bible <https://PNG.Bible>
>>>> Signal/Telegram/WhatsApp/Telephone: +1 808-333-6921
>>>> Skype: kahunapule • Telegram: @kahunapule • Facebook:
>>>> fb.me/kahunapule <https://www.facebook.com/kahunapule>
>>>>
>>>
>>>
>>> _______________________________________________
>>> sword-devel mailing list:sword-devel at crosswire.org
>>> http://crosswire.org/mailman/listinfo/sword-devel
>>> Instructions to unsubscribe/change your settings at above page
>> --
>>
>> Peace,
>> */Michael Johnson/**
>> 26 HIWALANI LOOP • MAKAWAO HI 96768-8747*• USA
>> mljohnson.org <https://mljohnson.org/> • eBible.org
>> <https://eBible.org> • WorldEnglish.Bible
>> <https://WorldEnglish.Bible> • PNG.Bible <https://PNG.Bible>
>> Signal/Telegram/WhatsApp/Telephone: +1 808-333-6921
>> Skype: kahunapule • Telegram: @kahunapule • Facebook:
>> fb.me/kahunapule <https://www.facebook.com/kahunapule>
>>
>>
>> _______________________________________________
>> sword-devel mailing list:sword-devel at crosswire.org
>> http://crosswire.org/mailman/listinfo/sword-devel
>> Instructions to unsubscribe/change your settings at above page
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page
>
>
> _______________________________________________
> sword-devel mailing list:sword-devel at crosswire.org
> http://crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://crosswire.org/pipermail/sword-devel/attachments/20250301/7adab792/attachment-0001.htm>
More information about the sword-devel
mailing list