[sword-devel] Remote Module Repository Wiki

[Jonathan Morgan]

A lot here depends on evaluation of pros and cons.  I personally support
HTTP, zipped modules, and one central file (like mods.d.tar.gz) to give a
list of all the books and where to find them for at least some of the
following reasons.  I will try and capture why succinctly:
For ZIPs:
1. Minimising the number of GETs issued (and potentially file size and
download speed): I think this is particularly important for loading the
initial list of modules, which should preferably be as fast as it possibly
can be.  It also makes a difference for image modules like Dore's Woodcuts
with lots of loose files (which I remember taking a very long time to
install with Install Manager).
2. Provides one unambiguous URL for a module.
3. Uses the same format for distribution stand-alone and in InstallManager.
4. Removes most/all of the directory traversal problems involved with the
installed book structure: This has potential advantages of speed and working
with a greater number of servers as well as programmability.

For HTTP:
1. HTTP is a more common setup than FTP.  Thus it is probably more
accessible to a wider group of publishers (and quite possibly users as
well).  For example, it would certainly be possible to host ZIP files on
website / file sharing services such as Google Sites or DropBox, and even a
mods.d.tar.gz (though keeping it up to date could be a bit troublesome).
FTP support is a whole different kettle of fish.

I understand you want to make things easier for publishers (and that is not
a bad goal), but for me the "holy grail" is not using the same format, but
providing the best user experience.  Speed is an important part of that, and
to me that almost guarantees two separate format, since the best format for
transferring over a network is almost guaranteed to be different from the
best one for reading on disk.  Philosophically, I think it is usually better
to put more work on publishers rather than on consumers, because there will
usually be a lot more consumers than publishers.  [I do think supporting
filesystem installs from the directory structure as you discuss in your
"viral" section makes sense - I just don't think it makes sense over a
network.  Also having multiple paths to get modules from makes sense].

I suspect having standard Unix scripts to set up the file structure (for big
publishers running their own Apache installation) and other helpers might
make it not too much harder than maintaining the raw directory structure.

One other thing: Troy, do your plans for sword-modules.org (or whatever it
was) have any bearing on this?  For example, if it were relatively easy for
a repository to be set up there by Joe Random or Small Publisher X and
sword-modules.org provided FTP, it might lessen the advantage of HTTP over
FTP.

Jon

On Sat, Nov 6, 2010 at 1:35 AM, Troy A. Griffitts <scribe at crosswire.org>wrote:

> There are a number of finely delineated topics here we should separate.
>
> 1) Data format for our "module installation source".
>
>  i.e, is it a list of zips, is it an existing install set, etc.
>
>  Simply to be explicit, we have obvious use cases which come into play:
>
>  a) Installing our software -- all implementations will be different
> because we have a variety of packages running on a variety of platforms.
>  b) Installing modules from a module installation source
>  c) Running our software
>
> The final item (c) seems out of place but really is important.  We have
> successfully achieved a common data storage format for an installation
> set of books (i.e. "library/mods.d/..., library/modules/...) across a
> vast number of platforms.  Sure this hierarchy of files moves around to
> different locations on different platforms, but once the location is
> pinpointed, everything under that location remains the same for any of
> our platforms.
>
> The 'holy grail' for implementation of the (1) above is for both (b) and
> (c) use cases to be handled by the SAME dataset, and is what we are
> working towards.  In other words, if an installed module library can,
> itself, also be an installation source for other instances of any SWORD
> software, then distribution of Bibles becomes viral. This brings us to
> topic 2.
>
> 2) The viral ability END USERS to share modules.
>
> If I can, for example,
>
> o       use the builtin installer of my SWORD software on my Android device
> to
> install my important modules
> o       then plug it into my friends computer and install modules FROM my
> android device WITHOUT ANY PREPARATION
> o       then from his computer right-click on the top module library folder
> and "Share as... 'SWORD Modules'",
> o       then go to any computer on the network and install modules from
> that
> "SWORD Modules" share,
> o       then install FileZilla on one of those computers and let anyone
> over
> the Internet install Bibles from that location,
> o       then point IIS or Apache to that location (http transport needs
> work
> but is on the way-- more talk below)
>
> ... distribution of Bibles then becomes viral.  It is spontaneous.  No
> preparation.  I have my library of books I use regularly with me and if
> someone wants one of my books, then they can install straight from my
> library.
>
>
> 3) The TRANSPORT TECHNOLOGY used to enable this over the Internet.
>
> We chose FTP because it was more practical.  It is very difficult to
> implement the programmatic traversal of a directory and file hierarchy
> over HTTP.  Apache and IIS all spew out different html for directory
> listings.
>
> Now having said this, Nic Carter submitted the first pass at this which
> works for our apache server (and likely many apache servers which spew
> out the vanilla html index page like here:
> http://crosswire.org/ftpmirror/pub/
>
> So, HTTP is currently possible in the engine and if we get push back
> from publishers or find users not able to use FTP due to firewall
> restrictions, we plan to fully support HTTP moving forward.  We simply
> need to break out our directory parsing code and make it work well with
> a few of the more mainstream httpd services like Apache and IIS.
>
>
> 4) A single zip file for installation of a module (or modules) is simply
> a zip of (1c) above.
>
> Of the 3 'zip' formats we support for download from our server, only the
> 'rawzip' format should survive.  I believe I can safely say we all agree
> on that.  This is in essence still the same thing.  It is a module
> repository of exactly 1 module in a difference storage medium.  Heck it
> could be 5 installed modules all zipped up and it would still be the
> same thing.  As I see it, it is an end user preference left to the
> exercise of frontend developers.  "Can my users operate their file
> browser and drag and drop to my app easier than they can navigate their
> "Browse to folder" dialog box in "Install SWORD Modules from path..."
> option in my installer?  If so, then I should support drag and drop on
> my frontend-- after all it is simply an unzip.  But the user gets no
> benefit of our full installer: not description, to versioning upgrade
> information if they already have an instance of that module installed,
> to category, no language, etc.
>
> 5) If we find we cannot achieve this 'no prep' installation utopia in
> the real world and it is necessary to have an intermediate format.
> Matthew Talbert had a great suggestion on IRC last night to simply add a
> 'prep for publishing' (or in his words simply "publish") option to the
> InstallMgr interface which would create (1) "module installation source"
> data format if indeed it technically must be different.  Great idea if
> we need to concede to 2 disparate data formats, but I am still hopeful
> we can avoid this extra step for would-be module distributors-- end
> users and publishers alike.
>
>
> Hope this separates out thoughts a little better.
>
>
> Troy
>
>
>
> On 11/05/2010 01:15 PM, Greg Hellings wrote:
> > On Fri, Nov 5, 2010 at 8:02 AM, Peter von Kaehne <refdoc at gmx.net> wrote:
> >> I think Troy, the concern is correct.
> >>
> >> For the publisher with some decent IT muscle and budget a proper repo
> must be better, but for the small town church with a website and a couple of
> modules to share - zip and http is a must.
> >>
> >> Having a multiplicity of methods of getting modules into the system
> would certainly be easier.
> >>
> >> My preference:
> >>
> >> 1) keep current methods - it is best for huge numbers of modules and it
> is probably also best for anyone with enough money to have a fixed ip and a
> server, able to run anonymous ftp
> >
> > While I very strongly agree that FTP should not be held up as the best
> > solution for everyone (I have worked with and for several
> > organizations who flat out refused to ever allow FTP of any type to be
> > a part of their systems and who primarily ran IIS for their web
> > content), I'm not sure the argument from excessive cost is accurate.
> >
> > Annual DNS leasing from my provider is less than $40/year (I want to
> > say it's only about $17, but I might be remembering incorrectly)
> > Dedicated VPS hosting with dedicated IP address, about 20 GB of drive
> > space and full root access to my system is $19.95/mo.
> >
> > If $23 per month is excessive for anyone who really intends to
> > distribute modules, they can almost certainly find someone else in the
> > SWORD community who will be willing to host it for them at very little
> > to no cost.  Additionally I can host any low-volume traffic I want
> > from home by registering with a place like dyndns.com and setting up a
> > server on any connection that is moderately reliable (my home). The
> > option is not viable for people in developing countries but is
> > certainly more than viable for any first world country I'm aware of.
> >
> > --Greg
> >
> > _______________________________________________
> > sword-devel mailing list: sword-devel at crosswire.org
> > http://www.crosswire.org/mailman/listinfo/sword-devel
> > Instructions to unsubscribe/change your settings at above page
>
>
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20101108/0c247871/attachment-0001.html>