[sword-devel] Remote Module Repository Wiki

[Greg Hellings]

On Mon, Nov 8, 2010 at 5:06 AM, Jonathan Morgan <jonmmorgan at gmail.com> wrote:
> On Mon, Nov 8, 2010 at 7:48 AM, Greg Hellings <greg.hellings at gmail.com>
> wrote:
>>
>> On Sun, Nov 7, 2010 at 8:28 AM, Jonathan Morgan <jonmmorgan at gmail.com>
>> wrote:
>> > A lot here depends on evaluation of pros and cons.  I personally support
>> > HTTP, zipped modules, and one central file (like mods.d.tar.gz) to give
>> > a
>> > list of all the books and where to find them for at least some of the
>> > following reasons.  I will try and capture why succinctly:
>> > For ZIPs:
>> > 1. Minimising the number of GETs issued (and potentially file size and
>> > download speed): I think this is particularly important for loading the
>> > initial list of modules, which should preferably be as fast as it
>> > possibly
>> > can be.  It also makes a difference for image modules like Dore's
>> > Woodcuts
>> > with lots of loose files (which I remember taking a very long time to
>> > install with Install Manager).
>>
>> I'm not sure what your metrics are, but Ubuntu's apt for me hits about
>> 70+ files on a half dozen or more servers to update its cache and this
>> takes less than 2 seconds.  It takes installmgr almost no time to
>> process a single refresh.
>
> Well, I just tried it with Xiphos on Windows and it took mine 4 or 5 seconds
> to refresh the main CrossWire repository.  My enduring memory of
> InstallManager is that it feels slow.  My guess is that the difference is
> due to going from Oz to US rather than from US to US (I do have a reasonably
> fast network connection).  Apt on Ubuntu is a lot faster, but that is
> because it is mirrored to an Oz site and doesn't have the latency.

On longer hauls over today's speeds, physical limitations of the speed
of light dominates for these small files.  Even at only gigabit speeds
within the USA this can happen.  Transocean it can be very prominent.

>
> I'm assuming that we have among our target audience:
> 1. People who are on the other side of the world from the US.
> 2. People who do not have a fast network connection.
>
> The other thing is that I believe the Install Manager case we are testing is
> in fact the case I advocate: It is grabbing mods.d.tar.gz rather than doing
> FTP file traversal and grabbing one conf file at a time.  I don't know how
> to test it without rebuilding SWORD, but I think the InstallMgr would grab
> one conf file at a time and so have a lot of latency from places like here.
> This (a repository not requiring mods.d.tar.gz) is the simple FTP server
> setup that is being promoted.

This is basically the model that Ubuntu repositories use.  Every
repository has a single giant file with meta info on all the available
files.  However, Debian also ships with a package (debmirror) which
makes it almost trivial to setup a local mirror, it will create the
tar file, etc.  Any brief little Perl script could do this, which is
what Debmirror uses, or even a Python script which could be run raw as
Python on Linux and even compiled into a binary for our Windows
friends.  Then the barrier to entry is no bigger than saying "Point an
FTP or HTTP server at your directory, set this little utility to run
daily, and you're good to go."

--Greg