[sword-devel] Remote Module Repository Wiki

[Jonathan Morgan]

On Mon, Nov 8, 2010 at 7:48 AM, Greg Hellings <greg.hellings at gmail.com>wrote:

> On Sun, Nov 7, 2010 at 8:28 AM, Jonathan Morgan <jonmmorgan at gmail.com>
> wrote:
> > A lot here depends on evaluation of pros and cons.  I personally support
> > HTTP, zipped modules, and one central file (like mods.d.tar.gz) to give a
> > list of all the books and where to find them for at least some of the
> > following reasons.  I will try and capture why succinctly:
> > For ZIPs:
> > 1. Minimising the number of GETs issued (and potentially file size and
> > download speed): I think this is particularly important for loading the
> > initial list of modules, which should preferably be as fast as it
> possibly
> > can be.  It also makes a difference for image modules like Dore's
> Woodcuts
> > with lots of loose files (which I remember taking a very long time to
> > install with Install Manager).
>
> I'm not sure what your metrics are, but Ubuntu's apt for me hits about
> 70+ files on a half dozen or more servers to update its cache and this
> takes less than 2 seconds.  It takes installmgr almost no time to
> process a single refresh.
>

Well, I just tried it with Xiphos on Windows and it took mine 4 or 5 seconds
to refresh the main CrossWire repository.  My enduring memory of
InstallManager is that it feels slow.  My guess is that the difference is
due to going from Oz to US rather than from US to US (I do have a reasonably
fast network connection).  Apt on Ubuntu is a lot faster, but that is
because it is mirrored to an Oz site and doesn't have the latency.

I'm assuming that we have among our target audience:
1. People who are on the other side of the world from the US.
2. People who do not have a fast network connection.

The other thing is that I believe the Install Manager case we are testing *
is* in fact the case I advocate: It is grabbing mods.d.tar.gz rather than
doing FTP file traversal and grabbing one conf file at a time.  I don't know
how to test it without rebuilding SWORD, but I think the InstallMgr would
grab one conf file at a time and so have a lot of latency from places like
here.  This (a repository not requiring mods.d.tar.gz) is the simple FTP
server setup that is being promoted.


> The main problem is that we are not setup to handle display from
> multiple sources and also not to refresh from all the repositories at
> once, they have to be refreshed one at a time.  This isn't so ideal to
> the user's experience, but that's how it is done.  I really doubt we
> need to worry about the performance of hitting 30 or 40 repositories
> if that is what the user selects to enable - and if we do, then that
> indicates we need to rethink how we're hitting the repositories,
> because the limit certainly isn't the number we'd be hitting at that
> point.
>

If the vision is having individual publishers publishing their own work,
then having 30 or 40 repositories would be a big success - and I don't think
we can scale to that point (I don't see any prospect of that any time soon,
but if that is the vision it's worth thinking about whether we can support
it).

Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20101108/1b78eac9/attachment-0001.html>