[sword-devel] Remote Module Repository Wiki

[Nic Carter]

On 06/11/2010, at 10:42 PM, Troy A. Griffitts wrote:

>> I was going to download the mods.d.tar.gz file (over HTTP) and use 
>> that for moduleinformation.  Then download the ZIP of the module
>> over HTTP.  If a repo doesn't have the zip folder (or the
>> mods.d.tar.gz file), then I would either have a fall-back to use the
>> normal InstallManager, or perhaps I'd just say that PocketSword
>> wouldn't support that repo. But after a year of PocketSword doing
>> things the "right" way, I've decided to take the JSword approach and
>> hope that doesn't cause too many issues...  :(
> 
> :)
> 
> OK, This is vital for me to understand your experience. I presume the
> "right" way you speak of above is simply using the InstallMgr API.
> 
> Why are you unhappy with the InstallMgr API?  Does it not work?  Is it
> too hard to use?  You mention speed, below...

The "right" way is by directly plugging into the InstallMgr API and getting it to do all the work.  :)  In order to get it to work, I've made a couple of rather minor changes, but that's not an issue at all, and I'm very happy to maintain my own set of patches required to get the SWORD lib working properly on an iOS device.  :)  (or, as an aside, I could submit some patches that have #defines for iOS, but I'm not too fussed.  Perhaps they would be of interest if someone wanted to use the C++ lib for Android, but I'm not sure.)

I want to switch to using HTTP only in PocketSword.  Mobile phone carriers are rather horrid to deal with.  My old carrier here in Australia had transparent proxies and all sorts of horrid things!  Other carriers do other evil things, as well.  And I am fairly certain some of them don't have FTP in the common use-set scenario for their users, even in this day and age of wireless internet, and so don't care if it works very well or not...  :(  I have observed first hand some annoying things with different carriers here -- so perhaps the US is ok where you only have one carrier for the iPhone & hence only one set of horridness to deal with?  ;)  But, HTTP is so commonplace that it's harder to break it.  I have yet to add proper proxy support into PocketSword, but that is high up on the list.  But given that I'm thinking about rewriting the download code for PocketSword, I've put it off and I'll do both at once?
BTW, I think the InstallMgr API works fairly well and is extremely easy to use.  :) When I started work on PocketSword (I was helping Ian out at first), I tackled the downloading of modules as my first task and I found it fairly easy to figure out how to get it to work.  :)

>> my personal tests on an iOS device show that accessing the CrossWire 
>> repo via FTP is reasonably slower than accessing it via HTTP.
> 
> Really?  I wouldn't have expected this at all.

This may have to do with phone carriers?  altho, I'm fairly certain I tested this over WiFi as well...  In my experience, it feels like the "internet" is moving away from FTP and using HTTP for everything?  Just like HTML is now being used for many different things that it was never intended to be used for...  ;)  But, firewalls and proxies often screw with FTP much more than HTTP and many orgs/companies simply block FTP, hence people moving to HTTP to get around those issues.  :(

>> I want the reasonably large performance gains that I can gain from 
>> simply switching protocols!  :)
> 
> Yes, indeed.  This is a very valid concern.  Let me ask, what FTP plugin
> were you using, ftplib or curl?  I am wondering if it might be the
> difference in plugin.  I recently switched BibleCS's InstallMgr GUI to
> use the ftplib system instead of CURL and surprisingly seem to have less
> troubles and more responsiveness.  There is still one major shortcoming
> of ftplib-- it cannot download directly to a buffer for a directory
> listing, we use a temporary file-- but I hope to resolve that in the
> next day or two.
> 
> Speed is a very real and practical concern and if we have a problem, we
> need to resolve it.

I see you just committed that change.  :)  I have only tested with curl -- as that is the only way to get HTTP downloading working.  :)  I never had the option to use ftplib, as downloading to a temporary file under iOS is NOT a good idea...  :(
I could test with ftplib, but given the other objections I have to FTP (*sigh* phone carriers  *sigh*), I'm hoping to avoid FTP.  :(

> On 06/11/2010, at 1:35 AM, Troy A. Griffitts wrote:
>> 
>>> 2) The viral ability END USERS to share modules.
>>> 
>>> If I can, for example,
>>> 
>>> o	use the builtin installer of my SWORD software on my Android 
>>> device to install my important modules o	then plug it into my 
>>> friends computer and install modules FROM my android device WITHOUT
>>> ANY PREPARATION o	then from his computer right-click on the top
>>> module library folder and "Share as... 'SWORD Modules'", o	then go
>>> to any computer on the network and install modules from that "SWORD
>>> Modules" share, o	then install FileZilla on one of those computers
>>> and let anyone over the Internet install Bibles from that location,
>>> o	then point IIS or Apache to that location (http transport needs
>>> work but is on the way-- more talk below)
>>> 
>>> ... distribution of Bibles then becomes viral.  It is spontaneous.
>>> No preparation.  I have my library of books I use regularly with
>>> me and if someone wants one of my books, then they can install 
>>> straight from my library.
>> 
>> I don't think this should hold back usability in other ways.
> 
> In what way does usability relate to storage format or network transport?

Ok, usability may be the wrong word?  But, making things less complicated in order to avoid download issues is probably a very good thing.  And I consider parsing a file listing via HTTP a "more complicated" thing.  I'm happy for us to do more work (like creating a "prepare for publishing" method) in order to try to make things work in more cases (ie: other webservers dishing out their own version of the file listing).  If all we need to do is download 2 files, mods.d.tar.gz & then ESV.zip (as an example), that reduces places where things can go wrong...  :)
Does that make sense?  So, not "usability" but instead reliability?  Perhaps it has something to do with my lack of trust of webservers dishing things up properly all the time & lack of speed of download on some phone carriers, causing timeouts and stuff like that?  :(
Of course, perhaps many of my objections are relevant for mobile devices only and not really for other computing devices?  ;)

>> I would rather that we provided some other functionality (or tool) to
>> make this easy for end users to do, rather than make this mean that
>> the final install of a module is a complete install source.  I love
>> the concept, but I think that your example is something that 0.0001%
>> of users would be interested in, given the nerdery required to know
>> how to do that!  ;)
> 
> I don't understand the suggestion or the "nerdiness" of the current
> method.  We (the engine) does provide install functionality and we
> (frontend developers) do provide a tool that makes this easy for end
> users.  Users already know how to install modules in their favorite
> software.  The user wishing to install follows their familiar procedure
> to install.  When prompted from which Remote Repository they would like
> to install, the user simply clicks a 'browse to local folder' button.  I
> believe most frontends already support this for installing from a CD.

Hmmm, ok, point taken.  I just wonder how many users know they can do this?  Surely it's easier to simply re-download a module from the repo it comes from?  Or in countries where they don't want to be connecting to a "christian" server, they could use the "publish..." command?  Perhaps by implementing a "publish..." command, more users would know that they are able to quickly and easily share their modules?  :)

>> BUT I liked the suggestion made in another post of having a "prepare
>> for distribution" or "publish" menu item that would then 
>> automatically prepare a module (or selection of modules) for 
>> distribution.  :)  Use that menu item as step one, and then you can 
>> proceed through that list of examples that Troy has.  As an added 
>> extra bonus, you get to specify the location to which to "Save" the 
>> module/s ready for distribution.  That way a user doesn't even need 
>> to know where her/his modules are installed to, but instead can say 
>> "desktop" (yes, I'm using the LCD of a user who does everything on 
>> their desktop!) and then easily find that location & "share" that 
>> folder.  :)  (or it could be specified as a network drive, for easy 
>> access to everyone on the network, etc.)
> 
> The "prepare for distribution" is currently unnecessary.  I hope I'm not
> being daft and missing something, but how can adding a step improve user
> experience?  unless...

hehehe -- ok, good point.  :)  But not knowing that you can share modules would eliminate user experience of that altogether!  ;)

> Others have voiced that users might not want to expose their entire
> module set, so I like your idea slightly modified to "Install Subset of
> Books To ..."feature for these users, but for your objection, a simple
> "Show library location" or even "Open library location" which would
> launch their file explorer to the library path, would be cool.

Yes, that would be cool, and I think that would improve the user experience.  But, perhaps it's simply educating users they can actually do this?  But, again, I'd think that it's often quicker and easier to simply get the other user to download the module themselves from the original repo?  But, yes, the persecuted church argument means we really want sharing to be possible.  :)

>> I could easily be persuaded to add an "export" feature to 
>> PocketSword, which would work over WiFi or 3G (similar to the current
>> way of manually installing a module from a ZIP file), so you could
>> share modules on your device with friends.  Of course, this would
>> need to be switched on manually, and wouldn't always be running in
>> the background, so as to conserve battery!  But would be quite funky.
>> :)  But given you need to press a button to switch it on, at that
>> point I could get PocketSword to "prep" the selected modules for
>> distribution.  :)
> 
> When you plug in an iphone, is the location where pocketsword installs
> modules browseable from the hosting computer as USB Storage?  If so, you
> already have the feature.
> 
> Are you suggesting that it takes a nerd to know where to browse to find
> the books?   I can understand that.  And to solve that problem, I could
> see an 'export' function be useful.  But wouldn't it be more useful to
> have something like "Show PocketSword Library Location".  This solution
> would prevent the unnecessary duplication of the entire library of books
> simply because the installing user doesn't know where to browse.

Ahh, the iPhone doesn't work like that.  :)  Apple sandbox each and every app you install, so for eg:  I have PocketSword installed on my laptop in the iOS simulator to:
114:758A718D-5535-4DB4-B8E5-AD27403F1F06 nicc$ pwd
/Users/nicc/Library/Application Support/iPhone Simulator/4.1/Applications/758A718D-5535-4DB4-B8E5-AD27403F1F06

On the actual device, the equivalent would be something similarly a mess...  :D
And that random string is random and you can't read or write to that folder...  So, you either jailbreak and get access, or I have to write code in PocketSword to expose the modules.  :)

> Thanks again for the conversation.  Anyone who's made it this far in
> this email, you must really care about the topic! :)

I hope I make sense?  I am a native English speaker (as much as you consider an Australian to be a native English speaker!), but sometimes don't communicate things as clearly as I should, especially as I often only get time to write nice long emails at various interesting times of the day...  ;)

To summarise, I think HTTP is the direction we should go, as firewalls and (transparent?) proxies can make life hard for FTP.  People work hard to make HTTP work anywhere and everywhere, as everyone wants to jump on to Google and complain if they can't, whereas there is much less demand for FTP now-a-days.  :(  This may be made worse by the fact that I am spending a lot of my time dealing with (silly) mobile phone carriers!  And given the issues we have with HTTP file parsing, I think this provides a good time to look into the possibility of rethinking how we have repos set up.  If we provided tools in order to initialise a repo, that would hopefully make things easier for Publishers to have their own repos.  :D  We want things to be easy for both end users and for repo maintainers, so it's important that whatever we do, life is easy for both groups.  :)

Let me know if there's anything else you think needs elaboration on?  Unfortunately I am mostly thinking of myself and PocketSword, so I understand if people disagree with what I've said.  :)


Thanks, ybic
	nic...  :)