[sword-devel] Sword Modules Access/Format

[Greg Hellings]

On Fri, Dec 23, 2011 at 6:03 PM, Ivo Benner <skython at web.de> wrote:
> Hello,
>
> I want to write a javascript based bible application that should be able
> to download and use SWORD modules.
> Its javascript because it should run as offline app on Linux, OSX,
> Windows and even on all major mobile devices.
>
> The Sword modules are binary files on a ftp server, so i've got some
> issues so solve.
> Binary file reading is possible with modern browsers and on mobile
> devices with the FileReader API or AJAX with ArrayBuffers.
> There are even libraries for zip and zlib compressions.

Since your application will be running in one of these and will not be
on a web page, those APIs should be available to you.

>
> Since it's a Javascript application the best format would probably be
> JSON and I already wrote a converter to convert bible modules to JSON,
> but the files are quite huge.

JSON might be a good way to do the import, but you probably don't want
to actually store and manipulate the modules this way. Reading a
full-sized JSON file into a JavaScript environment (a full module like
the KJV can run up into the 5-10MB range or more) will take a very
long time to parse. When you're dealing with embedded worlds like
cellphones and tablets the loading and parse time for a JSON file of
that size can be as high as a few minutes depending on many different
variables.

After retrieving the file in JSON from the server you probably will
want to store it in a JavaScript friendly database environment if
you're not going to be working with a SWORD engine embedding. Most
modern web browsers (that is to say: those other than IE) come
equipped with the IndexDB which is akin to schema-less/NoSQL databases
like MongoDB. Some of them also have the older but deprecated WebDB
which is a SQL database (defined in the specification to be compatible
with SQLite and usually implemented by an actual instance of SQLite
itself). Most embedded environments also have access to SQLite (it is
iPhone's preferred method for applications to store user data, and I'm
fairly sure its available on Android as well). You will find that
lookup, retrieval and search times in a database will be MUCH better
than loading and parsing a JSON file for embedded devices usually by
several orders of magnitude.

The static JSON files are probably a great method of distribution, but
I wouldn't rely on storing modules in that format.

>
> My current problem with the binary format is that i cannot access the
> files on http://crosswire.org with an AJAX request since there is an
> "Origin Policy" which doesn't let other websites or local javascript
> files fetch data from it.

This is a limitation in the browser. If you are running in a
stand-alone JavaScript environment, like v8 from the command line, a
Chrome browser plugin, etc., there are settings you can activate to
allow you access to domains other than the origin domain. I have a
browser plugin for Chrome that is able to access pages from
http://www.biblegateway.com even though that is not its origin domain.
The cross-domain limitation is a browser-enforced thing and is not a
limitation of JavaScript. The mechanism to allow arbitrary domain
access is probably different for each environment you will target, but
the cross-domain origin policy is pretty much limited only to web
pages you load and not to anything else.

>
> So here are my questions:
> Am I allowed to convert and host JSON converted Sword modules or are
> there any licensing issues?

Each module will be different. You are welcome to look through the
list of modules on the CrossWire repositories. Many of them are public
domain or have open distribution rights. Those you are free to
distribute as you would like. There are others which are listed as
under copyright however CrossWire has been given the rights to
distribute the files - those you would not be legally able to host on
another site.

However, if you get to that point, I would be willing to work with you
to do format-shifting of those which are permissible and host them on
CrossWire's site. If format shifting is allowed for the CrossWire-only
modules, then the effort to host on CrossWire's site in JSON format
would be beneficial.

As I mentioned above, I have a start on a Chrome plugin that is able
to access BibleGateway, and the effort to put SWORD modules into a
JavaScript-friendly format is one I've already put some brain power
into and would benefit my plugin, so I'd be happy to work with you on
that.

> Can you allow AJAX cross origin requests for the sword files? For an
> Apache webserver this would be:
>
>  Header set Access-Control-Allow-Origin "*"

This is not a server thing. This is a client-side thing as I address above.

>
> within the directory rule. I'm accessing the files via HTTP:
> http://www.crosswire.org/ftpmirror/pub/sword/raw/
> Cross domain/origin FTP access is not possible with AJAX as far as i know.

I am unfamiliar with a JavaScript environment that supports FTP at
all. However, if we format shift to JSON we can host the files under
any scheme that we like. HTTP would obviously be what we would target
so as to avoid the difficulty of FTP. I'm sure in some environments
there are FTP libraries that can be accessed from embedded JavaScript
interpreters, but for the broadest appeal HTTP would be much more
desirable.

--Greg

>
> If you can help me or if you have any questions concerning my
> application please let me know.
>
> Ivo
>
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page