[sword-devel] JavaScript & SWORD modules

[Greg Hellings]

On Sun, Jan 19, 2014 at 6:34 PM, Matej Cepl <mcepl at redhat.com> wrote:

> On Sun, Jan 19, 2014 at 01:57:41PM -0600, Greg Hellings wrote:
>
>> Multiple efforts have been made to allow JavaScript implementations to
>> directly read a Sword module file format, but this can be fraught with
>> difficulty.
>>
>
> Well, this seems to be dismissing a possibility of only-JS Sword app too
> quickly. Especially, considering that I have https://github.com/zefanja/
> biblez-ng installed on my Firefox OS and it works.
>

I'm not dismissing it, as in my email moments ago to Stephen I was only
hoping to save JavaScript programmers from a JS-only implementation, which
can be fraught with all the difficulty of maintaining a port for what is a
very complex system already that sits on top of nearly 20 years of
development.


>
> But of course, let all flowers bloom, we can have multiple solutions to
> one problem. However, I see immediately couple of problems:
>
> * Offline access. Of course, we can store megabyte JSONs in the   phone,
> instead of storing native XML, but I am not sure   dealing with huge JSONs
> is that much faster than some   reasonably good access to XML? Is it? How
> large is the Book   of Psalms in JSON?
>

I wasn't concerning myself with the client-side storage. There's a very
large number of possibilities
1) Storing the JSON files as-is. For browser work this would require
something like the HTML 5 File Apis. However, unless you're dealing with
browser plug-ins or FirefoxOS type of apps, this probably isn't an option.
2) IndexedDB, which is supported in most browsers (it looks like IE added
support in 10, although I haven't bothered to look at how limited that
might or might not be). Typically limited to 5MB per origin, although there
are plenty of ways to get around that or request it be overridden for
things like Chrome plugins.
3) The localStorage functionality.
4) Other vendor-specific options in other environments that I might not be
savvy to (Chrome used to have WebSQL which died in the birth pangs of
IndexedDB as one out of date example)

Storage wasn't the point. Getting the data to the browser, mobile OS, or
other JavaScript environment was the point. The best storage solution will
entirely depend on the environment being targeted and the purpose for which
the data is being requested.


>   And if you say that users are online 100% time than it is   just not
> true. Firefox OS is meant for low-end phones in poor   countries where 3G
> bandwidth (don’t even start to hope for   4G) is quite often metered. Heck,
> it is metered even here in   relatively affluent Prague, Czechia (and I
> know that this is   not that much huge exception of Czechia). Not
> mentioning   metro, inside of churches, and other closed spaces where
> connection is either extremely poor or non-existent.
>

I wasn't even thinking of this as a situation where it would be a live
online usage. That's why I included the WebSocket functionality in the
controller. I intended it for bulk imports. If it was just one request
every time the user requested a single passage or entry then leaving it as
REST would be just great. But if the user wants to import all of Webster's
dictionary, I wanted to provide a method where that could be done with
maximal efficiency while still preventing the server from needing to render
out the entire text at once.


> * OSIS uses almost universally milestoned-XML and there is   a reason for
> that. It seems to me that JSON is by its nature   exact opposite of
> milestones. How would you deal with   chapters, verses, pericopes, crossing
> each other?


OSIS does, but Sword modules don't. Sword modules come in pretty much
well-defined blocks when you ask for a chapter:verse. Because, while it
makes great science to have Book > Section > Paragraph, the real world is
that people want to know what that passage was their preacher talked about
today and he referred to it as 1 Thessalonians 5:16-18.  My sample
implementation, as I said, is not designed to be production worthy. Nowhere
is this more obvious than the fact I'm rendering out the plain, stripped
text. This was partly a purposeful choice, to make it simple, and partly
because JSword does not offer any choices besides plain text or OSIS.

If I was doing this for a production situation I'd have opted to render out
HTML instead but that required far more effort to either lift the XSLT from
Bible Desktop or implement this with the Java JNI bindings and call Sword's
HTML rendering options (more likely still I would have implemented it on
top of a simple Python WSGI application and leveraged the Sword bindings).
By switching the rendering to OSIS and implementing your own transformation
(every major browser I know of has XSLT support) logic on it, you could
choose to handle pericopes, footnotes, alternate readings, Strong's
support, interlinear, etc. From my very rudimentary understanding of JSword
(this is the first thing I've ever written against it) it would be almost
trivial to switch this output to OSIS, and only a little bit more error
handling would make it robust enough for production use. I even already
built in some basic caching so that no request will require re-rendering.
This might become very memory intensive very quickly, so it could easily be
dropped on a production system, I just wanted to save my little VPS
instance some processing time on large requests such as all possible
entries in Webster's.


>
>  The README file has basic build and run directions. I have a sample of it
>> up and running at http://www.thehellings.com:10001/ and ws://
>>
>
> And front-end demo (Web or CLI)?
>

My purpose was to strip away that layer and just show that there is not a
need, from the desires I have seen on this list so far, for a pure
JavaScript implementation. Not that it can't be done or that people will
fight you on it if you really want to put in that level of effort. I only
wanted to demonstrate that the results can also be acquired through JSON
which is so native to JavaScript and then we JavaScript developers can do
what we do best: focus on the front-end presentation of the UI and the
rendering of the material. Why tackle more of the stack than is necessary?
If you can stand on the shoulders of 20 years of development effort that's
gone into Sword (and I don't know how much, but plenty that's gone into
JSword) to build your utility, why wouldn't you? That's what Xiphos,
BibleTime, BPBible, BibleCS, xulsword, PocketSword, Bishop, AndBible, the
web study tool, numerous helper scripts written in Perl and Python, Bible
Desktop, Eloquent/MacSword, and still yet others have all opted for. All of
them, and more, rely on only two implementations of the Sword engine
because the shared effort provides these people the ability to value-add,
rather than toil away fixing all the bugs and reverse engineering the file
structure.

Please, I don't mean to discourage any work with a JavaScript front end.
I've put my own efforts into one in the past, and spend more than 70% of my
professional time developing in JavaScript. If that is your "itch", please
"scratch it". But also, please at least consider the advice from people who
have worked with Sword for a long time (I'm still a newcomer and I've been
here 9 years now) when we say that there might be a better way than
attempting to re-implement the entire engine. I wanted to do this because
myself and some of the "old timers" on the list have repeated stated over
the past year that you can get this data exported to a JavaScript friendly
format without much trouble. It might seem like we don't put much stock
into our own statements because no examples or effort was forthcoming
because most of us are either busy with other projects (e.g. Troy on the
engine, DM Smith on JSword, Chris Little on versifications and modules) or
busy with our home/work lives. So consider this a first pass at a JSON
frontend for JSword, which I'm happy to continue working with other
collaborators on so that you can go on to use it or take inspiration from
it and hopefully save everyone a bit of work.

If a basic JavaScript demo for accessing the data would help, I'd be happy
to put something barebones together that shows the fetching of the data.

--Greg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20140119/3ae5187e/attachment-0001.html>