[sword-devel] Parallel display of modules with varying v11ns

DM Smith dmsmith at crosswire.org
Fri Feb 28 17:02:25 MST 2014 

On Feb 28, 2014, at 2:41 PM, Troy A. Griffitts <scribe at crosswire.org> wrote:

> Костя,
> 
> IOn 02/28/2014 08:14 AM, Костя Маслюк wrote:
>> Ok.
>> 
>> I have got following:
>> http://crosswire.org/~kalemas/work/v11nmapping/paralleldisplay.html
> 
> Amazing!  This looks really great!  Daniel 3 is a nice test chapter.  Your output looks very nice. I will play around with your updates to the test and send mine.
> 
>> /me cant get rid of feeling that Troy still did not disabled his
>> screen filter that rips everything i write to him
> 
> Костя, no, I'm sorry for not replying inline in my last email.  Much of what I wrote was in response to your emails, but it wasn't obvious because I did not post inline. (notice the repentance with this email)
> I read everything you wrote and was excited to start the conversation again, and concluded that if we can just prove that one implementation CAN handle pretty well a majority of the cases, then we can move forward and commit to this API interface we're trying. The theoretical conversation wasn't going anywhere and a proof of concept seemed to be the best way forward.  As far as the implementation, I am concerned about your same points, that SWORD and JSword need to have a common set of mapping data and ideally a common storage format for that data.  I'm not concerned about the size and speed immediately as we can always improve the implementation.
> 
> I just would like the programming interface and how we intend for it to be used by consumers to be solid; I don't want frontend developers to have to change their code.  I think our proof of concept should satisfy this.
> 
> As for the shared mapping data and storage mechanism, we need to collaborate with JSword.
> 
> Conceptually, I have always been leery of a 'superset meta v11n' concept to do this mapping.  It seems the most straightforward way if we can establish this superset, but conceptually it practically prevents things like mappings between the different versifications of Josephus-- which is a very real problem we'd like to solve with the same mechanism.
> 
> I believe you are going from X -> KJV+ -> Y right now.
> 
> I think this logic is fine but was hoping for the internal data to be boiled down generically to optimized deltas somehow,e.g.,: X->KJV { verseShift(Ps.9.21-:10.1); chapterShift(Ps.10-112:+1) ... }
> and then when asked to map from X -> Y, we could look at our mappings and find the most optimized path.  It may still be X->KJV->Y, but it may also be X->Y or JosephusLoeb -> JosephusWhiston.
> 
> If we force the concept of a superset KJV+ v11n scheme into our mapping concept, I am afraid it will limit us and we will continually have to update this meta v11n when we create new modules and find new strange things.
> 
> Chris can comment, but simply mapping the various LXX editions to each other, alone, can be daunting to think about.
> 
> This all is aside from the API mechanism on which we are working presently, but just offered for discussion between JSword and SWORD and others when considering how we wish to represent and persist these mappings.
Regarding the mappings, JSword currently maintains them in text files. The were derived or inspired by Костя work. Chris Burrell did the work. The files contain lines of the form:
left=right
where left is a verse or verse range in a versification
and right is an intermediate (KJV in the current implementation) that has the same number of verses as on the left.

The left and the right are encoded as simplified osisRef ranges of the form:
osisID or osisID-osisID
the osisID can have a subIdentifer, typically of the form !a or !b.
Every osisID has 3 parts: Book, Chapter and Verse and optionally a subIdentifier.

We've written an osisID and osisRef parser optimized for these. We found that our general purpose parser that allows for varied user input was way too slow to load the file (on low end Android devices).
This is a simplified parser because we don't handle "work" or "grain" or space separated lists of the above. And simplified because Book, Chapter and Verse have to be present.

A minor exception is made for the verse count needing to be the same: verse 0 is considered to have zero width. This is needed especially for Psalms.

The subIdentifiers are used for transitive mappings from one non-KJV v11n to another non-KJV v11n.

There are several places where a single verse maps to more than one verse. (or visa versa) This is addressed by having multiple rules that cumulatively result in the correct mapping.

The text file is an intermediate form that is read and compiled into core if it is needed.

That core representation is immaterial for this discussion.

The nice thing about the JSword implementation is that the parallel module code has been part of JSword for quite a while and has been updated such that the front-end does not need to know anything at all about versification mapping.

The impact of any change to the mappings is transparent to JSword frontends.

The shortcoming of the current methodology is that it maps only those verses that are part of the KJV without the Apocrypha. There are mechanisms to declare that verses don't map to the KJV and visa versa.

I think this is fine. At least for now.  At any point and time we can change the intermediate with appropriate mappings and there will be no impact to JSword frontends.

I forget who mentioned it, but it is reasonable for the mapping files to be compiled into a form useable by the engine. If our files are not used, we'll work on compiling the common files into what we need.

In Him,
	DM


> 
> Troy
> 
> 
> 
> 
>> 2014-02-28 9:48 GMT+04:00 Troy A. Griffitts <scribe at crosswire.org>:
>>> Костя,
>>> 
>>> Tonight I spent some time adding a new example to the engine's code examples
>>> tree for displaying Bibles in parallel.  It basically rips off the XHTML
>>> header, styles, and footer from SWORDWeb and then executes a small, isolated
>>> function to output the parallel display.  This small function can be our
>>> playground to test our stuff to see how we've done.  This will force us to
>>> implement the use case for our work at least once to see how ugly the code
>>> gets.  Right now, it looks good, like we expect, but there is no logic yet
>>> to handle any case but 1:1 translation.
>>> 
>>> I've checked the example in because I think this will be a handy example for
>>> frontends to follow when we get something working nice.
>>> 
>>> I feel it is important, before we commit to an API mechanism, that we
>>> consume that mechanism at least once, trying to solve the use case for which
>>> it was conceived-- at least at a basic level.
>>> 
>>> Those who are interested to just see the minimum code required to display in
>>> parallel, but don't wish to check out the latest SVN, can have a look here
>>> (at the parallelDisplay(...) method):
>>> 
>>> http://crosswire.org/svn/sword/trunk/examples/tasks/parallelbibles.cpp
>>> 
>>> the example can be run and tested with something like:
>>> 
>>> ./parallelbibles KJV ESV jn.3.16 > paralleltest.html
>>> firefox paralleltest.html
>>> 
>>> You can see the output from this test run here:
>>> http://crosswire.org/~scribe/paralleltest.html
>>> 
>>> Let's collaborate! :)
>>> 
>>> Troy
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On 02/26/2014 02:56 PM, Костя Маслюк wrote:
>>> 
>>> Oh, i just get what you meant about speed and size of translation. What you
>>> would like to achieve beyond i have implemented? It is optimized in speed
>>> and is very lightweight in size.
>>> 
>>> As a bonus it can be  used in per translation versification concept.
>>> 
>>> The only thing i would like to change is to slightly increase size, adding
>>> one byte per rule to store rule type, so it can handle difficult cases in
>>> future with backward compatibility.
>>> 
>>> 26.02.2014 23:00 пользователь "Troy A. Griffitts" <scribe at crosswire.org>
>>> написал:
>>>> One positive thing from the previous thread is the reminder of Kosta's
>>>> proposed implementation for translation between modules of varying v11n.
>>>> 
>>>> The accusation of irresponsibility is warranted, not for delaying the
>>>> patch submission, but for delaying the discussion toward a resolution and
>>>> buyin by a consensus of frontends.
>>>> 
>>>> To sum up:
>>>> 
>>>> We have refactored and isolated translation to a single point within the
>>>> engine. Basically, when you set the value of one VerseKey from a VerseKey
>>>> with differing v11n, translation will happen. This propogates naturally to
>>>> many places in the engine. For example it will allow one to set the LXX
>>>> module from a key obtained from the KJV module:
>>>> 
>>>> lxx.setKey(KJV.getKey());
>>>> 
>>>> 
>>>> The question still on the table is: how useful is this for the primary use
>>>> case of displaying in parallel modules with varying v11ns?
>>>> 
>>>> A secondary question is how can we optimize, in both speed and size, the
>>>> translation. The JSword team is beginning to implement their own mechanism
>>>> and I would like to hear about their experience.
>>>> 
>>>> There are open threads on this with many of my, and others, thoughts and
>>>> concerns. I would appreciate it if commenters might consider searching the
>>>> list history before commenting.
>>>> 
>>>> My theoretical question is, what logic do we want to use to create a
>>>> parallel display? There are many hard cases we haven't resolved, even if the
>>>> resolution is "we simply don't handle that, and what you'll see is X."
>>>> 
>>>> I know the STEP tools have a parallel display implementation. I have no
>>>> idea if its behavior in corner cases is acceptable to most.
>>>> 
>>>> --
>>>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>>> _______________________________________________
>>>> sword-devel mailing list: sword-devel at crosswire.org
>>>> http://www.crosswire.org/mailman/listinfo/sword-devel
>>>> Instructions to unsubscribe/change your settings at above page
>>> 
>>> 
>>> _______________________________________________
>>> sword-devel mailing list: sword-devel at crosswire.org
>>> http://www.crosswire.org/mailman/listinfo/sword-devel
>>> Instructions to unsubscribe/change your settings at above page
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> sword-devel mailing list: sword-devel at crosswire.org
>>> http://www.crosswire.org/mailman/listinfo/sword-devel
>>> Instructions to unsubscribe/change your settings at above page
>> _______________________________________________
>> sword-devel mailing list: sword-devel at crosswire.org
>> http://www.crosswire.org/mailman/listinfo/sword-devel
>> Instructions to unsubscribe/change your settings at above page
> 
> 
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20140228/6473d48b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4145 bytes
Desc: not available
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20140228/6473d48b/attachment-0001.p7s>

More information about the sword-devel mailing list