[sword-devel] OSIS markup for gen books and devotionals

[Troy A. Griffitts]

Thanks for the email Karl.  Yes, when I read Laurie's concise and 
informative report of how we do on different features and output 
formats, I was grateful in my heart and planned to use her report for 
improving the XHTML filters.  I agree, these are bugs and we need to fix 
them.  I started the XHTML filters with 2 hopes: that those who use 
their own filter sets would consider collaborating together on this 
filter set provided with the engine, and that those who use the XHTML 
filters would improve them.  We all have 5 projects we're working on 
these days and as usual, what is urgent for our own work, or what is 
easy and fun, is usually highest priority for each of us.  I'm happy to 
see a commit to improve the XHTML filters.

Regarding your TTS impl, have you considered calling 
module.stripText().  This should give a reasonable plain-text 
representation of a module entry.

On 10/08/2014 11:33 AM, Karl Kleinpaste wrote:
> Longish ramble.
>
> I'm still finding our lack of attention or interest regarding 
> consistent output somewhat disappointing.  David wrote a lot a couple 
> weeks back about this, but some of it just plain bugs me, and no one 
> else followed up at all.  Some of the bugs-me is non-specific, some is 
> very specific.
>
> On 09/23/2014 02:32 PM, David "Judah's Shadow" Blue wrote:
>> Now all this might sound pedantic and that front-ends should just 
>> render what the engine sends, but imagine a frontend that sends the 
>> text through a TTS engine for visually impaired persons. This 
>> frontend would have no use for HTML formatting, but it would care 
>> what the underlying markup that this HTML represents is.
> I have 2 reactions to this observation.
>
> 1. I don't know if any other frontends are capable of it, but Xiphos 
> has been TTS-friendly for ~8 years.  (Cf. Read Aloud in the View menu 
> for walking straight through a Bible, or using mouse-swept regions + 
> context menu invocation.)  And indeed, as noted, I don't care what the 
> markup looks like, indeed I ship the text through a tag stripper 
> before it goes out to Festival.  There is no consideration at all to 
> what was there, it is all blindly removed and simply shipped for 
> speaking.  Yes, one could hypothetically say that a change of voice 
> could be used in the TTS case, but -- this is important -- _nobody cares_.
>
> You see, for as long as I've been around, there has always been a huge 
> amount of talk within Sword about The Wonderful Things That Could Be 
> Done.  But the real world's bottom line is that the five-9s proportion 
> of our actual user base has a straightforward goal: Good, productive 
> Bible study using quality tools.  Precious few are deaf or blind, and 
> almost none are interested in full-tilt syntactic analysis tools.  So 
> all the "see, the markup could let Joe Handicap have the text 
> delivered in his Special Way" really doesn't mean squat in the real world.
>
> 2. I'm not arguing against OSIS.  Not all, in fact.  Nor against 
> handicap support.  But what I'll say about this sort of over-attention 
> to the hyperactive extremes of what could be done routinely leads us 
> to miss, or deliberately avoid, the underlying problem.
>
> That problem, as addressed by Laurie Fooks' experimentation, is that 
> collectively we do a kind of poor job of consistent rendering of 
> what's under there.  For the five-9s crowd, what they want is good, 
> consistent display of textual content.  For the moment, ignore the 
> potential blind or deaf user of our apps.  The problem is that we, the 
> frontend developers, have a pronounced tendency to produce /different 
> things/ for the /same text/, as displayed in the usual panes of our 
> apps.  Is that because of the frontend itself, or because of what the 
> frontend gets from the engine?
>
> That is where Laurie's experimentation reached, and where we 
> collectively fall over the cliff.  We are accelerating at 32f/s^2 
> toward going */splat/* on the canyon floor below while blandly 
> discussing how nifty a more extensive TTS would be.
>
> As far as the "not a Bible-reading program at all" crowd...I 
> absolutely do not care.  The syntactic analyzer portion of our 
> userbase is past the nine-9s crowd.
>
> It occurs to me that my "N-9s" nomenclature may be unclear or 
> unknown.  Five-9s is 99.999%.  Nine-9s is 99.9999999%.  I use these in 
> mild hyperbole.  When I say that the five-9s crowd wants just to see 
> good Bible textual display and whatever features can readily go with 
> that, I mean that out of a hundred thousand users, just one of them is 
> outside that usual space.  Now, I said it's hyperbole, and someone 
> will tell me that they have a whole community of users for whom TTS is 
> crucial...totalling all of 10 people, or even 100 people.  Well, fine, 
> you've moved the issue from five-9s to four-9s, 99.99%, and ... 
> whoopie.  You have done nothing to make me feel better.
>
> Besides, as noted, Xiphos has TTS, it's proven to be quite a bit more 
> welcome than I ever intended it to be -- it was my first significant 
> hack on then-GnomeSword, /which //I did //as an exercise /for code 
> familiarity's sake -- and to my knowledge there exists not one Xiphos 
> user who is a Xiphos user /because of /TTS.  They are Xiphos users 
> because there are other aspects they like, having to do with workflow 
> or presentation or search capability or whatever else, and prefer 
> these aspects over other Sword apps, and TTS is a pleasantly useful 
> side capability they are happy to put to use.
>
> ...
>
> The problem is that, for given OSIS constructs, the user -- and, let 
> us not be remiss, the publisher -- cannot be assured that what arrives 
> at the user's eyeballs is even readable, much less provided as intended.
>
> Why is this?  Xiphos uses straight engine output from XHTML filters.  
> BibleTime uses the engine but with its own filters. JSword apps have 
> their own entire separate engine, and JSword is now driving almost 
> half of all Sword apps out there.  Within apps using the C++ engine, 
> BibleCS uses a different set of filters (RTF) which evidently do not 
> produce the same visual result as the XHTML filters.  Consistency is 
> lacking because the end result of all these widely differing 
> filtrations cannot be counted upon.  Is BibleCS' visual display 
> different because of inherent limitations or differences from Xiphos 
> vis-a-vis RTF vs. HTML display targets?  I'm not in a position to 
> render an opinion on that.
>
> But I can say that module authors like Laurie are stuck: They are 
> forced to use a least common denominator set of features of OSIS, 
> being the preferred markup methodology, because the interpretation of 
> that markup, as rendered into a display window, is not consistent, 
> cannot be counted upon.
>
> Laurie:
>> >If we don't have a high level of commonality then I am concerned that
>> >we are losing the purpose of having a common "engine"
> David:
>> Well, no not necessarily, the purpose of the engine is to read the 
>> modules and then provide the content in a way that the front end can 
>> meaningfully use for its purpose.
> This makes me want to throw things.
>
> The purpose of the engine is to produce the text according to the 
> filters which its client, the frontend, has chosen, and from which 
> that client expects consistency of output result.
>
> That's all.
>
> Anything else avoids the question.  A frontend's "meaningful use" is 
> irrelevant if the engine's product is not in line with what the module 
> author wanted.  Cf. Laurie's experience with tables and so forth.  The 
> filters are supposed to produce output from an OSIS table spec, 
> however that looks in the original markup, into something that makes 
> sense in a frontend's HTML or RTF or WhateverElse widget.  When those 
> filters don't do that, crud gets displayed, crud like Laurie found.
>
>> You can look at it this way, if the engine determined how frontends 
>> should display the text, what would the point of multiple front ends be?
> Wow, David, can you say "red herring"?
>
> The reasons for picking a particular frontend have precious little to 
> do with that.  What drives Xiphos users?  Historically, for as long as 
> I've been involved with the code, it's been user interaction features 
> and visual quality, especially for user-created and -editable content, 
> which is why Xiphos has user annotations and editable genbooks and 
> tree-structured bookmarks and multi-bookmarks and verse highlights and 
> dynamically resizable images and a bunch of other things.  That's what 
> users on the mailing lists have asked for, so that's what we've produced.
>
> That the frontends should display the same text in closely the same 
> way should be axiomatic, and is precisely what Laurie is looking for.  
> And David whistled right past it.  It's not textual display that 
> differentiates today's frontends, it's their other features far beyond 
> "can it display text properly?"  We haven't even achieved that 
> level-zero axiom.  We should be ashamed.
>
> Besides this, I believe a hug/e, //HUGE/ aspect of this is bound up in 
> an issue that is nearly rejected outright, when it's given any 
> attention at all: How does The Publisher want it to look?  Do we even 
> bother to ask?  If we ask, do we listen to the answer?  Laurie wants 
> OSIS tables to behave a certain way, namely, according to its own 
> spec.  Has anybody taken her thoughts to heart?  Have any filter fixes 
> been done in the name of accommodating what she perceives as bugs?
>
>> But until people need the additional features, they won't be added. 
>> Images are a good example, this is very new to the code, but wasn't 
>> put in until someone felt the need.
> Images have been supported in the engine for as long as I've been 
> around, again about 8 years.  This is not new by any reasonable 
> meaning of the word.  By comparison, the XHTML filters are less than 2 
> years old and are already in wide use.  There has been an ImageSampler 
> module since long before I was around.
>
> Laurie:
>> >I do feel that the system / project would benefit
>> >from establishing (or publishing/clarifying same) minimum standards
>> >for markup functionality
> David:
>> This is a laudable goal (with caveats for for the aforementioned 
>> cases where presentation is not visual or doesn't care about 
>> formatting for other reasons). But this is very difficult to do.
> You might as well have said "we can't fix bugs."
>
> The fact that Laurie can report that Feature X (such as table display) 
> doesn't work consistently in OSIS markup as rendered in all the 
> frontends means that the engines, both C++ and JSword, are buggy.  
> Mothing more and nothing less.  They need to be fixed so that they do 
> what they're expected to do, so that they behave according to spec.  
> There /is/ a spec, y'know, a document that says there are minimum 
> standards for markup functionality, just as Laurie asked immediately 
> above.  Once that's done right, if any further frontend bugs remain in 
> the resulting output as displayed, then we in frontend development 
> will have our own bugs to fix.  Until then, it's an engine problem.
>
> --karl
>
>
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20141008/ee8d5b36/attachment-0001.html>