Well, after looking into this further, I'm thinking that adding another OSIS style in stand-off markup would probably hurt us more than help us, as it would cause even more fragmentation in OSIS document structures. The key for accessibility is consistency, and this can be achieved server-side via transformations, e.g. either BCV or BSP hierarchies may be chosen. As long as the data is stored internally in an unambiguous specific and precise format, any structure can be generated from it. If the user really wants stand-off markup, a TEI document could be returned (which would closely reflect the internal Token and TokenStructure models we have so far, which utilize stand-off “markup” (references): <a href="http://github.com/openscriptures/api/blob/master/models.py">http://github.com/openscriptures/api/blob/master/models.py</a> ).<br>
<br>Troy, I would very much like to see your best practices doc which recommends a single specific method for encoding OSIS. Such a document could detail the default OSIS format for the Open Scriptures Web API.<br><br>Thanks,<br>
Weston<br><br>P.S. It seems as if the OSIS core & user mailing lists have died down again?<br><br><br><div class="gmail_quote">On Thu, Jan 21, 2010 at 9:40 AM, Weston Ruter <span dir="ltr"><<a href="mailto:westonruter@gmail.com">westonruter@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">Troy:<div class="im"><br><blockquote style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;" class="gmail_quote">
I did say that since OSIS allows different ways to mark the same
structure, we have an importer which attempts to accept any valid OSIS
doc and _normalizes_ that doc into a form of OSIS we find easiest for
our engine to process. It is still OSIS, just a form of OSIS with all
structures represented in a single way.<br></blockquote></div><div><br>Thank you for clarifying this, and also for sharing some of this history behind the development of OSIS.<br><br><blockquote style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;" class="gmail_quote">
[We chose to] augment the specification with a 'best practices' doc which recommends
a single specific method for encoding OSIS.<br></blockquote> <br>I don't think I have seen this best practices doc. Is this something you use internally at CrossWire as part of your importer script? Could you direct me to it? I like the approach you took, allowing varying OSIS encodings but recommending only one of them. This is similar to the development of XHTML 1.0 dialects, where you are allowed to use the Transitional doctype, but the Strict doctype is recommended. Doing this for OSIS could answer the need for an unambiguous single markup language. The best practices document would need to contain the practices that are endorsed by at least the majority of players; the others could abstain and still use their preferred (deprecated) dialect of OSIS. Along with this best practices doc, an official normalizer script that translates OSIS into the recommended encoding would be great.<br>
<br>I look forward to your thoughts about stand-off markup encoding of OSIS, especially how well it might serve as the new recommended way to unambiguously encode OSIS.<br><br>Thanks!<br>Weston<br><br></div><br><div class="gmail_quote">
2010/1/19 Troy A. Griffitts <span dir="ltr"><<a href="mailto:scribe@crosswire.org" target="_blank">scribe@crosswire.org</a>></span><div><div></div><div class="h5"><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Weston Ruter wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
... Troy, as you've said before, you can't actually use OSIS as your raw data format at CrossWire because an OSIS document can be authored in many different ways and so there is much more programming logic that is needed to handle all of the possible OSIS styles.<br>
</blockquote>
<br>
Hey Weston,<br>
<br>
Hope to have time for a thoughtful response to more of your suggestions, but just wanted to clear a couple things up first:<br>
<br>
I hope I never implied that we can't/don't use OSIS internally as our primary markup standard.<br>
<br>
I did say that since OSIS allows different ways to mark the same structure, we have an importer which attempts to accept any valid OSIS doc and _normalizes_ that doc into a form of OSIS we find easiest for our engine to process. It is still OSIS, just a form of OSIS with all structures represented in a single way.<br>
<br>
Even so, we still don't use any plain text format as our "raw data format". We typically compress and index documents when they are imported into our engine. You can ask our engine for OSIS, HTML, RTF, GBF, ThML, or plaintext and it will do its best to give you the data in the requested format.<br>
<br>
None of this to argue against your point: OSIS has multiple ways to encode a single structure in a document.<br>
<br>
The real answer to this is not technical. I too am frustrated with this. But many people working at many organizations were consulted when developing the OSIS specification. They gave great insights to how they work. Sometimes they even made demands with an ultimatum that they would absolutely not use the specification if a certain feature was not added to the spec.<br>
<br>
OSIS could have been technically finished in less than a year. It took us 3 years to get buy-in from all the participating organizations.<br>
<br>
In the end, the purpose of OSIS was to build collaboration between organizations. We could have developed a much easier to use technical specification which no one would have used, or conceded to demands to gain buy-in, and augment the specification with a 'best practices' doc which recommends a single specific method for encoding OSIS. We chose the later.<br>
<br>
Implementing code against the spec now, it makes our importer a pain in the butt to write, but in the end, we get what we want: a single OSIS style that our engine knows how to work with, and multiple supporting organizations producing OSIS documents.<br>
<font color="#888888">
<br>
<br>
Troy.</font><div><div></div><div><br>
<br>
<br>
<br>
If we could define a single document structure, however, one<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
that is a subset of the freedom that OSIS provides (perhaps taking cues from OXES), we could then have an XML format for scripture that would be suited for efficient interchange and application traversal.<br>
<br>
Currently we have the problem of two overlapping hierarchies: BSP and BCV. However, there could be potentially multiple versification systems, so there could be even more than two overlapping hierarchies, probably why the <p> element isn't currently milestonable. To get around the problem of overlapping hierarchies, what if we introduced stand-off markup into the equation? The words of scripture themselves could all be located in a flat structure as siblings; then in the header there could be multiple CONCUR sections (views) that list out the elements which belong to the various parts of the hierarchies<br>
<br>
For example, the current approach:<br>
<br>
<p><br>
<verse osisID="Example.1.1" sID="Example.1.1" /><br>
<w id="w1">Then</w><br>
<w id="w2">he</w><br>
<w id="w3">said</w><w id="p1">,</w><br>
<q marker="“" sID="Example.1.1.q1" /><br>
<w id="w4">Let</w><br>
<w id="w5">us</w><br>
<w id="w6">go</w><w id="p2">...</w><br>
</p><br>
<p><br>
<w id="w7">but</w><br>
<verse eID="Example.1.1" /><br>
<verse osisID="Example.1.2" sID="Example.1.2"/><br>
<w id="w8">don't</w><br>
<w id="w9">forget</w><br>
<w id="w10">your</w><br>
<w id="w11">backpack</w><w id="p3">.</w><br>
<q marker="”" eID="Example.1.1.q1" /><br>
<verse eID="Example.1.2" /><br>
</p><br>
<br>
<br>
<br>
Could instead appear as (I'm making up these element names):<br>
<br>
<concur><br>
<view type="verse" osisID="Example.1.1" xpointer="range(#w1, #w7)" /><br>
<view type="verse" osisID="Example.1.2" xpointer="range(#w8, #q2)" /><br>
<view type="quote" xpointer="range(#q1, #q2)" /><br>
<view type="para" xpointer="range(#w1, #p2)" /><br>
<view type="para" xpointer="range(#w7, #q2)" /><br>
</concur><br>
<content><br>
<w id="w1">Then</w><br>
<w id="w2">he</w><br>
<w id="w3">said</w><w id="p1">,</w><br>
<w id="q1">“</w><w id="w4">Let</w><br>
<w id="w5">us</w><br>
<w id="w6">go</w><w id="p2">...</w><br>
<w id="w7">but</w><br>
<w id="w8">don't</w><br>
<w id="w9">forget</w><br>
<w id="w10">your</w><br>
<w id="w11">backpack</w><w id="p3">.</w><w id="q2">”</w><br>
</content> <br>
By structuring a document like this, multiple overlapping hierarchies can be cleanly defined, although they are separated from the underlying content: this however, provides the benefit of clearing up the confusion as to where the <verse>, <p>, and <q> elements should be placed: in the concur section, they each can share references to the same content elements and so their boundaries are specified at the exact same location. This means that XML processors would be able to consistently handle each of the hierarchies as they interweave throughout the content data.<br>
<br>
Efraim Feinstein and James Tauber introduced me to this approach to structuring markup. See also: <a href="http://www.tei-c.org/Guidelines/P4/html/NH.html#NHCO" target="_blank">http://www.tei-c.org/Guidelines/P4/html/NH.html#NHCO</a><br>
<br>
Weston<br>
<br>
</blockquote>
<br>
</div></div></blockquote></div></div></div><br>
</blockquote></div><br>