[tyndale-devel] Strongs Tagging for ESV
Troy A. Griffitts
scribe at crosswire.org
Sat Mar 19 16:34:35 MST 2011
Nice job guys. Just a point of clarification:
On 03/19/2011 01:04 PM, David Instone-Brewer wrote:
> 4) merge the resultant text with the verb parsing in the tagged KJV
I'm confused a bit about where the NASB and KJV come into play with your
tagging efforts.
> Since starting this, I've heard from Troy who originally organised the
team who tagged the NASB.
> He says his method is:
We did not tag the NASB. We tagged the KJV. I would not use the NASB
markup if I was doing this project, to avoid any copyright infringement
of Lockman's data.
Troy
On 03/19/2011 09:54 PM, David Instone-Brewer wrote:
> Dear Rob
>
> I've been doing some experiments with Gen.1 to work out a system.
> I've found a method which works really well - the whole tagging of
> Gen.1 has been done correctly by automatic comparisons and it has only
> gone wrong in a few verses.
> I've tried using Stanfords parsing engine at
> http://nlp.stanford.edu:8080/parser/
> but this didn't fix it. I've attached a file listing my experiments
> and their results.
>
> I think what would fix it is a semantic domain dictionary. What's
> happened is that the two versions are too different in v. 11:
>
> ESV: And God said, "Let the earth sprout vegetation, plants yielding
> seed, and fruit trees *bearing fruit in which is their seed, each
> according to its kind, on the earth*." And it was so.
> NASB: Then God said, "Let the earth sprout vegetation: plants yielding
> seed, /and /fruit trees *on the earth bearing fruit after their kind
> with seed in them*"; and it was so.
>
> The change in order in the words in bold makes it too difficult for
> the comparison program to match things up.
>
> I think we will need humans at these points, but I think we can
> highlight the likely places where problems exist.
>
> Tomorrow I'll have a go at producing the whole text of Genesis, so you
> have some data to play with
>
> David IB
>
>
> ============
> THe process we are attempting is:
>
> 1) convert the NASB XML text to something which looks like a
> BibleWorks exported text
> (ie each verse on one line starting with a simple ref (eg Gen 1:1 In
> the beginning...)
>
> 2) use the Word 2003+ text comparison tools (which are much superior
> to Word 97) to compare the text of both versions producing something
> like:
>
> Gen 1:2 <w H776>The earth</w> was *formless *<w
> H8414>*formless*</w> <w H922>and void</w>, and <w H2822>darkness
> </w> <w H5921>was over</w> the <w H6440>*sur*face </w> <w H8415>of
> the deep</w>*, and . And *<w H7307>the Spirit </w> <w H430>of
> God</w> was <w H7363!b>*moving hovering *</w> <w H5921>over</w>
> the <w H6440>*sur*face </w> <w H4325>of the waters. </w>.
>
> 3) create a site where human can easily correct this automatic markup
> - eg the proof of concept here
> <http://www.slowley.com/tagger-proof-of-concept/example.html>.
>
> 4) merge the resultant text with the verb parsing in the tagged KJV
>
> Since starting this, I've heard from Troy who originally organised the
> team who tagged the NASB. He says his method is:
>
> 1) starts with a lemma tagged text, the KJV, and CrossWay's ESV
> data in OSIS format.
> 2) the ESV module is iterated each verse at a time and is
> processed as such:
> 3) the OSIS markup is stripped from the ESV text and positioning
> information is retained
> 4) a word table is built from the KJV text:
> KJV Word 1 | Strongs #
> KJV Word 2 | Strongs #
> 5) a second table is build from the ESV text:
> ESV Word 1 |
> ESV Word 2 |
> 6) these tables are passed to a function which is responsible
> solely for the logic to fill in the second part of the second
> table with Strong's numbers.
> 7) the returned table is used to reconstitute the the OSIS tags to
> the ESV text including word-level Strong's markup.
> See a screenshot for the community collaboration tool for KJV
> Strongs markup project is at http://crosswire.org/sword/kjv2003/#ss
> We're hoping to convert it to a web application instead of a
> standalone Java GUI, but that hasn't happened yet.
> I'd love to work together on this effort. Please keep me posted
> on any progress and let me know if I can help in anyway.
> Troy
>
>
>
>
>
> At 10:18 17/03/2011, Robert Slowley wrote:
>> So, presumably if you could script it to break each chapter in to a
>> separate file, do the comparisons, and then re-export as a single file
>> we could import that in to a tool like mine so a human could fix the
>> errors and do the bits the auto-comparison failed to do.
>>
>> On Tue, Mar 15, 2011 at 8:19 AM, David Instone-Brewer
>> <davidinstonebrewer at gmail.com> wrote:
>> > From the automatic comparisons produced by Word, we get:
>> >
>> > Gen 1:1 <w H7225>In the beginning,</w> <w H430>God</w> <w
>> > H1254!a>created</w> <w H8064>the heavens</w> <w H776>and the earth
>> </w>.
>> > Gen 1:2 <w H776>The earth</w> was <w H8414>without form</w> <w
>> H922>and
>> > void</w>, and <w H2822>darkness</w> <w H5921>was over</w> the <w
>> > H6440>face</w> <w H8415>of the deep</w>. And <w H7307>the Spirit</w> <w
>> > H430>of God</w> was <w H7363!b>hovering </w> <w H5921>over</w> the <w
>> > H6440>face</w> <w H4325>of the waters </w>.
>> >
>> > - ie the first two verses are already perfectly tagged. In fact
>> there aren't
>> > any problems in Gen.1 till we get to v.5:
>> >
>> > Gen 1:5 <w H430>God</w> <w H7121>called</w> <w H216>the light</w> <w
>> > H3117>Day</w>, <w H2822>and the darkness</w> <w H7121>he called</w> <w
>> > H3915>Night.</w>. And <w H6153>there was evening</w> <w H1242>and
>> there was
>> > morningthe first</w>, <w H259>one</w> <w H3117>day</w>.
>> >
>> > The problem is that Word gives up making these comparisons after a few
>> > chapters.
>> > Some of these problems can be cleared up by macros.
>> >
>> > David IB
>> >
>> > At 00:43 15/03/2011, Robert Slowley wrote:
>> >
>> >> I think I can produce a better text to produce something which has
>> less to
>> >> correct.
>> > What do you mean?
>> >
>> >> It would be useful to have transliterated Hebrew and a single-word
>> meaning
>> >> instead of the numbers.
>> > I have an electronic copy of the stuff you get on popups on
>> >
>> http://classic.net.bible.org/verse.php?search=Genesis%201:30&book=genesis&chapter=1&verse=30
>> <http://classic.net.bible.org/verse.php?search=Genesis%201:30&book=genesis&chapter=1&verse=30>
>>
>> > for Strongs already - which I was planning to integrate. If the
>> > numbers are replaced with 'transliterated Hebrew' or a 'single-word
>> > meaning' what specifically would that mean?
>> >
>> > For instance on
>> >
>> http://classic.net.bible.org/verse.php?search=Genesis%201:30&book=genesis&chapter=1&verse=30
>> <http://classic.net.bible.org/verse.php?search=Genesis%201:30&book=genesis&chapter=1&verse=30>
>>
>> > for the strongs reference h03651, which is the transliterated hebrew,
>> > and which is the single word meaning?
>> >
>> >> It would be useful to divide the top line by the tagging, not by any
>> >> English
>> >> parsing
>> >> eg Gen.1.30 || and to every thing (h3605 )||
>> >> instead of || and to every (h3605) || thing (h3605 ) ||
>> > In the case of Genesis 1:30 the text behind it is:
>> > NASB: ... <w H3605>and to every</w> <w H3605>thing</w> ...
>> >
>> > Presumably there is a reason for the text to have two separate sets of
>> > words both tagged individually with H3605? Or is it just a markup
>> > error?
>> >
>> > Presumably in some cases it words should be merged if they have the
>> > same strongs and are next to each other, but in other cases, this
>> > isn't the case, e.g. Isa 6:3
>> >
>> http://classic.net.bible.org/verse.php?search=isa%206:3&book=isa&chapter=6&verse=3
>> <http://classic.net.bible.org/verse.php?search=isa%206:3&book=isa&chapter=6&verse=3>
>>
>> >
>> > Has:
>> >
>> > <w H6918>Holy</w>, <w H6918>Holy</w>, <w H6918>Holy</w>, is the <w
>> > H3068>Lord</w> <w H6635>of hosts</w>
>> >
>> > because the Hebrew has swdq repeated 3 times, and I assume that the
>> > reader who understands Strong's gets this indication by it being
>> > repeated rather than there being <w H6918>Holy, Holy, Holy</w>. Is
>> > that right?
>> >
>> >> It might be better to have the bottom line with a separate box for
>> very
>> >> word. Sometimes we will want to divide things up differently
>> > As I see it we have 'phrases' (a set of one or more words) which may
>> > have one or more strongs references. In some cases a set of words with
>> > have a shared strongs reference, but in other cases like Isa 6:3 sets
>> > of contiguous words may have the same strongs references but still be
>> > separate 'phrases'. As I see it there's no automatically working this
>> > out.
>> >
>> > What I was thinking was to have some algorithm that tries to
>> > automatically map the NASB strongs annotations on to the ESV text,
>> > similar to what I have already crudely done here. That can either try
>> > to group things as the NASB does (where a set of contiguous words
>> > share a strongs reference), or do what I have done here (which is
>> > easier) which is to automatically group words in to a 'phrase' where
>> > they share the same strongs references.
>> >
>> > Either way not all of the ESV can be automatically annotated in this
>> > way, the annotation will be wrong in some cases, and the automated
>> > grouping may be wrong in some cases. So I was thinking of making the
>> > interface such that once the automated grouping has been attempted the
>> > end user can click on a box which will make it selected, then click on
>> > the next box to the left or right (and so on), when this is done a
>> > button for "merging in to a phrase" would appear - then if this is
>> > clicked they would be made in to a phrase and could have their strongs
>> > references assigned. Alternatively clicking on a box that represents a
>> > phrase of one or more words will cause a "demerge" button to appear
>> > that will separate out all the words. This will allow the end user to
>> > handle both types of situation.
>> >
>> > I also thought some sort of "This verse is tagged correctly" button
>> > would be good. In some cases the program will annotate everything, but
>> > it will still need to be checked by a human - and a human may wish
>> > their annotation to be checked by someone else for quality purposes.
>> > When a verse is marked as correct, it can have a tick or something,
>> > and there can be a page of "verses that need work" which it would
>> > automatically be removed from. Does that sound sensible?
>> >
>> > We have easy access to the SBLGNT (with apparatus) and Leningrad
>> > Codex. Is it worthwhile including those for each verse? I don't know
>> > what process an annotator would go through, and what level of
>> > knowledge of the original languages they would use.
>> >
>> > I worked a bit today on tidying up the classes I've written, and
>> > improving the processing of the text (in the next few weeks I'll send
>> > you a list of the suspicious stuff I found while processing your files
>> > ;-) ). I'm away next week for my 1st year's anniversary holiday - but
>> > after that can start work on making this in to an actual web app that
>> > would be useful rather than a static web page demo of the sort of
>> > thing I had in mind.
>> >
>> > Any thoughts / comments / ideas appreciated!
>> >
>> > It'd probably be a good idea to see if we can improve the automatic
>> > annotation of the ESV from the NASB if we can, as any progress made
>> > here before people start manually annotating / checking will reduce
>> > the amount of man hours needed to complete the task.
>> >
>> > -Rob
>> > --
>> > http://www.slowley.com/
>> >
>> > "On two occasions, I have been asked [by members of Parliament],
>> > 'Pray, Mr. Babbage, if you put into the machine wrong figures, will
>> > the right answers come out?' I am not able to rightly apprehend the
>> > kind of confusion of ideas that could provoke such a question."
>> > -- Charles Babbage (1791-1871)
>>
>>
>>
>> --
>> http://www.slowley.com/
>>
>> "On two occasions, I have been asked [by members of Parliament],
>> 'Pray, Mr. Babbage, if you put into the machine wrong figures, will
>> the right answers come out?' I am not able to rightly apprehend the
>> kind of confusion of ideas that could provoke such a question."
>> -- Charles Babbage (1791-1871)
>
>
> _______________________________________________
> tyndale-devel mailing list
> tyndale-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/tyndale-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/tyndale-devel/attachments/20110319/4c0b3413/attachment-0001.html>
More information about the tyndale-devel
mailing list