[jsword-devel] Getting the global key list for a book

Chris Burrell chris at burrell.me.uk
Fri Feb 15 14:31:34 MST 2013


SO here's the method I came up with, with one uncertainty about whether
maxIndex should be "-1" or not, but without it, i seem to be reading too
far.

    public Key getFastGlobalKeyList() throws BookException {
        ZVerseBackendState rafBook = null;
        try {
            rafBook = initState();

            String v11nName =
getBookMetaData().getProperty(ConfigEntryType.VERSIFICATION).toString();
            Versification v11n =
Versifications.instance().getVersification(v11nName);

            Testament[] testaments = new Testament[] {
                    Testament.OLD, Testament.NEW
            };

            Key globalList =
PassageKeyFactory.instance().createEmptyKeyList(v11n);
            Passage passage = KeyUtil.getPassage(globalList, v11n);
            BitwisePassage bitwisePassage = null;
            if(passage instanceof BitwisePassage) {
                bitwisePassage = (BitwisePassage) passage;
                bitwisePassage.raiseEventSuppresion();
                bitwisePassage.raiseNormalizeProtection();
            }


            for (Testament currentTestament : testaments) {
                RandomAccessFile compRaf = currentTestament ==
Testament.NEW ? rafBook.getNtCompRaf() : rafBook.getOtCompRaf();

                // If Bible does not contain the desired testament, then
false
                if (compRaf == null) {
                    // no keys in this testament
                    continue;
                }

                int maxIndex = v11n.getCount(currentTestament) -1;

                // Read in the whole index, a few hundred Kb at most.
                byte[] temp = SwordUtil.readRAF(compRaf, 0, COMP_ENTRY_SIZE
* maxIndex);

                // for each block of 10 bytes, we consider the last 2 bytes.
                for (int ii = 0; ii < temp.length; ii += COMP_ENTRY_SIZE) {
                    // can this be simplified to temp[8] == 0 && temp[9] ==
0?
                    int verseSize = SwordUtil.decodeLittleEndian16(temp, ii
+ 8);

                    //can this be optimized even further - i.e. why
decodeOrdinal, when add() go simply pass in and store an ordinal
                    if (verseSize > 0) {
                        int ordinal = ii / COMP_ENTRY_SIZE;
                        if(bitwisePassage != null) {
                            bitwisePassage.addVersifiedOrdinal(ordinal);
                        } else {
                            globalList.addAll(v11n.decodeOrdinal(ordinal));
                        }
                    }
                }
            }

            if(bitwisePassage != null) {
                bitwisePassage.lowerNormalizeProtection();
                bitwisePassage.lowerEventSuppressionAndTest();
            }

            return globalList;
        } catch (IOException e) {
            throw new BookException(JSMsg.gettext("Unable to read key list
from book."));
        } finally {
            IOUtil.close(rafBook);
        }
    }



On 15 February 2013 20:26, Chris Burrell <chris at burrell.me.uk> wrote:

> Hi DM
>
> I'm about to test the following method for at least all Z backends:
>
> Maybe you can tell me what you think of it:
>
> *getFastGlobalKeyList*
>
> public Key getFastGlobalKeyList() throws BookException {
>         ZVerseBackendState rafBook = null;
>         try {
>             rafBook = initState();
>
>             String v11nName =
> getBookMetaData().getProperty(ConfigEntryType.VERSIFICATION).toString();
>             Versification v11n =
> Versifications.instance().getVersification(v11nName);
>
>             Testament[] testaments = new Testament[] {
>                     Testament.OLD, Testament.NEW
>             };
>
>             Key globalList =
> PassageKeyFactory.instance().createEmptyKeyList(v11n);
>             Passage passage = KeyUtil.getPassage(globalList, v11n);
>             BitwisePassage bitwisePassage = null;
>             if(passage instanceof BitwisePassage) {
>                 bitwisePassage = (BitwisePassage) passage;
>                 bitwisePassage.raiseEventSuppresion();
>                 bitwisePassage.raiseNormalizeProtection();
>             }
>
>
>             for (Testament currentTestament : testaments) {
>                 RandomAccessFile compRaf = currentTestament ==
> Testament.NEW ? rafBook.getNtCompRaf() : rafBook.getOtCompRaf();
>
>                 // If Bible does not contain the desired testament, then
> false
>                 if (compRaf == null) {
>                     // no keys in this testament
>                     continue;
>                 }
>
>                 int maxIndex = v11n.getCount(currentTestament);
>
>                 // Read in the whole index, a few hundred Kb at most.
>                 byte[] temp = SwordUtil.readRAF(compRaf, 0,
> COMP_ENTRY_SIZE * maxIndex);
>
>                 // for each block of 10 bytes, we consider the last 2
> bytes.
>                 for (int ii = 0; ii < temp.length; ii += 10) {
>                     // can this be simplified to temp[8] == 0 && temp[9]
> == 0?
>                     int verseSize = SwordUtil.decodeLittleEndian16(temp,
> 8);
>
>                     //can this be optimized even further - i.e. why
> decodeOrdinal, when add() go simply pass in and store an ordinal
>                     if (verseSize > 0) {
>                         if(bitwisePassage != null) {
>                             bitwisePassage.addVersifiedOrdinal(ii % 10);
>                         } else {
>                             globalList.addAll(v11n.decodeOrdinal(ii % 10));
>                         }
>                     }
>                 }
>             }
>
>             if(bitwisePassage != null) {
>                 bitwisePassage.lowerNormalizeProtection();
>                 bitwisePassage.lowerEventSuppressionAndTest();
>             }
>
>             return globalList;
>         } catch (IOException e) {
>             throw new BookException(JSMsg.gettext("Unable to read key list
> from book."));
>         } finally {
>             IOUtil.close(rafBook);
>         }
>     }
>
>
> *addVersifiedOrdinal (is just a simplification of add, so that we can
> give the ordinals directly, rather than converting to a verse and back
> again).*
> /**
>      * A shortcut to adding a key, by ordinal. The ordinal needs to be
> taken from the same versification as the passage being created.
>      *
>      * @param ordinal the ordinal
>      */
>     public void addVersifiedOrdinal(int ordinal) {
>         Versification v11n = getVersification();
>         optimizeWrites();
>
>             store.set(ordinal);
>
>         // we do an extra check here because the cost of calculating the
>         // params is non-zero and may be wasted
>         if (suppressEvents == 0) {
>             Verse verse = v11n.decodeOrdinal(ordinal);
>             fireIntervalAdded(this, verse, verse);
>         }
>     }
>
>
>
> On 15 February 2013 20:22, DM Smith <dmsmith at crosswire.org> wrote:
>
>> That's what http://www.crosswire.org/tracker/browse/JS-246 is all about.
>>
>> There are a good number of places that we get all the keys for a module.
>> But the construction of the list is dumb.
>>
>> The question is what do you want? Do you want all the possible keys in a
>> versification? Or do you want the keys for the verses that are actually in
>> a module?
>>
>> If you only want chapters that exist, consider working off of
>> Versification to get the book names and the chapters. For each chapter, get
>> the number of verses and grab one (maybe verse 1 or one halfway into the
>> chapter) on the assumption that if one exists then the rest do.
>>
>>
>> Calling book.contains(key) is what you want to do. It goes against the
>> backend to determine whether the verse is in the module.
>>
>> Don't call book.getGlobalKeyList(); We probably should deprecate the
>> method. It is too easy to call it and it is very expensive for what it does.
>>
>> I have a 3 day weekend and hope to finish the av11n work. It impacts
>> this. Will probably add iterators for a versification.
>>
>> In Him,
>>         DM
>>
>>
>>
>> On Feb 15, 2013, at 2:14 PM, Chris Burrell <chris at burrell.me.uk> wrote:
>>
>> > Hi all
>> >
>> > getGlobalKeyList() seems to do a lot of work against the file system.
>> >
>> > I'm attempting to create a sitemap, which includes a URL to every
>> chapter. Presumably getGlobalKeyList is what I want to ensure that I'm only
>> displaying those chapters that exist. I was intending to do:
>> >
>> > globalKeyList = getGlobalKeyList();
>> > for(each book in versification) {
>> >     myBookKey = book.getValidKey(book.getShortName())
>> >     myBookKey.retainAll(globalKeyList);
>> >
>> >     if(myBookKey.getCardinality() > 0) {
>> >         outputSiteMapChapter();
>> >     }
>> > }
>> >
>> > So for each version I call getGlobalKeyList(), but that in terms seem
>> to make individual reads (through the contains()) method for every
>> potential key in the versification.
>> >
>> > public final Key getGlobalKeyList() {
>> >         if (global == null) {
>> >             Versification v11n =
>> Versifications.instance().getVersification(versification);
>> >             global = keyf.createEmptyKeyList(v11n);
>> >             Key all = keyf.getGlobalKeyList(v11n);
>> >             for (Key key : all) {
>> >                 if (contains(key)) {
>> >                     global.addAll(key);
>> >                 }
>> >             }
>> >         }
>> >         return global;
>> > }
>> >
>> > The contains() method above makes a backend call every time.
>> >
>> > Isn't there a way to obtain the list of keys from the index? I'm not
>> quite sure where the versification comes in here, or is it there simply to
>> map the names of the keys to their positions in the index file? If so, can
>> we simply read all keys in OT and NT, and check
>> >
>> > Doing this on GlobalKeyList() would also speed up index creation.
>> >
>> > Finally, it seems contains() only works on a single key, or takes the
>> first key/verse if it's a passage. Presumably that is correct?
>> >
>> > Chris
>> >
>> > _______________________________________________
>> > jsword-devel mailing list
>> > jsword-devel at crosswire.org
>> > http://www.crosswire.org/mailman/listinfo/jsword-devel
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/jsword-devel/attachments/20130215/cdf78aca/attachment-0001.html>


More information about the jsword-devel mailing list