[jsword-devel] Getting the global key list for a book
DM Smith
dmsmith at crosswire.org
Fri Feb 15 15:05:57 MST 2013
I like this.
Couple of comments.
Why call it getFastGlobalKeyList? Shouldn't it have the same name? replacing the old?
We should figure out a way to not need a call to passage instanceof BitwisePassage. Since every passage knows it's versification, it knows how to decodeOrdinal. Maybe we can add an overloaded add(int) to Passage?
On Feb 15, 2013, at 4:31 PM, Chris Burrell wrote:
> SO here's the method I came up with, with one uncertainty about whether maxIndex should be "-1" or not, but without it, i seem to be reading too far.
>
> public Key getFastGlobalKeyList() throws BookException {
> ZVerseBackendState rafBook = null;
> try {
> rafBook = initState();
>
> String v11nName = getBookMetaData().getProperty(ConfigEntryType.VERSIFICATION).toString();
> Versification v11n = Versifications.instance().getVersification(v11nName);
>
> Testament[] testaments = new Testament[] {
> Testament.OLD, Testament.NEW
> };
>
> Key globalList = PassageKeyFactory.instance().createEmptyKeyList(v11n);
> Passage passage = KeyUtil.getPassage(globalList, v11n);
> BitwisePassage bitwisePassage = null;
> if(passage instanceof BitwisePassage) {
> bitwisePassage = (BitwisePassage) passage;
> bitwisePassage.raiseEventSuppresion();
> bitwisePassage.raiseNormalizeProtection();
> }
>
>
> for (Testament currentTestament : testaments) {
> RandomAccessFile compRaf = currentTestament == Testament.NEW ? rafBook.getNtCompRaf() : rafBook.getOtCompRaf();
>
> // If Bible does not contain the desired testament, then false
> if (compRaf == null) {
> // no keys in this testament
> continue;
> }
>
> int maxIndex = v11n.getCount(currentTestament) -1;
>
> // Read in the whole index, a few hundred Kb at most.
> byte[] temp = SwordUtil.readRAF(compRaf, 0, COMP_ENTRY_SIZE * maxIndex);
>
> // for each block of 10 bytes, we consider the last 2 bytes.
> for (int ii = 0; ii < temp.length; ii += COMP_ENTRY_SIZE) {
> // can this be simplified to temp[8] == 0 && temp[9] == 0?
> int verseSize = SwordUtil.decodeLittleEndian16(temp, ii + 8);
>
> //can this be optimized even further - i.e. why decodeOrdinal, when add() go simply pass in and store an ordinal
> if (verseSize > 0) {
> int ordinal = ii / COMP_ENTRY_SIZE;
> if(bitwisePassage != null) {
> bitwisePassage.addVersifiedOrdinal(ordinal);
> } else {
> globalList.addAll(v11n.decodeOrdinal(ordinal));
> }
> }
> }
> }
>
> if(bitwisePassage != null) {
> bitwisePassage.lowerNormalizeProtection();
> bitwisePassage.lowerEventSuppressionAndTest();
> }
>
> return globalList;
> } catch (IOException e) {
> throw new BookException(JSMsg.gettext("Unable to read key list from book."));
> } finally {
> IOUtil.close(rafBook);
> }
> }
>
>
>
> On 15 February 2013 20:26, Chris Burrell <chris at burrell.me.uk> wrote:
> Hi DM
>
> I'm about to test the following method for at least all Z backends:
>
> Maybe you can tell me what you think of it:
>
> getFastGlobalKeyList
>
> public Key getFastGlobalKeyList() throws BookException {
> ZVerseBackendState rafBook = null;
> try {
> rafBook = initState();
>
> String v11nName = getBookMetaData().getProperty(ConfigEntryType.VERSIFICATION).toString();
> Versification v11n = Versifications.instance().getVersification(v11nName);
>
> Testament[] testaments = new Testament[] {
> Testament.OLD, Testament.NEW
> };
>
> Key globalList = PassageKeyFactory.instance().createEmptyKeyList(v11n);
> Passage passage = KeyUtil.getPassage(globalList, v11n);
> BitwisePassage bitwisePassage = null;
> if(passage instanceof BitwisePassage) {
> bitwisePassage = (BitwisePassage) passage;
> bitwisePassage.raiseEventSuppresion();
> bitwisePassage.raiseNormalizeProtection();
> }
>
>
> for (Testament currentTestament : testaments) {
> RandomAccessFile compRaf = currentTestament == Testament.NEW ? rafBook.getNtCompRaf() : rafBook.getOtCompRaf();
>
> // If Bible does not contain the desired testament, then false
> if (compRaf == null) {
> // no keys in this testament
> continue;
> }
>
> int maxIndex = v11n.getCount(currentTestament);
>
> // Read in the whole index, a few hundred Kb at most.
> byte[] temp = SwordUtil.readRAF(compRaf, 0, COMP_ENTRY_SIZE * maxIndex);
>
> // for each block of 10 bytes, we consider the last 2 bytes.
> for (int ii = 0; ii < temp.length; ii += 10) {
> // can this be simplified to temp[8] == 0 && temp[9] == 0?
> int verseSize = SwordUtil.decodeLittleEndian16(temp, 8);
>
> //can this be optimized even further - i.e. why decodeOrdinal, when add() go simply pass in and store an ordinal
> if (verseSize > 0) {
> if(bitwisePassage != null) {
> bitwisePassage.addVersifiedOrdinal(ii % 10);
> } else {
> globalList.addAll(v11n.decodeOrdinal(ii % 10));
> }
> }
> }
> }
>
> if(bitwisePassage != null) {
> bitwisePassage.lowerNormalizeProtection();
> bitwisePassage.lowerEventSuppressionAndTest();
> }
>
> return globalList;
> } catch (IOException e) {
> throw new BookException(JSMsg.gettext("Unable to read key list from book."));
> } finally {
> IOUtil.close(rafBook);
> }
> }
>
>
> addVersifiedOrdinal (is just a simplification of add, so that we can give the ordinals directly, rather than converting to a verse and back again).
> /**
> * A shortcut to adding a key, by ordinal. The ordinal needs to be taken from the same versification as the passage being created.
> *
> * @param ordinal the ordinal
> */
> public void addVersifiedOrdinal(int ordinal) {
> Versification v11n = getVersification();
> optimizeWrites();
>
> store.set(ordinal);
>
> // we do an extra check here because the cost of calculating the
> // params is non-zero and may be wasted
> if (suppressEvents == 0) {
> Verse verse = v11n.decodeOrdinal(ordinal);
> fireIntervalAdded(this, verse, verse);
> }
> }
>
>
>
> On 15 February 2013 20:22, DM Smith <dmsmith at crosswire.org> wrote:
> That's what http://www.crosswire.org/tracker/browse/JS-246 is all about.
>
> There are a good number of places that we get all the keys for a module. But the construction of the list is dumb.
>
> The question is what do you want? Do you want all the possible keys in a versification? Or do you want the keys for the verses that are actually in a module?
>
> If you only want chapters that exist, consider working off of Versification to get the book names and the chapters. For each chapter, get the number of verses and grab one (maybe verse 1 or one halfway into the chapter) on the assumption that if one exists then the rest do.
>
>
> Calling book.contains(key) is what you want to do. It goes against the backend to determine whether the verse is in the module.
>
> Don't call book.getGlobalKeyList(); We probably should deprecate the method. It is too easy to call it and it is very expensive for what it does.
>
> I have a 3 day weekend and hope to finish the av11n work. It impacts this. Will probably add iterators for a versification.
>
> In Him,
> DM
>
>
>
> On Feb 15, 2013, at 2:14 PM, Chris Burrell <chris at burrell.me.uk> wrote:
>
> > Hi all
> >
> > getGlobalKeyList() seems to do a lot of work against the file system.
> >
> > I'm attempting to create a sitemap, which includes a URL to every chapter. Presumably getGlobalKeyList is what I want to ensure that I'm only displaying those chapters that exist. I was intending to do:
> >
> > globalKeyList = getGlobalKeyList();
> > for(each book in versification) {
> > myBookKey = book.getValidKey(book.getShortName())
> > myBookKey.retainAll(globalKeyList);
> >
> > if(myBookKey.getCardinality() > 0) {
> > outputSiteMapChapter();
> > }
> > }
> >
> > So for each version I call getGlobalKeyList(), but that in terms seem to make individual reads (through the contains()) method for every potential key in the versification.
> >
> > public final Key getGlobalKeyList() {
> > if (global == null) {
> > Versification v11n = Versifications.instance().getVersification(versification);
> > global = keyf.createEmptyKeyList(v11n);
> > Key all = keyf.getGlobalKeyList(v11n);
> > for (Key key : all) {
> > if (contains(key)) {
> > global.addAll(key);
> > }
> > }
> > }
> > return global;
> > }
> >
> > The contains() method above makes a backend call every time.
> >
> > Isn't there a way to obtain the list of keys from the index? I'm not quite sure where the versification comes in here, or is it there simply to map the names of the keys to their positions in the index file? If so, can we simply read all keys in OT and NT, and check
> >
> > Doing this on GlobalKeyList() would also speed up index creation.
> >
> > Finally, it seems contains() only works on a single key, or takes the first key/verse if it's a passage. Presumably that is correct?
> >
> > Chris
> >
> > _______________________________________________
> > jsword-devel mailing list
> > jsword-devel at crosswire.org
> > http://www.crosswire.org/mailman/listinfo/jsword-devel
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/jsword-devel/attachments/20130215/0b02982f/attachment-0001.html>
More information about the jsword-devel
mailing list