[jsword-devel] Getting the global key list for a book
Chris Burrell
chris at burrell.me.uk
Fri Feb 15 14:31:34 MST 2013
SO here's the method I came up with, with one uncertainty about whether
maxIndex should be "-1" or not, but without it, i seem to be reading too
far.
public Key getFastGlobalKeyList() throws BookException {
ZVerseBackendState rafBook = null;
try {
rafBook = initState();
String v11nName =
getBookMetaData().getProperty(ConfigEntryType.VERSIFICATION).toString();
Versification v11n =
Versifications.instance().getVersification(v11nName);
Testament[] testaments = new Testament[] {
Testament.OLD, Testament.NEW
};
Key globalList =
PassageKeyFactory.instance().createEmptyKeyList(v11n);
Passage passage = KeyUtil.getPassage(globalList, v11n);
BitwisePassage bitwisePassage = null;
if(passage instanceof BitwisePassage) {
bitwisePassage = (BitwisePassage) passage;
bitwisePassage.raiseEventSuppresion();
bitwisePassage.raiseNormalizeProtection();
}
for (Testament currentTestament : testaments) {
RandomAccessFile compRaf = currentTestament ==
Testament.NEW ? rafBook.getNtCompRaf() : rafBook.getOtCompRaf();
// If Bible does not contain the desired testament, then
false
if (compRaf == null) {
// no keys in this testament
continue;
}
int maxIndex = v11n.getCount(currentTestament) -1;
// Read in the whole index, a few hundred Kb at most.
byte[] temp = SwordUtil.readRAF(compRaf, 0, COMP_ENTRY_SIZE
* maxIndex);
// for each block of 10 bytes, we consider the last 2 bytes.
for (int ii = 0; ii < temp.length; ii += COMP_ENTRY_SIZE) {
// can this be simplified to temp[8] == 0 && temp[9] ==
0?
int verseSize = SwordUtil.decodeLittleEndian16(temp, ii
+ 8);
//can this be optimized even further - i.e. why
decodeOrdinal, when add() go simply pass in and store an ordinal
if (verseSize > 0) {
int ordinal = ii / COMP_ENTRY_SIZE;
if(bitwisePassage != null) {
bitwisePassage.addVersifiedOrdinal(ordinal);
} else {
globalList.addAll(v11n.decodeOrdinal(ordinal));
}
}
}
}
if(bitwisePassage != null) {
bitwisePassage.lowerNormalizeProtection();
bitwisePassage.lowerEventSuppressionAndTest();
}
return globalList;
} catch (IOException e) {
throw new BookException(JSMsg.gettext("Unable to read key list
from book."));
} finally {
IOUtil.close(rafBook);
}
}
On 15 February 2013 20:26, Chris Burrell <chris at burrell.me.uk> wrote:
> Hi DM
>
> I'm about to test the following method for at least all Z backends:
>
> Maybe you can tell me what you think of it:
>
> *getFastGlobalKeyList*
>
> public Key getFastGlobalKeyList() throws BookException {
> ZVerseBackendState rafBook = null;
> try {
> rafBook = initState();
>
> String v11nName =
> getBookMetaData().getProperty(ConfigEntryType.VERSIFICATION).toString();
> Versification v11n =
> Versifications.instance().getVersification(v11nName);
>
> Testament[] testaments = new Testament[] {
> Testament.OLD, Testament.NEW
> };
>
> Key globalList =
> PassageKeyFactory.instance().createEmptyKeyList(v11n);
> Passage passage = KeyUtil.getPassage(globalList, v11n);
> BitwisePassage bitwisePassage = null;
> if(passage instanceof BitwisePassage) {
> bitwisePassage = (BitwisePassage) passage;
> bitwisePassage.raiseEventSuppresion();
> bitwisePassage.raiseNormalizeProtection();
> }
>
>
> for (Testament currentTestament : testaments) {
> RandomAccessFile compRaf = currentTestament ==
> Testament.NEW ? rafBook.getNtCompRaf() : rafBook.getOtCompRaf();
>
> // If Bible does not contain the desired testament, then
> false
> if (compRaf == null) {
> // no keys in this testament
> continue;
> }
>
> int maxIndex = v11n.getCount(currentTestament);
>
> // Read in the whole index, a few hundred Kb at most.
> byte[] temp = SwordUtil.readRAF(compRaf, 0,
> COMP_ENTRY_SIZE * maxIndex);
>
> // for each block of 10 bytes, we consider the last 2
> bytes.
> for (int ii = 0; ii < temp.length; ii += 10) {
> // can this be simplified to temp[8] == 0 && temp[9]
> == 0?
> int verseSize = SwordUtil.decodeLittleEndian16(temp,
> 8);
>
> //can this be optimized even further - i.e. why
> decodeOrdinal, when add() go simply pass in and store an ordinal
> if (verseSize > 0) {
> if(bitwisePassage != null) {
> bitwisePassage.addVersifiedOrdinal(ii % 10);
> } else {
> globalList.addAll(v11n.decodeOrdinal(ii % 10));
> }
> }
> }
> }
>
> if(bitwisePassage != null) {
> bitwisePassage.lowerNormalizeProtection();
> bitwisePassage.lowerEventSuppressionAndTest();
> }
>
> return globalList;
> } catch (IOException e) {
> throw new BookException(JSMsg.gettext("Unable to read key list
> from book."));
> } finally {
> IOUtil.close(rafBook);
> }
> }
>
>
> *addVersifiedOrdinal (is just a simplification of add, so that we can
> give the ordinals directly, rather than converting to a verse and back
> again).*
> /**
> * A shortcut to adding a key, by ordinal. The ordinal needs to be
> taken from the same versification as the passage being created.
> *
> * @param ordinal the ordinal
> */
> public void addVersifiedOrdinal(int ordinal) {
> Versification v11n = getVersification();
> optimizeWrites();
>
> store.set(ordinal);
>
> // we do an extra check here because the cost of calculating the
> // params is non-zero and may be wasted
> if (suppressEvents == 0) {
> Verse verse = v11n.decodeOrdinal(ordinal);
> fireIntervalAdded(this, verse, verse);
> }
> }
>
>
>
> On 15 February 2013 20:22, DM Smith <dmsmith at crosswire.org> wrote:
>
>> That's what http://www.crosswire.org/tracker/browse/JS-246 is all about.
>>
>> There are a good number of places that we get all the keys for a module.
>> But the construction of the list is dumb.
>>
>> The question is what do you want? Do you want all the possible keys in a
>> versification? Or do you want the keys for the verses that are actually in
>> a module?
>>
>> If you only want chapters that exist, consider working off of
>> Versification to get the book names and the chapters. For each chapter, get
>> the number of verses and grab one (maybe verse 1 or one halfway into the
>> chapter) on the assumption that if one exists then the rest do.
>>
>>
>> Calling book.contains(key) is what you want to do. It goes against the
>> backend to determine whether the verse is in the module.
>>
>> Don't call book.getGlobalKeyList(); We probably should deprecate the
>> method. It is too easy to call it and it is very expensive for what it does.
>>
>> I have a 3 day weekend and hope to finish the av11n work. It impacts
>> this. Will probably add iterators for a versification.
>>
>> In Him,
>> DM
>>
>>
>>
>> On Feb 15, 2013, at 2:14 PM, Chris Burrell <chris at burrell.me.uk> wrote:
>>
>> > Hi all
>> >
>> > getGlobalKeyList() seems to do a lot of work against the file system.
>> >
>> > I'm attempting to create a sitemap, which includes a URL to every
>> chapter. Presumably getGlobalKeyList is what I want to ensure that I'm only
>> displaying those chapters that exist. I was intending to do:
>> >
>> > globalKeyList = getGlobalKeyList();
>> > for(each book in versification) {
>> > myBookKey = book.getValidKey(book.getShortName())
>> > myBookKey.retainAll(globalKeyList);
>> >
>> > if(myBookKey.getCardinality() > 0) {
>> > outputSiteMapChapter();
>> > }
>> > }
>> >
>> > So for each version I call getGlobalKeyList(), but that in terms seem
>> to make individual reads (through the contains()) method for every
>> potential key in the versification.
>> >
>> > public final Key getGlobalKeyList() {
>> > if (global == null) {
>> > Versification v11n =
>> Versifications.instance().getVersification(versification);
>> > global = keyf.createEmptyKeyList(v11n);
>> > Key all = keyf.getGlobalKeyList(v11n);
>> > for (Key key : all) {
>> > if (contains(key)) {
>> > global.addAll(key);
>> > }
>> > }
>> > }
>> > return global;
>> > }
>> >
>> > The contains() method above makes a backend call every time.
>> >
>> > Isn't there a way to obtain the list of keys from the index? I'm not
>> quite sure where the versification comes in here, or is it there simply to
>> map the names of the keys to their positions in the index file? If so, can
>> we simply read all keys in OT and NT, and check
>> >
>> > Doing this on GlobalKeyList() would also speed up index creation.
>> >
>> > Finally, it seems contains() only works on a single key, or takes the
>> first key/verse if it's a passage. Presumably that is correct?
>> >
>> > Chris
>> >
>> > _______________________________________________
>> > jsword-devel mailing list
>> > jsword-devel at crosswire.org
>> > http://www.crosswire.org/mailman/listinfo/jsword-devel
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/jsword-devel/attachments/20130215/cdf78aca/attachment-0001.html>
More information about the jsword-devel
mailing list