[sword-devel] Too many milestones can cook your buffer?
DM Smith
dmsmith at crosswire.org
Mon May 5 11:50:03 EDT 2025
The BlockType, if I remember correctly, is the unit of compression/retrieval from the module and the size of the block is constrained (to what size I don’t remember) to prevent “integer” overflow in the index.
For a given input, the larger the block size of what is being compressed, the smaller the total size of the module. Since most SWORD applications show a chapter at a time, it is a better unit for retrieval. If you were to have a commentary using the module format, you might need VERSE as your unit.
> On May 2, 2025, at 10:55 PM, David Haslam <dfhdfh at protonmail.com> wrote:
>
> Update:
>
> Rebuilding the module with BlockType=CHAPTER solved the buffer issue. Hurrah!
>
> The cmd file now reads:
> cd ..
> xiphos\imp2vs Import\IMP\KhmerNTw.raw.imp.txt -z -b 3 -o .\modules\texts\ztext\khmerntw 1>log\imp2vs.log 2>log\error.log
>
> pause
>
> Useful lesson learned about module building!
> cf. This is the first time ever that I have needed to use a BlockType smaller than BOOK.
>
> Aside: Now that I am using the milestone marker form of the ZWSP, diatheke starts each lexical Khmer word on a new line, though Xiphos displays the verse text correctly!
>
> Any ideas towards an explanation?
>
> Best regards,
>
> David
>
> Sent with Proton Mail <https://pr.tn/ref/SWXT9A5YZ67G> secure email.
>
> On Friday, May 2nd, 2025 at 4:48 PM, David Haslam <dfhdfh at protonmail.com> wrote:
>> ztext
>>
>> The module was built using the following cmd file stored in a directory called Make under my SWORD path.
>>
>> cd ..
>> xiphos\imp2vs Import\IMP\KhmerNTw.raw.imp.txt -z -o .\modules\texts\ztext\khmerntw 1>log\imp2vs.log 2>log\error.log
>>
>> pause
>>
>> Best regards,
>>
>> David
>>
>> Sent with Proton Mail <https://pr.tn/ref/SWXT9A5YZ67G> secure email.
>>
>> On Friday, May 2nd, 2025 at 4:35 PM, Greg Hellings <greg.hellings at gmail.com> wrote:
>>> Which driver have you used? RawText? zText? That might influence a buffer size.
>>>
>>> On Fri, May 2, 2025, 9:07 AM David Haslam <dfhdfh at protonmail.com <mailto:dfhdfh at protonmail.com>> wrote:
>>>> While experimenting with the KhmerNT module eXperimental improvement, I encountered a new issue after creating modules using imp2vs
>>>>
>>>> Earlier today, I had the "bright idea" of replacing all the ZWSP between Khmer words with the following OSIS element. NB. The marker attribute contains a ZWSP character
>>>>
>>>> <milestone marker="" type="x-lexical-word-divider" subtype="x-ZWSP"/>
>>>>
>>>> Nothing wrong with that, you might assume, like I did.
>>>> It's perfectly valid OSIS according to our schema and wiki.
>>>> It's also much more explicit than simply leaving the ZWSP characters invisible to the naked eye.
>>>>
>>>> Only a few smaller books in the NT ever got compiled!
>>>> Or if they did compile OK, afterwards SWORD cannot read parts of them.
>>>>
>>>> So, I thinks to myself, "let's make it a bit smaller", and I then tried again with this abbreviated attribute value.
>>>>
>>>> <milestone marker="" type="x-LWD" subtype="x-ZWSP"/>
>>>>
>>>> A few more NT books were compiled this time, but the largest one visible was Hebrews.
>>>> The apparently missing books just show as empty verses in Xiphos and with emptyvss.
>>>>
>>>> I have attached the log file generated during the second attempt at module build.
>>>> NB. The error.log was empty, which now rather makes me think it's a SWORD output issue.
>>>>
>>>> Using diatheke and mod2imp after the module was built, one or either of these gave this repeated message:
>>>>
>>>> ERROR: not enough room in the out buffer during decompression.
>>>>
>>>> Evidently "Too many milestones can cook your buffer?"
>>>>
>>>> I'd be happy to supply the IMP file and/or the module for further investigation.
>>>> I think this is a serious issue. I'm surprised that it's never been encountered hitherto.
>>>>
>>>> Best regards,
>>>>
>>>> David
>>>>
>>>> Sent with Proton Mail <https://pr.tn/ref/SWXT9A5YZ67G> secure email.
>>>> _______________________________________________
>>>> sword-devel mailing list: sword-devel at crosswire.org <mailto:sword-devel at crosswire.org>
>>>> http://crosswire.org/mailman/listinfo/sword-devel
>>>> Instructions to unsubscribe/change your settings at above page
>>
>
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://crosswire.org/pipermail/sword-devel/attachments/20250505/641d5d60/attachment.htm>
More information about the sword-devel
mailing list