[sword-devel] Too many milestones can cook your buffer?

DM Smith dmsmith at crosswire.org
Mon May 5 11:50:03 EDT 2025


The BlockType, if I remember correctly, is the unit of compression/retrieval from the module and the size of the block is constrained (to what size I don’t remember) to prevent “integer” overflow in the index.

For a given input, the larger the block size of what is being compressed, the smaller the total size of the module. Since most SWORD applications show a chapter at a time, it is a better unit for retrieval. If you were to have a commentary using the module format, you might need VERSE as your unit.

> On May 2, 2025, at 10:55 PM, David Haslam <dfhdfh at protonmail.com> wrote:
> 
> Update:
> 
> Rebuilding the module with BlockType=CHAPTER solved the buffer issue. Hurrah!
> 
> The cmd file now reads:
> cd ..
> xiphos\imp2vs Import\IMP\KhmerNTw.raw.imp.txt -z -b 3 -o .\modules\texts\ztext\khmerntw  1>log\imp2vs.log 2>log\error.log
> 
> pause
> 
> Useful lesson learned about module building!
> cf. This is the first time ever that I have needed to use a BlockType smaller than BOOK.
> 
> Aside: Now that I am using the milestone marker form of the ZWSP, diatheke starts each lexical Khmer word on a new line, though Xiphos displays the verse text correctly!
> 
> Any ideas towards an explanation?
> 
> Best regards,
> 
> David
> 
> Sent with Proton Mail <https://pr.tn/ref/SWXT9A5YZ67G> secure email.
> 
> On Friday, May 2nd, 2025 at 4:48 PM, David Haslam <dfhdfh at protonmail.com> wrote:
>> ztext
>> 
>> The module was built using the following cmd file stored in a directory called Make under my SWORD path.
>> 
>> cd ..
>> xiphos\imp2vs Import\IMP\KhmerNTw.raw.imp.txt -z -o .\modules\texts\ztext\khmerntw  1>log\imp2vs.log 2>log\error.log
>> 
>> pause
>> 
>> Best regards,
>> 
>> David
>> 
>> Sent with Proton Mail <https://pr.tn/ref/SWXT9A5YZ67G> secure email.
>> 
>> On Friday, May 2nd, 2025 at 4:35 PM, Greg Hellings <greg.hellings at gmail.com> wrote:
>>> Which driver have you used? RawText? zText? That might influence a buffer size.
>>> 
>>> On Fri, May 2, 2025, 9:07 AM David Haslam <dfhdfh at protonmail.com <mailto:dfhdfh at protonmail.com>> wrote:
>>>> While experimenting with the KhmerNT module eXperimental improvement, I encountered a new issue after creating modules using imp2vs
>>>> 
>>>> Earlier today, I had the "bright idea" of replacing all the ZWSP between Khmer words with the following OSIS element. NB. The marker attribute contains a ZWSP character
>>>> 
>>>> <milestone marker="​" type="x-lexical-word-divider" subtype="x-ZWSP"/>
>>>> 
>>>> Nothing wrong with that, you might assume, like I did.
>>>> It's perfectly valid OSIS according to our schema and wiki.
>>>> It's also much more explicit than simply leaving the ZWSP characters invisible to the naked eye.
>>>> 
>>>> Only a few smaller books in the NT ever got compiled! 
>>>> Or if they did compile OK, afterwards SWORD cannot read parts of them.
>>>> 
>>>> So, I thinks to myself, "let's make it a bit smaller", and I then tried again with this abbreviated attribute value.
>>>> 
>>>> <milestone marker="​" type="x-LWD" subtype="x-ZWSP"/>
>>>> 
>>>> A few more NT books were compiled this time, but the largest one visible was Hebrews.
>>>> The apparently missing books just show as empty verses in Xiphos and with emptyvss.
>>>> 
>>>> I have attached the log file generated during the second attempt at module build.
>>>> NB. The error.log was empty, which now rather makes me think it's a SWORD output issue.
>>>> 
>>>> Using diatheke and mod2imp after the module was built, one or either of these gave this repeated message:
>>>> 
>>>> ERROR: not enough room in the out buffer during decompression.
>>>> 
>>>> Evidently "Too many milestones can cook your buffer?"
>>>> 
>>>> I'd be happy to supply the IMP file and/or the module for further investigation.
>>>> I think this is a serious issue. I'm surprised that it's never been encountered hitherto.
>>>> 
>>>> Best regards,
>>>> 
>>>> David
>>>> 
>>>> Sent with Proton Mail <https://pr.tn/ref/SWXT9A5YZ67G> secure email.
>>>> _______________________________________________
>>>> sword-devel mailing list: sword-devel at crosswire.org <mailto:sword-devel at crosswire.org>
>>>> http://crosswire.org/mailman/listinfo/sword-devel
>>>> Instructions to unsubscribe/change your settings at above page
>> 
> 
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://crosswire.org/pipermail/sword-devel/attachments/20250505/641d5d60/attachment.htm>


More information about the sword-devel mailing list