[sword-devel] Too many milestones can cook your buffer?

[David Haslam]

How does Dom’s module release script determine the right BlockType for a module?

On Mon, May 5, 2025 at 16:50, DM Smith <[dmsmith at crosswire.org](mailto:On Mon, May 5, 2025 at 16:50, DM Smith <<a href=)> wrote:

> The BlockType, if I remember correctly, is the unit of compression/retrieval from the module and the size of the block is constrained (to what size I don’t remember) to prevent “integer” overflow in the index.
>
> For a given input, the larger the block size of what is being compressed, the smaller the total size of the module. Since most SWORD applications show a chapter at a time, it is a better unit for retrieval. If you were to have a commentary using the module format, you might need VERSE as your unit.
>
>> On May 2, 2025, at 10:55 PM, David Haslam <dfhdfh at protonmail.com> wrote:
>>
>> Update:
>>
>> Rebuilding the module with BlockType=CHAPTER solved the buffer issue. Hurrah!
>>
>> The cmd file now reads:
>>
>>> cd ..
>>> xiphos\imp2vs Import\IMP\KhmerNTw.raw.imp.txt -z -b 3 -o .\modules\texts\ztext\khmerntw 1>log\imp2vs.log 2>log\error.log
>>> pause
>>
>> Useful lesson learned about module building!
>> cf. This is the first time ever that I have needed to use a BlockType smaller than BOOK.
>>
>> Aside: Now that I am using the milestone marker form of the ZWSP, diatheke starts each lexical Khmer word on a new line, though Xiphos displays the verse text correctly!
>>
>> Any ideas towards an explanation?
>>
>> Best regards,
>>
>> David
>>
>> Sent with [Proton Mail](https://pr.tn/ref/SWXT9A5YZ67G) secure email.
>>
>> On Friday, May 2nd, 2025 at 4:48 PM, David Haslam <dfhdfh at protonmail.com> wrote:
>>
>>> ztext
>>>
>>> The module was built using the following cmd file stored in a directory called Make under my SWORD path.
>>>
>>>> cd ..
>>>> xiphos\imp2vs Import\IMP\KhmerNTw.raw.imp.txt -z -o .\modules\texts\ztext\khmerntw 1>log\imp2vs.log 2>log\error.log
>>>> pause
>>>
>>> Best regards,
>>>
>>> David
>>>
>>> Sent with [Proton Mail](https://pr.tn/ref/SWXT9A5YZ67G) secure email.
>>>
>>> On Friday, May 2nd, 2025 at 4:35 PM, Greg Hellings <greg.hellings at gmail.com> wrote:
>>>
>>>> Which driver have you used? RawText? zText? That might influence a buffer size.
>>>>
>>>> On Fri, May 2, 2025, 9:07 AM David Haslam < dfhdfh at protonmail.com> wrote:
>>>>
>>>>> While experimenting with the KhmerNT module eXperimental improvement, I encountered a new issue after creating modules using imp2vs
>>>>>
>>>>> Earlier today, I had the "bright idea" of replacing all the ZWSP between Khmer words with the following OSIS element. NB. The marker attribute contains a ZWSP character
>>>>>
>>>>> <milestone marker="​" type="x-lexical-word-divider" subtype="x-ZWSP"/>
>>>>>
>>>>> Nothing wrong with that, you might assume, like I did.
>>>>> It's perfectly valid OSIS according to our schema and wiki.
>>>>> It's also much more explicit than simply leaving the ZWSP characters invisible to the naked eye.
>>>>>
>>>>> Only a few smaller books in the NT ever got compiled!
>>>>> Or if they did compile OK, afterwards SWORD cannot read parts of them.
>>>>>
>>>>> So, I thinks to myself, "let's make it a bit smaller", and I then tried again with this abbreviated attribute value.
>>>>>
>>>>> <milestone marker="​" type="x-LWD" subtype="x-ZWSP"/>
>>>>> A few more NT books were compiled this time, but the largest one visible was Hebrews.
>>>>> The apparently missing books just show as empty verses in Xiphos and with emptyvss.
>>>>>
>>>>> I have attached the log file generated during the second attempt at module build.NB. The error.log was empty, which now rather makes me think it's a SWORD output issue.
>>>>>
>>>>> Using diatheke and mod2imp after the module was built, one or either of these gave this repeated message:
>>>>>
>>>>> ERROR: not enough room in the out buffer during decompression.
>>>>>
>>>>> Evidently " Too many milestones can cook your buffer?"
>>>>> I'd be happy to supply the IMP file and/or the module for further investigation.
>>>>> I think this is a serious issue. I'm surprised that it's never been encountered hitherto.
>>>>>
>>>>> Best regards,
>>>>>
>>>>> David
>>>>>
>>>>> Sent with [Proton Mail](https://pr.tn/ref/SWXT9A5YZ67G) secure email.
>>>>> _______________________________________________
>>>>> sword-devel mailing list: sword-devel at crosswire.org
>>>>> http://crosswire.org/mailman/listinfo/sword-devel
>>>>> Instructions to unsubscribe/change your settings at above page
>>
>> _______________________________________________
>> sword-devel mailing list: sword-devel at crosswire.org
>> http://crosswire.org/mailman/listinfo/sword-devel
>> Instructions to unsubscribe/change your settings at above page
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://crosswire.org/pipermail/sword-devel/attachments/20250505/3c93c9a1/attachment.htm>