[sword-devel] Q about compressed modules

DM Smith dmsmith at crosswire.org
Sat Jan 27 08:21:56 MST 2018


Technical answer:
Using osis2mod to do compression is easiest. It does compression as it builds the module.

As Michael noted, we compressed all of the Bible and commentary modules in the CrossWire repository. We did this with mod2zmod.

The two use identical compression, but they are not identical in results in some corner cases.

When osis2mod creates a module it appends data to the data file. One characteristic of building modules is that the input file does not have to be in order. The index is always in v11n order and the data file is always in the order in which the verses are provided.

The -a (append) allows for building a module incrementally and for correcting verses. When used for correcting verses in an uncompressed module, it will put the new verse at the end of the data file and update the entry in the index file to point to it. It does not remove the old text. A compressed module is more complicated, but it likewise does not replace the corrected verse. I think, but am not sure, that it duplicates the entire compressed block in which the verse resides before appending.

The other corner case is when a verse is not in the module’s v11n. This will append the verse to what’s already in the module. It does this by getting the old text out, appending the additional text to it and then writing that to the module, updating the index. Osis2mod has been optimized to “write later” the prior verse to minimize this duplication in the data file.

Many utilities can only read from a repository known to the SWORD engine. There are a few ways to make a repository be seen. From memory, it will look for SWORD_PATH first, then the current folder, then well-known locations for the mods.d folder. It will use the first that it finds. Then it will look for the module’s conf by the provided name. That file gives the relative location of the module’s files.

mod2zmod will read the verses from the module and will create a new module with the indicated compression. I’ve not used it in years, so I don’t know where it puts the new module. At CrossWire we have a convention where the new module needs to be placed. There’s a cleanup that needs to take place afterwards.

From memory, mod2zmod has a corner case that will not create the same index or data file. This is especially true of commentary modules. If a “verse” is a range (e.g. Matt.1.1-Matt.1.5), osis2mod will put the data into the module once and have the different index slots point to the same offset and length. But mod2zmod will replicate the content. In this case, 5 times, with the index of these 5 pointing to the different copies. It would be a good optimization to change this to save the last index and not duplicate the content (hopefully it has been done already).

I prefer osis2mod.

In Him,
	DM

> On Jan 26, 2018, at 5:49 PM, John Dudeck <john.dudeck at sim.org> wrote:
> 
> In the Wiki, on the http://wiki.crosswire.org/DevTools:Modules page it gives directions for using mod2zmod for compressing modules.
> 
> However osis2mod.exe has command-line switches for creating compressed modules.
> 
> My question is: do these produce equivalent results? Which is preferred?
> 
> For GenBooks, is mod2zmod the only way to compress them?
> 
> Related question re mod2zmod. When it says "First you will need to install the module so that it can be accessed using the SWORD engine", exactly what does this mean? I am building each module in its own branch of an svn repository, and would like to create the compressed module in that tree. This is working fine using osis2mod. Is there a way to do the same with mod2zmod?
> 
> Thanks and sorry for the newby questions.
> 
> John Dudeck
> Programmer at Editions Cle                             Lyon, France
> john.dudeck at sim.org                            john at editionscle.com
> --
> Sign in Swiss restaurant:
> "Our wines leave you nothing to hope for."
>   
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20180127/b065e170/attachment.html>


More information about the sword-devel mailing list