[sword-devel] HowTo: create ztext module?

[L.Allan-pbio]

DM Smith wrote:
> On May 8, 2006, at 8:36 PM, Greg Hellings wrote:
>
>>
>> This brings up another interesting question (in my opinion).  Why 
>> are
>> there several standard modules which are distributed without
>> compression?  Things like the the ASV, the Vulgate and the WEB are
>> all distributed in uncompressed format.  Might it be beneficial for
>> us to zip those up (especially the ASV and WEB, which I would
>> imagine are both popular modules?) and distribute them in a ztext
>> format?  Are there any advantages to having them in rawtext rather
>> than ztext, except for minor performance advantages?  Just curious!
>
> In my opinion ztext serves two purposes:
> 1) Since it is not a simple compression of the files (i.e. you can't
> run unzip on them) but an internal compression of parts of the file,
> it raises the importance of the SWORD api in accessing them,
> providing a bit of information hiding.
> 2) The SWORD api downloads the files individually, and having
> compressed files improves download performance.
>
> One may argue that it also reduces the disk footprint, which it 
> does.
> But in today's world of large disks (even my old win 95 laptop has a
> 20G drive), I don't think that is much of an issue.
>
> The drawback is that the client application needs to uncompress the
> data on the fly and it does so into memory. Most use book 
> compression
> so the entire book needs to be unzipped. Using the principle of
> locality, caching the most recent book's uncompression results in
> reasonable performance, since most requests for a verse are within
> the same book as the last verse requested. The anomalous case is the
> returning of a ranked verses for search of a common word. (Ranked
> searches is the normal behavior of Lucene).
>
> I can appreciate a goal of encouraging the use of the SWORD api for
> access of a module's content, whether it was a deliberate or an
> accidental goal, but there are other, simple ways to achieve
> information hiding that don't have the performance penalty.
>
> While not simpler, there are ways to compress the files that don't
> use stream compression such that each verse can be handled
> independently.
>
> With regard to compression of the download file, I think that there
> are better ways to handle it as well. Rather than downloading the
> parts individually, we could change the installer to download a zip
> file of the entire contents.
>
> As a side note, JSword was changed to download the raw zips rather
> than the parts because ftp within Java stopped working after a
> security patch on Win XP and it did not work behind a proxy on a 
> Mac.
> Now we use http tunneling. The side benefit was that the downloads
> were faster, more reliable and when they failed, the cleanup was
> simpler. Also, the code was a bit simpler as well.
>
> _______________________________________________
> sword-devel mailing list: sword-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/sword-devel
> Instructions to unsubscribe/change your settings at above page