[sword-devel] File Compression (Was: Making Import Easier)

[DM Smith]

Greg Hellings wrote:
> Mike,
>
> On Mon, Apr 6, 2009 at 4:45 PM, Mike Hart <just_mike_y at yahoo.com> wrote:
>   
>> On this subject, why are ALL modules (especially beta mods) compressed? Once upon a time, compression made the world of POTS (telephone) based file transfer possible.. turning a 2 hour 5 meg download into a 20 minute download made sense.  Today with the exception of very low end mobile devices, the need for compression is zero, and the compression used by Sword is rather CPU intensive.  CPU speeds are driving what you can do with a given computer far more often than disk space. It would make more sense to optimize the compression for speed rather than maximum compression in cases where it is required, and to eliminate it altogether where possible.
>>
>> For final modules, it does make some sense to compress as a discouragement to make changes to other's work.  On beta modules however, the ability to 'view the source' to troubleshoot is in the hands of the few.  Considering the number of works in beta (and the trend is up), being able to do more end user troubleshooting is a good idea.
>>
>> I'm saying this in the hopes that an 'import' menu option outputs a rawtext module, not a zipped file.  Also, the companion 'compress' option as a separate menu item should be accompanied by an 'uncompress.' (with the ability to lock out 'unmodifiable' or non-beta modules.)
>>     
>
>
> It remains a fact that a large portion of the world does not have
> high-bandwidth download options nor does it have modern computers.  So
> we're still trying to accommodate people who are on POTS or slower
> systems with possibly tiny hard drives in remote areas of the world (a
> supposedly still significant portion of the clients of SWORD are in
> this scenario).
>
> I think there is a utility, something like zmod2mod or whatever, that
> will uncompress the module files, if that's what you're talking about
> (if there isn't, there *should* be one).  If you're talking about the
> .zip files that are downloaded from the website, that's because we're
> sending a directory structure of files through to the client instead
> of just a single file.
>
> --Greg
>   
I don't know if there is such a thing, but it is easy to acheive:

mod2imp will create SWORD's imp format for the module.
This can be rebuilt with imp2mod.

I think this should be noted on the beta page, as it is useful to have 
uncompressed modules for testing. There are a variety of tools that will 
do a raw lookup of a keyed entry. (Lookup is one.)


Other reasons for compression, once we get a module out of beta, is to 
take less space on the SWORD cd and to take less bandwidth on downloading.

In beta, some of the modules are very large and highly compressible. 
These at least should be compressed.

In Him,
    DM
>   
>> --- On Mon, 4/6/09, Greg Hellings <greg.hellings at gmail.com> wrote:
>>
>>     
>>> From: Greg Hellings <greg.hellings at gmail.com>
>>> Subject: [sword-devel] Making Import Easier
>>> To: "SWORD Developers' Collaboration Forum" <sword-devel at crosswire.org>
>>> Date: Monday, April 6, 2009, 4:19 PM
>>> DM and others involved in the import process --
>>>
>>> I was just chatting with Matthew Talbert about the process
>>> of making
>>> module import a much easier task than it is now.
>>> Currently, most of
>>> us are familiar with the process we go through when a new
>>> person tries
>>> to make a module.  They often fall into one of a few
>>> categories: they
>>> can't find the utilities because they're not
>>> packaged in their
>>> distro's libsword package, they can't use a command
>>> line comfortable,
>>> they have trouble finding the Windows build of the
>>> binaries, etc.
>>>
>>> Obviously it's possible for the front-ends to have the
>>> ability to
>>> create new modules and module content -- they do it with
>>> the Personal
>>> commentary, and I believe that Xiphos even allows an
>>> arbitrary number
>>> of personal commentaries to be created.  Wouldn't it be
>>> nice if it was
>>> easy for a module to be created programmatically from a GUI
>>> interface?
>>>  Why shouldn't a front-end have the ability to be
>>> pointed to a file on
>>> the user's hard drive (or even remotely to a web-based
>>> resource or any
>>> other number of resources) and have them press a button and
>>> voila!
>>> their module appears (Handling the config file wouldn't
>>> be terribly
>>> more difficult than any other form-based fill-in.  The
>>> frontend could
>>> even have predefined values if the developers wanted, etc)
>>> in their
>>> SWORD directories wherever they specify/have write
>>> permissions.
>>>
>>> Implementing this should come for almost free with a minor
>>> change set
>>> to the utilities/ directory.  Currently, AFAIK, most of the
>>> import
>>> utilities are written in a single .cpp file per utility,
>>> which
>>> includes a main() function.  If those utilities were
>>> refactored so
>>> that the option handling and file reading was in one file
>>> and the
>>> actual text processing and module writing were in a second
>>> file, that
>>> would allow those second files to be included directly into
>>> the
>>> library and be linked against by any front end.  The front
>>> end would
>>> then be able to read the file from whatever source it wants
>>> (local,
>>> web, ftp, gopher, ocr, whatever) and feed it into the same
>>> functions
>>> that are used by the official import utilities.  Then any
>>> front end
>>> who wants to play nice to the user like this would not have
>>> to
>>> reduplicate the efforts of the current utilities, nor would
>>> they have
>>> to hack the code out of the SWORD source and shoe horn it
>>> directly
>>> into their own application.
>>>
>>> Am I actually near the mark? Would this be something that
>>> could be
>>> easily handled without being too much work (and, if it was
>>> handled,
>>> would it be used by any of the front ends?)?  I think it
>>> would be a
>>> very user-friendly thing to have available, but it would
>>> take work
>>> from someone to refactor the utilities directory and also
>>> from the
>>> front end developers to build support for this into their
>>> applications.  I know I've been carrying the pumpkin
>>> for mod2osis for
>>> most of the recent work on it, and would be happy to make
>>> these
>>> changes to that program -- it wouldn't be the best one
>>> to include in
>>> this effort, since there's lots of work it still needs
>>> to convert
>>> non-OSIS original modules into its format well, but
>>> it's the only such
>>> utility that I'm closely familiar with.
>>>
>>> Thoughts, comments, accusations?
>>>
>>> --Greg