[sword-devel] using encryption with current osis2mod (r2435

[Jonathan Marsden]

Ben Morgan wrote:

> Then there is the question of why Jonathan has it working... Are you
> sure cipherraw hasn't been near the module? 

What is cipherraw?  If it is another SWORD utility, then, yes, I'm sure. 
  See below.

Only osis2mod (well, and zip and unzip, and diatheke to read a verse, 
once installed) have touched the modules I create during my tests, and 
this can be verified by reading the little script I'm using.  (One of 
the nice things about scripting tests is that they can be checked to see 
exactly what they are doing, in addition to being more easily repeatable).

 > Unmodified code from SVN?

Yes.  Or just whatever version of osis2mod is installed on the machine 
being tested.

Better yet, anyone with a (bash) shell account on crosswire.org should 
be able to duplicate my results there.  That is, we can all use the 
existing installed /usr/bin/osis2mod (I have no rights to install stuff 
into /usr/bin on crosswire.org, of course, so I'm testing with whatever 
was already in place, installed by someone on the SWORD team) which says 
is it r2337, and see that encryption works there.

A one line fix to my earlier make-kjvfull.sh script, so it works when 
osis2mod is too old to handle a - for the input filename, is to replace 
the line that runs osis2mod with

osis2mod `pwd`/modules/texts/ztext/$MODULENAMELC/ <(unzip -p 
$OLDPWD/kjvxml.zip kjvfull.xml ) -z $@

The above is one line of the script, and just uses bash process 
substitution instead of piping (and so avoids the - parameter to 
osis2mod).  I have updated the version at

   http://crosswire.org/~jmarsden/make-kjvfull.sh

so everyone can grab it and use exactly the same script I am using, so 
our tests should be comparable (in that regard at least).

With that small change to the script for backward compatibility, on the 
crosswire.org server itself, logged in as jmarsden, doing

   MODULENAME=KJV1 ./make-kjvfull.sh
   MODULENAME=KJV2 ./make-kjvfull.sh -c secret

creates two very different modules, in my testing at least.  I have made 
these available for download at

   http://crosswire.org/~jmarsden/KJV1.zip
   http://crosswire.org/~jmarsden/KJV2.zip

so others can check the binary file contents of them as needed, in any 
way they choose.  I can't make sense of encrypted hex dumps of 
compressed SWORD module data, so looking at it isn't all that helpful to 
me, and that's not a very automatable or reproducible test.

Instead, I tried using cmp and diff, and both confirm the ot.bzz and 
nt.bzz files differ, but that the nt.bzs, nt.bzv, ot.bzs and ot.bzv 
files are the same in KJV1 and KJV2.  I also checked that the md5sum and 
sha1sum values of the *.bzz files differ, but those of the others are 
the same, between corresponding files in the KJV1 and KJV2 results.

My test for encryptedness is that, after cleaning out ~/.sword (just in 
case anything in there might cause lack of repeatability somehow)

   rm -rf ~/.sword

and then installing both:

   mkdir -p ~/.sword
   unzip -qd ~/.sword KJV1.zip
   unzip -qd ~/.sword KJV2.zip

trying to display John 3:16 from each one

   diatheke -b KJV1 -k Jn 3:16
   diatheke -b KJV2 -k Jn 3:16

causes different results (KJV1 works, because it is unencrypted, and 
KJV2 fails (presumably because it is encrypted) with "ERROR: corrupt 
data during decompression.").

Lastly, as a further test to verify the error is a result of encryption, 
and not some other peculiar problem being accidentally triggered, doing

   echo CipherKey=secret >>~/.sword/mods.d/kjv2.conf
   diatheke -b KJV2 -k Jn 3:16

causes the error to go away, but using any value other than secret for 
the configured key does not change the error.

If anyone can see a problem with or flaw in this process, other than 
that bash shell scripting isn't directly portable to MS Windows unless 
you use Cygwin or GNUwin32 (!), please do speak up.  Let's be methodical 
and get to the cause of this.

If I feel suitably inspired, I may full automate the entire sequence:

   MODULENAME=KJV1 ./make-kjvfull.sh
   MODULENAME=KJV2 ./make-kjvfull.sh -c secret
   rm -rf ~/.sword
   mkdir -p ~/.sword
   unzip -qd ~/.sword KJV1.zip
   unzip -qd ~/.sword KJV2.zip
   diatheke -b KJV1 -k Jn 3:16
   diatheke -b KJV2 -k Jn 3:16
   echo CipherKey=secret >>~/.sword/mods.d/kjv2.conf
   diatheke -b KJV2 -k Jn 3:16

with appropriate tests on the expected diatheke output each time; at 
that point, this could be pretty close to being useful as an automated 
regression test script (I think).

Jonathan