[sword-devel] linking / testsuite

[Jonathan Marsden]

Ben Morgan wrote:

> If you have 66 books of the Bible and many different ways of
> referencing them, testing every single way isn't necessarily all that
> easy. Then multiply that by every locale that is there.

A regression test doesn't (usually) try to give complete coverage of 
anything, just enough tests to convince the team that functionality that 
is now "correct" does not later become "incorrect" without someone 
quickly noticing the change.

In this case, testing (say) 3 common abbreviations for each book name 
followed by the first verse of the first chapter, for each of the 66 
books, is 66 * 3 = 198 tests, and just using a single very boring locale 
(either C or en_US.UTF-8) would catch what happened recently -- and so 
would notify developers by failing if the change were to accidentally 
disappear (the wrong revision is tagged/checked out of svn by mistake, 
say) between now and the official 1.6 release, for example.

In practice, a really "good" regression test for abbreviations might 
include a couple of other locales too, perhaps one Western European one 
that needs a few accented Latin characters (French or German or Spanish, 
say), and one that needs its own non-Roman alphabet (Cyrillic, or 
Korean, or Arabic, maybe).

At this level of coverage, we'd be looking at a maximum of 66 * 3 * 3 = 
under 600 test items.  That's entirely doable, surely?

Beyond that, subject matter experts for each locale might come up with 
"special" "known to be peculiar" abbreviations they feel it would be 
good to add as additional tests... but that's moving away from the realm 
of basic regression testing, really, I think.

If I have time later tonight, once I have a sword 1.6.0RC2 package out 
for front end developers to test with, I'll see about creating the 198 
item single locale test and (if I succeed!) I'll contribute it to the 
SWORD codebase, just to demonstrate that creating such a test really is 
in fact doable :)

> I think the vk1 = vk2 change is
> more important to test than the other, really.

You provided no basis for this opinion.  I think that *any* significant 
change that happens after an rc release is worthy of a regression test, 
more or less by definition.  My reasoning: the developers thought the 
codebase was good enough to release, and it turned out not to be.  This 
most likely means that the bugs found were unexpected or subtle or hard 
to test for, and yet were deemed significant enough to warrant fixing 
before final release.  Therefore, surely, each such bug deserves a test 
in the regression test suite.

Basically, bugs you find in an RC release are often the "creme de la 
creme" of bugs... treat them with respect :)

Jonathan