[bt-devel] Testing BibleTime

[Martin Gruner]

Hi Eeli.

I agree with you that we should test as much as we can. But not everything can 
be tested by humans all the time. =)

We need to distinguish between permanent, automatic low-level testing (usually 
called "unit testing" and high-level ("integration") testing.

Unit testing should be done automatically, all the time. I have added code for 
this already, but it is in a very early stage and needs some more careful 
design (Joachim, you are very welcome to share your expertise here). 
I am not sure but it may be that we can only test backend stuff easily with 
unit testing. Unit testing is good for testing boundary conditions, as you 
described. 
More work needs to be done here. I'll keep you posted when the test system is 
ready to be tested. =)
Whenever we go cross-platform, unit testing will be even more important.

Integration tests are more difficult and may require human interaction. This 
is what we need to describe in a wiki page maybe. Not sure how to deal with 
this. Maybe we can create a table listing the different items to be tested 
and have somebody perform these tests before each new release, at least.

Testing is one of the very weak areas of BibleTime, and we need to work hard 
to change this situation.

A book recommendation: 
http://www.pragprog.com/titles/pad/practices-of-an-agile-developer. 
Very good.

Regards,

mg

Am Samstag, 3. Mai 2008 schrieb Eeli Kaikkonen:
> On Fri, 2 May 2008, Martin Gruner wrote:
> > Cool! We should setup a wiki page for this.
> >
> > > It would be great if there was a list of different things to test... I
> > > build it periodically and fire it up, but I don't know exactly what
> > > needs to be tested.
>
> The first thing is to find all "visible" features and simply use them,
> for example find all buttons and context menu items and use them one by
> one. This is quite easy but still can help us because something may has
> gone unnoticed especially after changing something.
>
> Read carefully all UI strings and see if they are good English, easy to
> understand and consistent with other strings. List lacking tooltips.
>
> You can also read "Closed bugs, to be verified" list on our Prerelease
> Bugs wiki page and test them and by yourself pick up some related
> features and try to use them also. I have found many bugs by using some
> feature for some unrelated reason, playing around for a while and
> bumping to something else.
>
>
> Then there are more sophisticated cases: first do something, then do
> something else. What should be tested depends on external and internal
> factors. By external I mean for example having no modules or having all
> modules installed (the common limit values for testing: MIN, MAX, MIN-1,
> MIN+1, MAX-1, MAX+1, 0, 1, -1 if possible). By internal factors I mean
> that architecturally/functionally some things depend on some other
> things, as implemented in the code. "If I change this code, that one
> should be tested also."
>
> These sophisticated cases are hard to find and their amount could be
> easily several thousands. Only developers can describe the internal code
> dependencies. Anyone can describe the external dependencies. Actually
> most of the bugs are found by accident when a user triggers some
> dependency which has not been triggered before. For example developers
> have a system with some modules preinstalled before they start writing
> code. Then the case when there are no modules installed will never be
> tested by them. A user who starts up with a clean system finds out that
> the app crashes immediately. Then it's important to describe the
> situation in plain English so that it can be tested. Ideally we should
> gather these kind of situations and add them to regression test list.
> Actually it would have not hurt to add some of our "Prerelease Bugs" to
> a test list even though they were fixed.
>
>
> Offtopic: Now I'm leaving for a motorcycle trip for two days. See you
> later :)
>
>   Yours,
> 	Eeli Kaikkonen (Mr.), Oulu, Finland
> 	e-mail: eekaikko at mailx.studentx.oulux.fix (with no x)
>
> _______________________________________________
> bt-devel mailing list
> bt-devel at crosswire.org
> http://www.crosswire.org/mailman/listinfo/bt-devel