[na-user] Beta Testing

dap14@chrysler.com dap14@chrysler.com
Fri, 14 Nov 1997 13:28:57 -0500


A week or so ago, Bill,W4AN posted a message about NA beta testing.
His comments, and those of other users who responded were right on.
Now its time for me to put my two cents in.

The was one time in NA's history where I relied on a external squad of
beta testers.  NA version 9 was a *MAJOR* revision, particularly due to
the configurable contest rules capability.  We had beta versions 9.0a
thru 9.0k that were put into the hands of these beta testers.
Ultimately, version 9 was one of our most difficult introductions with
three revs of bug fixes in the first month.  Perhaps this was to be
expected, but LTA had to mail not one, but two sets of replacement
disks to everyone who had purchased NA9.  Not a good experience.

Two things conspire to prevent NA from being fullly tested to the level
Bill described of Mathworks.  The first is manpower.  Tom and I both
have full time jobs, and he has a family.  This limits the amount of
time we can devote to testing.  The second is turnaround time.  If we
have a month long testing regimen, then we can't respond quickly to
user reports or new feature ideas.  One contester (not yet an NA user)
told me that he felt there should be no more than four releases a year,
his point being that he would prefer fewer "mature" versions that a
multitude of untested "versions du jour".  He has a valid point, but I
personally enjoy being able to quickly deliver programs with new ideas
and features to the user base.  This means either supreme luck, or a
good testing regimen.  To date we've relied on the former.

I was encouraged by the response to Bill's note.  A few of the replies
showed that they understand the difficulties of beta testing.  You
can't just load the program in the drive, load a couple of log files
and try a few functions to consider the program tested.  As Greg, K9IG
noted, there needs be a test plan that is documented and and then
executed for the program to be considered "tested".  This ensures
consistancy and allows repeatability.

A prospective beta tester does not need to be a software engineer.  He
or she should however, posess a reasonably logical mind and have a good
familiarity with how NA functions.  They also need to realize that they
are taking on a responsibility to the NA user community.  If they take
shortcuts, or don't get the testing done in a timely manner, its the
user who's affected when the feature or bug fix is not available.  At
the same time, we need to keep in mind that these beta testers will be
volunteers.  While there may be some perks from being a beta tester
(such as permanent access to the web download site) these probably
won't make up for the time required.  However, for most people like
this, personal satisfaction and be gratifying compensation.

As Bill suggested, we should categorize all of the program aspects of
NA and look for volunteers to test a given area.  I would like to find
testers whose personal interests and station activities align with the
features they are testing.  For example, W4AN for two-radio operation,
perhaps a serious single-op assisted person to test packet, a multi-op
station owner to test multi-computer networking, etc.  The test plan
would be a continuously evolving document posted on the DATOM web
page.  Each beta tester will formulate the testing plan for his or her
category which will be reviewed by K8CC to verify that it conforms to
the way the code was designed, then integrated into the master testing
document.

Here is a first draft of some proposed NA testing categories:

 Points and multiplier scoring
 Dupe or call related windows
 Multiplier releated windows
 Packet pperation
 Band map
 Two radio operation
 Multi-computer networking
 Computerized radio control
 Voice and CW keying
 Output files

So, how about it?  Send replies either to the reflector or
k8cc@ix.netcom.com.

73, Dave/K8CC

--
Submissions:              na-user@contesting.com
Administrative requests:  na-user-REQUEST@contesting.com
WWW:                      http://www.contesting.com/datom/
Questions:                owner-na-user@contesting.com