[TenTec] ARRL Product Review process -- an overview form a testing perspective

[Hare,Ed, W1RFI]

Here is a bit of info on the process ARRL uses to obtain and test equipment.  The details of
the test procedures are fully documented at:

http://www.arrl.org/members-only/prodrev/testproc.pdf

A number of articles have been written about Product Review. They are available at:

http://www.arrl.org/tis/info/rigbuy.html

----------
The Overall Process

The ARRL Product Review process all happens under the watchful eye of the QST Product
Review Editor. This is Brennan Price, N4QX at arrl.org.  Brennan selects equipment for 
Product Review based on availabilty and the amount of interest expected in the amateur
community. 

The Purchase

ARRL purchases equipment on the open market. For most equipment, we just call one of the
major amateur-equipment vendors, shop around a bit for price and have one shipped to us.
Sometimes, if we are buying "factory direct," we order one through a ham who doesn't work
at HQ, to help ensure that we will get a piece that is reasonably representative of what a ham 
will buy.  Do keep in mind, though, that this is a sample of one, and that there can be 
considerable variation from unit to unit for some parameters. Comparison shopping by 
fractional dBs of measured performance is not the most important purchase criterion. For most 
hams with modest antennas and casual operator, features and the way they are implemented 
are more important than raw technical performance.

The Test

The equipment is delivered, unused, to the ARRL Lab for testing. We want to put it through 
the tests BEFORE the reviewer gets it, to ensure that its performance will be "factory new" 
when tested. Also, in the event something is wrong with the equipment when received, we want
to ensure that a reviewer doesn't get the wrong impression by using a piece of broken gear. 
This occasionally happens and in that case, we contact the manufacturer, who pronounces
the gear defective and arranges to repair it.  In that case, the review will always indicate what 
happened and how it was resolved.  We then test the repaired unit in the normal fashion, 
redoing the tests that may have been done before the repair, if there is any chance that the
defect affected the result.

Locked Away in the ARRL Screen Room

The test process typically takes Mike Tracy a week or more, about 6 hours per day, for a
multiband, multimode transceiver. The standard test battery is performed and the raw
data written down on a log sheet. We may perform additional tests, based on curiosity or to 
investigate anything that looked interesting during the standard testing.  We often will get a few
additional samples of the model and run through a few of the major tests, mostly for curiosity. 
These would be borrowed from staff, friends, a local dealer or in some cases, supplied by
the OEM.  You will often see test results from these additional radios in the expanded test-result 
reports.  The results of the testing are summarized in the review and often in an expanded test-
result report that may be as much as 40 pages of test results, graphs and other things to keep the
techie-types busy for hours.

If problems are encountered in testing, this is investigated rather thoroughly by the Lab staff. 
If noted during testing, the test engineer will typically retest the parameter that wasn't right, 
running through a series of "good engineering troubleshooting" steps to ensure that the
problem encountered is real, and not caused by a bad cable, incorrectly set equipment or
test-engineer error. (They do happen.)  He then will consult with one of the other lab staff, 
typically me or Zack, to verify that what is found is an accurate measurement of the unit
under test.  This is part of the internal checks and balances used to ensure that what is
published is accurate.

Any major problems are also called to the attention of the manufacturer. The manufacturer 
knows his product better than we do (in most cases, anyway) and if something is not right, 
they generally can spot it.  This can indicate a defective radio, an anomaly or design problem 
or even an error by the ARRL Lab. If the manufacturer does not feel comfortable with the result, 
we go back one more time to triple check the measurement. Almost all of the time, this back and 
forth gets to the root of the problem. This is part of the checks and balances that does occasionally catch an ARRL testing error, although most of the time, the end result is that the manufacturer
agrees that we are correct in our measurement. Those times that we have made errors caught by
this final safety net, we have been most appreciative and glad that we are willing to put these 
checks and balances into the process, because the final end result is that the reader gets accurate
data. I don't mind being wrong, but I sure don't want to see 165,000 copies made of any mistakes 
I might make.

The Lab staff generally writes the test-result table that appears in print. We may also write a 
sidebar about some aspect of the testing, if that would be of interest to the readers.

The Reviewer

After the testing is complete, the radio goes to a reviewer. This is often HQ staff, paid a little 
extra to write up a review on their own time. It could also be a member of the ARRL "Official 
Family," or a contributing editor or ARRL Techncial Advisor. The reviewer plays with the radio 
for a few weeks, then writes up a review of the subjective aspects of the radio. The Product 
Review Editor then combines the subjective review with the Lab results and the end result is 
the Review you all see in print.  Typically, a few days before press time, a copy of the review 
is sent to the manufacturer, to give them an opportunity to correct any factual errors. They do 
sometimes also discuss our choice of words in the Review, and any changes are at the 
discretion of the Product Review editor.  I will always remember the time that a reviewer, in
noting that a minor feature had not been included in a radio, had text that said, "A pox on 
<Japanese manufacturer> for leaving off this feature." The US division of the manufacturer, 
upon seeing the text, called the editor back and asked them to change it, noting that this common
colloquial expression would be translated in a very negative way when read by the Japanese
members of the firm.  The text was changed. :-)

After the Review

As soon as the review is finalized in print, it is posted to the ARRL members-only page for 
members to download. After it is printed, we keep the equipment here for a few months, so
we can answer any questions about it that may crop up.  At some point after that, a solicitation
for bid it put into QST and the unit is sold, used, to the highest bidder. In some cases, folks have
bid more than list price, just to have "The" Product Review radio.  

Members and Manufacturers

The bottom line is, of course, the member.  ARRL wants to ensure that the members are given
accurate info about equipment being sold. To the best of everyone's ability, that is what is printed 
in QST. Of course, hindsight it always 20:20, so there is always room to improve.  Member 
feedback to qst at arrl.org about any aspect of QST is always appreciated.

But in the course of doing a review, it is also important that ARRL treat manufacturers fairly. 
It would  NOT do amateur radio any favors if ARRL were to be unfair to manufacturers in
its Product Review testing. There have been disagreements along the way, but almost without
exception, manufacturers recognize that ARRL is willing to work with them and to try to resolve
any issues that come up in a fair way. IMHO, this balance between the various issues is a
critical part of the process, and to the best of my ability, I will continue to do my part to make
it all happen.

Well, that's about it for me. Questions and discussion are most welcome.  Seeing as this all came
up as part of the Argo V review, I figured that this was a good place to follow up.  BTW, I do agree
with the Ten Tec web page that overall, the review was quite positive. And I am chomping at the bit to see the Orion in the screen room!

73, 
Ed Hare, W1RFI
ARRL Lab
225 Main St
Newington, CT 06111
Tel: 860-594-0318
Internet: w1rfi at arrl.org
Web: http://www.arrl.org/tis