Here is a bit of info on the process ARRL uses to obtain and test equipment.
The details of
the test procedures are fully documented at:
http://www.arrl.org/members-only/prodrev/testproc.pdf
A number of articles have been written about Product Review. They are available
at:
http://www.arrl.org/tis/info/rigbuy.html
----------
The Overall Process
The ARRL Product Review process all happens under the watchful eye of the QST
Product
Review Editor. This is Brennan Price, N4QX@arrl.org. Brennan selects equipment
for
Product Review based on availabilty and the amount of interest expected in the
amateur
community.
The Purchase
ARRL purchases equipment on the open market. For most equipment, we just call
one of the
major amateur-equipment vendors, shop around a bit for price and have one
shipped to us.
Sometimes, if we are buying "factory direct," we order one through a ham who
doesn't work
at HQ, to help ensure that we will get a piece that is reasonably
representative of what a ham
will buy. Do keep in mind, though, that this is a sample of one, and that
there can be
considerable variation from unit to unit for some parameters. Comparison
shopping by
fractional dBs of measured performance is not the most important purchase
criterion. For most
hams with modest antennas and casual operator, features and the way they are
implemented
are more important than raw technical performance.
The Test
The equipment is delivered, unused, to the ARRL Lab for testing. We want to put
it through
the tests BEFORE the reviewer gets it, to ensure that its performance will be
"factory new"
when tested. Also, in the event something is wrong with the equipment when
received, we want
to ensure that a reviewer doesn't get the wrong impression by using a piece of
broken gear.
This occasionally happens and in that case, we contact the manufacturer, who
pronounces
the gear defective and arranges to repair it. In that case, the review will
always indicate what
happened and how it was resolved. We then test the repaired unit in the normal
fashion,
redoing the tests that may have been done before the repair, if there is any
chance that the
defect affected the result.
Locked Away in the ARRL Screen Room
The test process typically takes Mike Tracy a week or more, about 6 hours per
day, for a
multiband, multimode transceiver. The standard test battery is performed and
the raw
data written down on a log sheet. We may perform additional tests, based on
curiosity or to
investigate anything that looked interesting during the standard testing. We
often will get a few
additional samples of the model and run through a few of the major tests,
mostly for curiosity.
These would be borrowed from staff, friends, a local dealer or in some cases,
supplied by
the OEM. You will often see test results from these additional radios in the
expanded test-result
reports. The results of the testing are summarized in the review and often in
an expanded test-
result report that may be as much as 40 pages of test results, graphs and other
things to keep the
techie-types busy for hours.
If problems are encountered in testing, this is investigated rather thoroughly
by the Lab staff.
If noted during testing, the test engineer will typically retest the parameter
that wasn't right,
running through a series of "good engineering troubleshooting" steps to ensure
that the
problem encountered is real, and not caused by a bad cable, incorrectly set
equipment or
test-engineer error. (They do happen.) He then will consult with one of the
other lab staff,
typically me or Zack, to verify that what is found is an accurate measurement
of the unit
under test. This is part of the internal checks and balances used to ensure
that what is
published is accurate.
Any major problems are also called to the attention of the manufacturer. The
manufacturer
knows his product better than we do (in most cases, anyway) and if something is
not right,
they generally can spot it. This can indicate a defective radio, an anomaly or
design problem
or even an error by the ARRL Lab. If the manufacturer does not feel comfortable
with the result,
we go back one more time to triple check the measurement. Almost all of the
time, this back and
forth gets to the root of the problem. This is part of the checks and balances
that does occasionally catch an ARRL testing error, although most of the time,
the end result is that the manufacturer
agrees that we are correct in our measurement. Those times that we have made
errors caught by
this final safety net, we have been most appreciative and glad that we are
willing to put these
checks and balances into the process, because the final end result is that the
reader gets accurate
data. I don't mind being wrong, but I sure don't want to see 165,000 copies
made of any mistakes
I might make.
The Lab staff generally writes the test-result table that appears in print. We
may also write a
sidebar about some aspect of the testing, if that would be of interest to the
readers.
The Reviewer
After the testing is complete, the radio goes to a reviewer. This is often HQ
staff, paid a little
extra to write up a review on their own time. It could also be a member of the
ARRL "Official
Family," or a contributing editor or ARRL Techncial Advisor. The reviewer plays
with the radio
for a few weeks, then writes up a review of the subjective aspects of the
radio. The Product
Review Editor then combines the subjective review with the Lab results and the
end result is
the Review you all see in print. Typically, a few days before press time, a
copy of the review
is sent to the manufacturer, to give them an opportunity to correct any factual
errors. They do
sometimes also discuss our choice of words in the Review, and any changes are
at the
discretion of the Product Review editor. I will always remember the time that
a reviewer, in
noting that a minor feature had not been included in a radio, had text that
said, "A pox on
<Japanese manufacturer> for leaving off this feature." The US division of the
manufacturer,
upon seeing the text, called the editor back and asked them to change it,
noting that this common
colloquial expression would be translated in a very negative way when read by
the Japanese
members of the firm. The text was changed. :-)
After the Review
As soon as the review is finalized in print, it is posted to the ARRL
members-only page for
members to download. After it is printed, we keep the equipment here for a few
months, so
we can answer any questions about it that may crop up. At some point after
that, a solicitation
for bid it put into QST and the unit is sold, used, to the highest bidder. In
some cases, folks have
bid more than list price, just to have "The" Product Review radio.
Members and Manufacturers
The bottom line is, of course, the member. ARRL wants to ensure that the
members are given
accurate info about equipment being sold. To the best of everyone's ability,
that is what is printed
in QST. Of course, hindsight it always 20:20, so there is always room to
improve. Member
feedback to qst@arrl.org about any aspect of QST is always appreciated.
But in the course of doing a review, it is also important that ARRL treat
manufacturers fairly.
It would NOT do amateur radio any favors if ARRL were to be unfair to
manufacturers in
its Product Review testing. There have been disagreements along the way, but
almost without
exception, manufacturers recognize that ARRL is willing to work with them and
to try to resolve
any issues that come up in a fair way. IMHO, this balance between the various
issues is a
critical part of the process, and to the best of my ability, I will continue to
do my part to make
it all happen.
Well, that's about it for me. Questions and discussion are most welcome.
Seeing as this all came
up as part of the Argo V review, I figured that this was a good place to follow
up. BTW, I do agree
with the Ten Tec web page that overall, the review was quite positive. And I am
chomping at the bit to see the Orion in the screen room!
73,
Ed Hare, W1RFI
ARRL Lab
225 Main St
Newington, CT 06111
Tel: 860-594-0318
Internet: w1rfi@arrl.org
Web: http://www.arrl.org/tis
|