[CQ-Contest] Contest Log Processing

[Doug Smith]

On Thu, 2009-01-29 at 16:01 +0100, Franki ON5ZO wrote:
> This might be a controversial issue but I think today's programming 
> languages provide powerfull tools to process ASCII data under the form of 
> XML.
> If contest logs would come under the XML format with standard tags to ID the 
> fields, parsing logs would be relatively easy.
> The .Net platform has some nice built in routines to write data to or read 
> from XML where minimal code is required.

As the logchecker for the Tennessee QSO Party I'd rather we stick with
Cabrillo.  It's not at all difficult to parse (even with the simplest
languages - such as QBASIC!), is human-readable, and can be easily
imported into spreadsheet applications.  

=========================================================================

To my knowledge W3KM's Cab Evaluator is the only generally-available
package for logchecking.  To be honest I'm a bit surprised there isn't
more information out there about what's used by other contests.  I
suppose there could be a belief release of code would risk making
cheating (or denial-of-service attacks!) easier - but I would think some
general concepts could be described without security risks.

=========================================================================

In the past, for the TNQP each log was converted to Cabrillo (if it
wasn't already) and imported into a spreadsheet.  The spreadsheet would
be sorted in various ways to find dupes, to assign QSO points, and to
count up multipliers.  It would then be exported to a CSV
(tab-delimited) text file and stuffed in a directory with all the other
CSV files.  We'd then go through and search all the text files for the
string "K0WA".  And take a note of any matching records that didn't list
the QTH as Kansas, and mark them as busted QSOs.  

Unfortunately finding busted *calls* in this scheme was nearly
impossible.  If a suspicious call (say, K1WA) was seen while
importing/marking mults we could check for a matching QSO from who we
thought was really worked - but I'm sure we missed a lot of busts.

=========================================================================

This year, all QSOs were brought into a MySQL database.  The Perl
scripts that did the importing assigned QSO points and checked for dupes
as they imported.  Further scripts:

- Checked each QSO for a valid QTH. (matching against another table in
the database which contained a list of valid QTHs)  This script also
marked a field in each QSO that was a new multiplier.

- Checked for bonus points - 100 for working K4TCG, 500 for making more
than ten QSOs from a county while mobile.  This script also assigned
bonus multipliers to mobiles.

- Cross-checked each QSO.  If there was a QSO like this:

Band	Mode	MyCall	MyQTH	HisCall	HisQTH
7	SSB	W9WI	ROBE	K0WA	KS

then there should be another QSO like this:

7	SSB	K0WA	KS	W9WI	ROBE

If the second QSO didn't exist but there *were* records specifying K0WA
in the MyCall field, then the first QSO was a bust.

- Prepared a report for each entrant showing the point value and
multiplier status of each QSO and noting if it was a busted QSO.

The time-consuming part, besides writing all the code, was going over
each report and ensuring every QSO that was flagged as "busted" actually
was, and that everyone got credit for everything they should have.  

One of the more vexing issues is dealing with inconsistent QTH
abbreviations.  ON - is it Ontario or Belgium?  OH: Ohio or Finland?  A
human can tell instantly (from the callsign) but it's going to require
more coding to get the computer to do so.  We had to manually edit a few
records where people entered something like "SA" for Saskatchewan or
"NB" for Nebraska instead of New Brunswick.

We have both Hardin and Hardeman Counties and tend to end up with
inconsistent abbreviations because of entrants like myself who don't
read the official abbreviation table until after sending in our logs.
(sorry, Kentucky QSO Party entrants who worked W9WI/m!)

I'm really not prepared to support this code so please don't ask for it!
I would imagine any reasonably competent Perl/SQL programmer (I'm not
one!) could produce something similar fairly quickly.

=========================================================================

On an only vaguely related subject, we received two entries that
required massive manipulation.  (luckily they didn't have many QSOs)  It
seems that the "received QTH" field of the Cabrillo file contained the
entire name and address of the station worked!

Obviously this was NOT transmitted over the air.  It looks as if these
entrants used a general QSO logging program that picked address data
from QRZ.com or some similar database and used it to populate the QTH
field.  This obviously raises ethical questions!