[RTTY] [CQ-Contest] LoTW confirmation rates

[Dave AA6YQ]

>>>AA6YQ comments below
 
In a message dated 2014-02-11 3:50:18 P.M. Coordinated Universal Tim, lists at subich.com writes:

It's easy  to hide a problem by spending $$$ to move it to faster hardware. 

To  date nothing significant has been done to resolve what the LotW Advisory  Committee calls LotW's "substantial technical debt."
A second developer has  not been hired to focus specifically on fixing basic issues with the LotW  code and implementing new awards
(e.g. WAZ).
Now staff wants a *second*  system (why don't they fix the disk system and use the old hardware?) as a  test platform, etc. and the
current IT Manager is wasting time on things  like the ARRL Centennial QSO Party ... and disabling features that don't  work
correctly.

LotW will get praise when it is fixed ... putting a  bigger engine in front of a transmission that is inefficient doesn't fix  the
problems in the transmission even though the car goes faster for a  while.


>>>There were 3 factors that contributed to LotW's "meltdown" in November 2012:

1. increasing load on the LotW server, caused in part by increased adoption and usage, and in part by the propensity of many users
to include every QSO in their log with each new submission to LotW

2. a long-standing defect that caused submitted log files to be discarded, triggered when the number of submitted but unprocessed
log files exceeded a particular threshold as a result of #1

3. the resulting exponential increase in re-submitted log files from users who assumed that their original submission had been
discarded, causing even more submitted log files to be discarded as a result of #2

>>>The defect was identified and corrected by Mike K1MK.

>>>One component of LotW's technical debt is that there is no performance model, nor is there any available hardware on which to run
a simulated load greater than what the  production LotW server currently experiences. Thus from a performance perspective, LotW is
flying blind. In the darkness of November 2012, Mike K1MK analyzed the LotW server's performance as best he could, concluded it was
severely I/O bound, and convinced ARRL management to upgrade the LotW server's hardware with a high-performance storage system. The
results speak for themselves.

>>>The performance of every significant online transaction processing system is limited by a bottleneck. Eliminate that bottleneck
with software or hardware changes , and a new performance bottleneck will be revealed. Thus the acquisition of a second set of
hardware identical to that currently supporting the production LotW server is not frivolous. It will enable ARRL staff to identify
the next performance bottleneck in the LotW server, methodically determine how best to eliminate that bottleneck, and ensure that
the appropriate corrective actions are taken before continuing increases in production server load make the bottleneck  apparent to
the LotW user community, as happened in November 2012.

>>>The second hardware platform will also serve as a backup in the event of a catastrophic failure in the current hardware, reducing
what could be a multiple-day LotW outage to a multiple-hour LotW outage.

>>>It is unfortunate that scarce LotW development resources were diverted by the ARRL Centennial award project. ARRL management had
already planned and announced this award before they understood the magnitude of LotW's technical debt. This should not happen
again, but it wouldn't hurt to emphasize to your ARRL Director that no new functional demands be placed on LotW until its technical
debt has been addressed.

       73,

               Dave, AA6YQ