Topband: FT-8 performance

[Marco Cogoni]

Hi,

I agree with Jerry. I spent a few weeks trying to use FT8 to obtain 
antenna radiation patterns and I discovered how the SNR is computed: 
it's totally flawed. Basically WSJTX computes the number in two steps: 
the first one estimates how strong the adjacent frequency bins are with 
respect to the bin of interest. Then this value is corrected by 
estimating the baseline noise in a 2.5kHz bandwidth.

The bad is that in this way the program is trying to unify a QRM concept 
(how strong adjacent noise is) with band noise (due to QRN or whatever 
wide band there is).

I found out the problem because using WSPR SNR estimations lead to 
antenna patterns that are very similar to NEC simulated antennas but FT8 
data produced WORSE patterns for the BEST antennas. This is due to the 
fact that best antennas receive far more signals so WSJTX gives a worse SNR.

I tried to talk with Joe Taylor about this, but he said that this method 
is what he thinks it is best for a general use, if I have a better one, 
just go and modify the code yourself.

I have a feeling that this way of computing the SNR was chosen because 
it makes you think, as Jerry pointed out, that FT8 can magically decode 
signals that are absolutely not hearable. That's highly dubious.

What FT8 does achieve is an extreme overlapping of signals over 2.5kHz, 
but at the expense of not really being able to tell the SNR...

If you want to dig deeper in the FT8 implementation have a look at AB1HL 
Robert Morris' FT8 Python implementation that is heavily commented and 
very very educational: https://github.com/rtmrtmrtmrtm/basicft8

He also wrote a full code/decode high performance software (not so easy 
to understand...): https://github.com/rtmrtmrtmrtm/weakmon

Hope it helps!


73,

marco / IS0KYB


On 01/08/19 15:17, K4SAV wrote:
> I get the feeling that I must be the only person that has ever tested 
> FT-8 to the extreme to see what it can do. It seems that everyone else 
> just assumes it will do what the published information says.  It will 
> not.  Below is a summary of my testing.
>
> First I did a bunch of testing to see if I could figure out how the 
> software determined the S/N number.  I measured the strength of a 
> station calling CQ and subtracted that from the S meter reading for 
> the 2500 Hz USB band and that resulted in a number very close to what 
> FT8 reports.  I repeated the test multiple times.
>
> Official documentation says FT8 can decode signals 24 dB below the 
> noise floor.  That's an interesting comment because FT8 has no way of 
> determining the level of the noise floor.  Even during the off period 
> when no one is transmitting it can't determine the noise floor because 
> the noise floor is not represented in the audio signal.  The receiver 
> ACG brings the noise back up when no one is transmitting.  Actually 
> for FT8 the amount of audio is greater in the off period than it is 
> when stations are transmitting.  The only way to measure the noise 
> floor is by making the measurement in the RF world, which FT8 can't do.
>
> Actually what the program does is count everything in the 2500 Hz 
> bandwidth as noise. (That is comprised mainly of strong signals, not 
> noise.) It is limited to decoding signals 24 dB below that level.  
> From that info you can guess that if you narrow the bandwidth to 
> eliminate most of the strong signals FT8 will decode weaker signals.  
> Yes that works.  Verified it myself and others have also found this to 
> be true.  You guys that like to operate FT-8 should take note of that 
> comment.
>
> I thought maybe it could decode stuff below the noise floor if all the 
> signals were also below the noise floor, so I tested for that.  I 
> found out that the program poops out long before the noise floor is 
> reached.  With my tests where I decreased the gain of signals from the 
> NE (just after sunset in the NE) by pointing my antenna west and 
> increasing the gain such that the biggest signals were 15 dB above the 
> noise floor, FT8 just about quits. There were probably 50 or more 
> stations on the band and it was making one decode about every 5 or 10 
> minutes.
>
> FT8 has a deep search mode where it uses stored data to make guesses 
> at call signs some times.  I tested that too.  Before I started WSJTX 
> I set up a poor signal to noise ratio test. Started WSJTX in the 
> normal mode and it did very poorly.  Then I turned on deep search, 
> increased the S/N and let the program look at the band for a little 
> while.  Then I went back to the poor S/N condition without turning 
> FT-8 off and turned on deep search and it made a lot more decodes.  
> Nearly all of those decodes were reported at -24 dB.  I think those 
> were guesses and it just assigns -24 dB for guesses.
>
> I was fooled by the results of a test I did when adding enough noise 
> to the audio to cover up the signals and FT8 continued to decode the 
> signals.  However I had previously had deep search on and it had 
> already memorized the band when I did that.  It was just guessing that 
> the same station is on the same frequency as previously.
>
> I keep hearing reports from people that claim they are getting decodes 
> without hearing anything in the audio. I set up conditions where that 
> should have happened, but it never decoded anything.
>
> In summary. it appears that on an almost dead band, CW (with narrow 
> bandwidth) has about a 15 dB advantage at decoding weak signals.  On a 
> very crowded band if FT-8 is using a 2500 Hz bandwidth, CW has a huge 
> (many dBs) advantage over FT-8 because FT-8 can only decode about 24 
> dB below whatever your S meter is reading.  At 2500 Hz bandwidth in 
> the FT-8 band on 160 my radio usually hangs at 20 to 30 dB over S9.
>
> I was using a TS-990S receiver with no audio processing or noise 
> limiting or blanking.
>
> If anyone else has run similar tests. I would love to hear about it.
>
> Jerry, K4SAV
> _________________
> Searchable Archives: http://www.contesting.com/_topband - Topband 
> Reflector