Topband: AGC (and now other work)

[rwmcgwier at comcast.net]

Thanks to Oleg, Tom, and others for all these details on their AGC work.  It has been very informative.

For the software defined receiver, I am working on three different things right at the moment in the way of noise/interference and level mitigation algorithms.  The first is the AGC which has been helped a lot by the discussion here and I have a lot grist for my experimental and simulation mill.  I am sure I can come up with a really good AGC that can be tailored to the users ear.  Many designers don't, forget, or ignore that the operators ears are part of the transfer function. Versatility and flexibility of approach are key.  The next order of business is pulse noise mitigation.  I absolute despise noise blankers.  They are of course makers of receiver "clicks" and for the exact same reasons your cw transmitter clicks if you don't shape the waveform.  Any time you turn a receiver off and then back on rapidly, you get a wide frequency modulation of the incoming signal.  This is mitigated somewhat by the filtering that always follows in which this on/off modulation is smeared
  over and the filter acts like an interpolator for the missing signal values.  But the out of passband signals that are inside the roofing filter are "modulated" by this on/off condition and spread out into the passband.  This is the cause of the noise blanker distortion you can hear so prominently in most radios with noise blankers. I am working on two approaches to pulse interference mitigation.  I implemented the first yesterday in the SDR-1000 Console and it works.  It is based on the ideas of speckle removal from images.  The current reigning king of those algorithms takes a rank ordered mean of the pixels in the area of the pixel under the algorithms pointer.  It then looks at various change or derivative values to see if it passes the test for looking like impulse noise.  I figured if this works in 2D(images) well enough to be the winner, then it ought to really work in 1D of a time domain signal.  I take neighboring samples of the current signal sample and sort them.
   I compute the rank order mean (take the average of the two middle samples in this rank ordered collection of signal values).  I then compute the difference of the ranked ordered samples from the current sample value and correct for the sign of the difference of the current signal value and the rank ordered mean.  If the first or second order deltas are too large (different thresholds), the signal sample is called a pulse.  I do NOT turn off the receiver (set the sample to zero), I replace it with the rank order mean.  This is much smoother and more nearly linear than blanking.  The mixing effects of what nonlinearity there is appears to be much smaller than with blanking.  This is really important in that we can set aggressive thresholds and even if we make a mistake about calling a signal value a pulse, the cost of doing that is mitigated by taking an "interpolated" value from nearby.  This is a very simple algorithm to code (other than you have to sort signal values quic
 kly).  It sounds really complicated, but it isn't and it works. The other approach was first introduced by Leif Asbrink in Linrad.  It is completely linear.  He trains the software to know what the impulse response of the receiver is and the correlates for the presence of a noise pulse and then after correcting for phase and amplitude (directly computed from the correlation), he SUBTRACTS the pulse from the incoming signal.  This is a completely linear process.  It will not modulate or distort in the normal ways but to the extent you have the impulse response wrong, it will actually ADD a pulse back in!   So it is not perfect.  The imperfection comes from assuming one impulse shape that applies everywhere and the requirement that the user do determination of the impulse response of the receiver through a process that is really easy to screw up (hooking a pulse source with a funky impedance and transfer function all of its own and just doing the procedure wrong).  I know how 
 to automate all of that and to make better assumptions of what the pulse shape will be like. The obstacle is doing this efficiently in a real time DSP process.  If this works, it will be the best noise pulse canceller ever produced anywhere, bar none.  I have heard Leif's Linrad working when the impulse response is right (Leif or someone with the correct test equipment and knowledge do the training/setup) and it is absolutely spectacular. I have my fingers crossed for this work.

Lastly, as I have gotten older, I just get really fatigued by hiss. Imagine my dismay when I turned on my FT-1000MP and had the EDSP off. With EDSP on, even with ANR turned off, the hiss is mostly gone. I just hate the performance of the LMS based automatic noise reduction and automatic notch filtering algorithms.  They converge too slowly and really don't track big changes in conditions well.  Recursive least squares converges brilliantly fast on a STATIONARY unchanging channel but then does even worse than LMS (LMS is what is in everyone's DSP in their radio) at tracking dynamics and RLS is
hideously expensive in the forms that do not have serious numerical problems.
Fortunately: from the Echo cancelling world, there comes a new collection of algorithms called Affine Projection Algorithms which solve our problems.  They converge almost as fast as RLS, they are only a couple of times more expensive than LMS and really do a beautiful job.  When I am done, I believe this will be the first application of this algorithm to automatic noise reduction in receivers.  From my receiver recordings (IF recording and AF recordings) and simulation and design in MATLAB, this algorithm performs spectacularly well.
Again, Leif Asbrink has done a thing in Linrad for "spur removal".  Here there is nothing I can do to improve on the theoretical algorithm.  The interfering tone and spur removal that I will eventually do, will be based on this and my main contribution will be a much more efficient implementation of the automated algorithm.  It does not filter, it subtracts the spurs and interferers by figuring out their amplitude and phase and then replicating them 180 out of phase and adding.  This eliminates them without notch filtering.  To the extent you get this wrong, it will add an interferer or spur back in!  So the automation must be done carefully and correctly. It is expensive and needs careful coding to make this possible.
After we overcome the design flaws in the SDR-1000 (Gerald is doing an add on board that fixes the problems), all of this code is going to make the SDR-1000 and other software receivers or DSP algorithms that use the code some sweet receivers.  As always, this will all be released in GPL form and articles are to be written for QEX.  I think W2GD is going to let me use the thing as a run station receiver next 160m season and N2NT is going to tell me where I am screwing up in order to get the things useful to the serious 160m operator. 

Any comments on all of this are not only welcome but encouraged.

73's
Bob
N4HY
> > I can add just a few comments
> > > Above threshold, the AGC clamps the audio output at a constant level
> with
> > no
> > > measureable change in audio output for increasing signal input.  There
> is
> > > definitely no gain "slope" and no other obvious magic.
> >
> > I tryed an AGC with flat response above threshold (and my AGC does so when
> > receiving
> > the digital modes), but I found a more pleasant sound when a stronger
> signal
> > are a bit louder,
> > so I prefer some gain slope above the threshold (10dB:1db ... 20dB:1dB).
> 
> A hardware AGC system must have something unusual inside if it has a
> threshold and then clamps perfectly. Perhaps very high gain and a threshold.
> 
> In my homebrew AGC circuits, I rectify IF voltage before the last detector.
> I can vary the gain of AGC amplifiers, or set bias on the first amplifier
> transistor (making it non-linear) to control a fast tip-in point. Varying
> gain changes the audio output vs. RF input level slope. It is very difficult
> to make it "all or nothing" at one certain level.
> 
> I find by far the best sound on weak signals occurs when there is some AGC
> as noise floor is approached, but not nearly as much AGC as when the signal
> is well up out of noise.
> 
> With very high gain in the AGC loop every noise pulse and every signal
> regardless of strength comes out the same exact audio level. While that is
> OK with a smooth steady hissing noise, if anything rough at all or any other
> signals are added to the mix all of the dynamics are lost. Absolutely
> everything from the audio has the same level.
> 
> What I found while listening last night and this morning is listening with
> hard-leveling AGC is like running volume up and clipping everything hard in
> a diode limiter at the receiver output. When it gets a little noisy I have
> to add long hang time to prevent "mush", and then that masks weak signals
> between noise pulses.
> 
> I'm strongly coming to believe the last thing we want (unless we like to
> constantly have knobs to adjust) is a very stiff AGC that all tips in at a
> knee, with absolutely no AGC below that point. What that would do is require
> we set the tip in point for the noise or QRM floor at that particular
> instant in time.
> 
> Paying attention to what I do, I move the volume up a bit when trying to
> copy a very weak signal that is just ESP and back down to a "normal setting"
> for everything else, yet at ALL times I can plainly hear a background hiss
> or noise. I just need to have that noise or hiss loud to dig out weak
> signals, and I can't tolerate any clipping or limiting on the noise or
> signal unless it is well out of noise. I find I never adjust AGC speed,
> except going between SSB and CW.
> 
> That's just how I like AGC.
> 
>  I also find the less I have to adjust stuff the easier life is. Finding the
> correct antenna and tuning the VFO are time consuming enough. Even logging
> the QSO is a distraction from copying weak signals.
> 
> 73 Tom
> 
> 
> _______________________________________________
> Topband mailing list
> Topband at contesting.com
> http://lists.contesting.com/mailman/listinfo/topband