[CQ-Contest] When is it OK to refuse to give out a QSO?

[John W]

As I type this, I am sitting here fuming mad and deliberating on whether to keep plugging away in the CQWW SSB contest or just shut off the PC and radio.    Blessed with three young children (two toddlers and an infant), I have extremely little time for radio these days. So I am doing a VERY part time effort in the contest, yet still trying to maximize my score for the few hours I can put in.  
It is Sunday morning. My wife is napping with the baby, and I have gotten the kids set up in front of the TV with their favorite show and I can finally sit down at the radio without interruption.  I have made only a few QSOs so far on 15m this weekend.  I just looked all over the band for Zones 3 and 4.  I found K5TR calling CQ, waited until he had nobody else calling him, and made a quick call, saying "W2ID for the mult".  He quickly replied "W2ID 59 4", I gave the report and logged the qso and mult. QSO made and logged, all in less than 3 seconds. Thank you!
Tuned around some more, and some more, and finally found N7DD for zone 3. Excited, I waited a few minutes until he had no callers, and then gave a quick call just as I did with K5TR.  Instead of replying with the expected "W2ID 59 3", he replied with "It's a little too late in the contest for that.  Go find a VE."   I was completely speechless, and couldn't even think of a reply before he called CQ again.  Then I replied with "I'll remember that in Sweepstakes."  He snarled back with "Yeah, you do that."
So not only will I remember it in Sweepstakes, so will everybody else who reads it here.
Now I have spent enough time typing this that I could have made at least 15 QSOs, which would have more than compensated for the lost multiplier.  Wondering whether I should go try and find another zone 3, or just go and watch Bob the Builder with my kids instead. At least they will take a few seconds out of their favorite TV show to talk to me...
73 & GL all (except N7DD that is),
JohnW2ID