Trust your wife's instincts and then do what you have to. Most women I know are usually right on the mark with their gut feelings.
Face to face is a tough thing to do, but like someone else mentioned, you can get your own read about the situations and his responses.
Could be a lonely guy that can relate better to women (he lives with his mother at 45) or a creep like your wife feels.
I have had neighbours in the past that seem to show up when my wife is out gardening and then proceed to waste an hour of her time talking about nothing important. All innocent, but boy can it be annoying. She has learned how to deal with them, which is more powerful than me talking to the guy. Yes, I keep my eyes and ears open.