nip it in the bud
let him know you do not want him on the property without prior permission
you can stick up a motion detector and or camera and tactfully tell him he is setting the cameras and motion detectors and it makes you think there a burglar on the property , till you check who there, so it would be better if he didnt come on the property univited
one it lets him know you dont want him there
and two makes him think he could be caught on camera if he comes
and if said right, he will not be insulted