It's already been mentioned, but you do have to live with these people as neighbours.
If you intended to call the police you should have told the man when he came over to talk with you.
People need to handle stuff like this themselves rather than call the police, complain to the OFC mod's or whatever.
I think Roy expressed my sentiments a little more eloquently.