Personally I cant see where too many men, especially in his position would just go and want to hold someones baby or infant. Now I could see his wife probably holding the baby and then saying to him,"here honey, you hold him". But for him to just out of the blue come in and want to hold your child that would just be ODD IMO. I dont think you have to much to worry about. BUT, if he does then:
You just stick to your guns and tell him outright "Id prefer you not hold him." Even tell him you dont want the child to get use to going to strangers and a lot of people have already been holding him.