A not so good situation????

The way to take care of a situation is to address that situation. Whether it causes a rift of not allowing him to keep doing it and not telling him anything does nothing for the situation. If I were you I'd have a talk with him and be upfront don't coward away because you don't want to create a problem. If you don't bring it up now then he will feel like he can continue to do this and if you want the best for your child it's better to get it over with now. Let him know that is unacceptable and if he doesn't change then he won't be allowed to see the child because you don't want someone doing that or saying those nasty comments.