In the end, Just remember If he really want to stay with you, He will pursue you. That's it!
We can't please everyone in life. If you know what you're doing is not wrong, If you're not hurting someone, If it's not really wrong then there is nothing to be worried about. Let's always give more and expect less.