I'm nearly 50 years old and I'll tell you like I tell my own son. Do what you know is right and don't worry about what others think about you. You really don't owe your friend any explanation, but you might want to talk to him about what is bothering you if you want to try to keep the friendship alive. If you're not satisfied with his further actions at least he'll know why you're not associating with him. In my opinion anyone who acts differently depending on the crowd they are with is not very self confident or very mature.