People care far too much about what other people think. Sure, people are social animals, and others opinions of you are important, but far less so that in the more primeval times when fitting in with the tribe was a matter of life and death. We have moved past primitive forms of collectivism such as that, and the western ideals of individualism, or at least they should insure, a degree of independence from the arbitrary prejudices of other people. "Civilization is the process of setting man free from men". In other words, you have the right to go against the crowd, and that's a responsibility you should take very seriously. Care what other people think - you need relationships to live a healthy life - but only care about what the right people think; surround yourself with the type of people moral people who will be good peers and role models.
That said, I think the aforementioned ideals are being threatened by the new-left social justice movement, so that's is something you should fight against whenever you see it.