I think you should work on becoming more uninhibited. I don't care what people think of me. I have a community of gamers, but it's very silent. They've hardly ever told me what they think of my games, but that doesn't matter to me. I continue working calmly.
Here's a Machiavelli quote that sums it up best:
"No matter how good a person you are, we are all the bad guys in someone's story."
"So enjoy your role and at least be a memorable villain."
Think about it. Remember your ex-girlfriends. There's bound to be one who hates you, right? Everyone has at least one. Anyone who says they don't is either a virgin or lying. That doesn't mean you're afraid of women, does it?