*awards patch, probably with a very low-quality drawing of a pony*
And yes, I've noticed the same thing. In fact, I've had minor arguments with friends over the subject. There's kind of a strange insistence that I should feel ashamed to call myself a brony, even though they are the ones who associate the term with the nasty side of the fanbase. I personally just use it to mean "a fan of the show," the same as Whovians or Homestucks.
Besides, "Brony" rolls off the tongue much more easily than "My Little Pony Fan." Or "Pegasister" for that matter, but that's a different discussion.