Which might be because parts of the fandom are annoying as hell. I actually think there was a demographic shift around 2002/2003. Before that I remember most people in this fandom were adults that see it as what it is. Entertainment. Nothing more, nothing less. You had your ultra-fans, but they werent really as loud as now.
Then when the Internet got more and more popular around 2003-2005 and it was far easier to consume anime instead of downloading them through dubious methods, now "children" could consume this entertainment product. And thats when it went downhill in my experience. You had 12 year old people trying to convince others that anime are mature, because in Elfenlied people die in a gory way.
You had often social-outcasts, who use that medium as some sort of escapism and in my experience dont really have much knowledge about it themselves, but pretend they do and kinda act like missionaries of anime/manga.
You have those people who use japanese phrases in their sentences because "Japan is the country where milk and honey flows".
The medium itself has problems, but I think the main reason why people think "anime-fans" are weird, social outcasts is because of those "weaboos". I mean if I go to the bookstore and want to buy manga, I can be sure 2-3 sit there in their cosplays reading manga for free and being loud.
A japanese teacher at my old university once told me that usually half of her language courses have those people, that either dress up as Visual Kei fans or go to a university lecture in their cosplay and she found it quite sad, that the country is reduced to this.
(And like I said. Usually the loudest and most obnoxious ones are the ones that dont even have much knowledge about their fandom/entertainment medium and the ones that you can talk to without any awkwardness and wouldnt notice they watch anime (because they dont define themselves through it) are the ones you could talk hours about the medium, historical changes, shifting in the medium and of course anime/manga itself).
Find out what you like by yourself. I would never go into an anime-club. I mean I go to an anime convention almost every year and you find a lot of the people I described in the upper paragraph there. But also the ones you can talk to on a normal level.