That sounds like you're attributing the blame to feminism as a whole rather than what i hope you actually meant which is that it's the fault of the "extreme feminists", like the ones who talk about how much they hate men and how bad men are and stuff, which im not sure even really exist and are definitely not the majority.
They certainly aren't helping by any measure, but I am attributing the blame to the general movement.
From the simple things, (for example, the name of the movement itself containing "fem" automatically causes one to assume it's only for women due to how prefixes normally work), as well as the perceived lack of focus of men's issues as well as women's issues, it's not terribly hard to understand why your average person who isn't either actively engaged in the movement or hasn't done their research doesn't understand what the movement is for.
In my particular situation, for instance, I never knew that Feminism was for equality of the sexes until I went out of my way to look up the actual definition after an argument, and I still meet people on a regular basis who don't know that the movement is for equality, not just the betterment of women.
It's something simple, really, but the effects are pretty clear.