It's the person and the points. The nuance is typically lost when people try and express themselves, but when someone says they're voting for someone because they're part of X group, it's not typically the entirety of the calculus. Someone voting for Barack Obama "because he's black" is unlikely to vote for Herman Cain or Ben Carson. Someone voting for Hillary Clinton "because she's a woman" isn't likely to vote for Carly Fiorina or Michelle Bachmann.I think it is more of a "care for the points, not the person."
For much of social thought, the person matters more than the points. Look at the Republican party to see this. You can't even get all of the candidates to talk together to come up with one unified plan of any substance.
I may be too Zen to really 'get' why person>points, because points can transcend people and groups if they're accountable. But I suppose I empathize with those think people can bring about points that work for them, because again, we can look at Republicans for the Law of Attraction regarding anti-reason.
And it's generally fine if personal identities don't matter for a given person's vote. The problem arises in telling someone else it should/shouldn't matter to them.
.................Voting for a woman only because its a woman might be understandable from an emotional standpoint, but its not smart from an intellectual standpoint.
I think you should have probably thought through the way you expressed your point a bit more. But read above.