Feel free to disagree here, but I think you have, or are positing, a dangerous misconception, which is that both claims on either side are valid. The reality is closer to something like: if the claim is made that "I have no opinion" in regards to something like same sex marriage, you are not taking an unbiased or neutral position. In fact, it points specifically towards an obvious bias, which is your disinterest in the given example issue. Disinterest, of course, being either evidence of or by itself, a lack of concern in regard. A lack of care.
In the case of your example of same sex marriage, could lack of care about the issue logically be called affirmation? No, I don't think so. But I do think the claim that you are against the cause specifically because of your neutrality, or lack of care, is a logically valid claim. The people who "didn't have an opinion" on same sex marriage before its legalization in the United States were on the side of inaction, of disinterest, of a lack of care. It was an implicit approval of the status quo which was against same sex marriage. That is not neutrality.
Also, I don't really understand your last claim. How could one could consider either side "despicable" and also have "no opinion"?
I think Alo0oy is right in saying that in most cases, political neutrality is choosing a side.