"The only thing I can think of" my ass. I call bullshit on this assumption based 'theory'. So instead I'll do the same and offer mine.
My guess is it's not testosterone levels since on average, Black women benefit from increased sizes of things that are scientifically proven to appeal to men and their sexual desires. Those being bigger breasts, hips, thighs and ass. Black women usually have larger such areas and are more hour glass figured, or more voluptuous. Such figures are not only biologically more attractive to men, but also better for child birth.
So the question is why then are more men less attracted to Black women (me included)? I think it's to do with socially induced stereo types, accepted opinions and prejudices. From a young age we're constantly, through media, nurture and other things shown what is more attractive or societies more 'ideal' look. And the vast majority of the time it ain't Black. Heck even the famous attractive Black girls on TV aren't really Black, not in the darkest sense anyway. These days they look more half-caste or brown than deep traditional African Black.
There's still an element of racism attached in all of it. I know because it's still prevalent among Asians. Heck, I grew up with it. The fairer the more attractive, the Blacker, the less attractive, less higher class, poorer, less educated etc. You go to Bangladesh and watch some TV. Every beauty advert is about how it can magically turn you from dark to fair and make you that much more attractive and beautiful as a result. It DISGUSTS me.
In typical sexist fashion, for some reason this is only extenuated with women. With Men, I guess because women tend generally to either be less shallow/fussy or have less choice, skin colour isn't as much of an issue.
Luckily media's portrayal of Black men, and the general stereo type is that whilst they may be many things less, what they do make is good lovers, good dancers, better body types, bigger penises, more smooth lovers and so forth and so forth (a lot of it probably true), hence why Black men are less likely to be stung with the same negatives.