Saying he's vulnerable to magic means is usual durability granted by his solar aura doesn't come into play.
ONLY if the nature of the magic specifically circumvents durability. Superman is as susceptible to a "Turn into a bunny" spell as Hulk is. Both are susceptible to weapons enhanced to "cut anything." But Superman isn't more susceptible to a strike from Harry Potter's broom, just because it is enchanted to fly (no aspect of its magical nature is to make it strike harder).
The latter perspective is that Superman is allergic to magic. Your interpretation isn't quite as bad, but it is at least that [just] his invulnerability is allergic to magic. This view can't make sense because it either requires a scientifically based character's power set to have some form of universal interaction or awareness or special sensitivity to ALL magics irrespective of their nature or rules. So making out with Wonder Woman should make him break out into hives (if just allergic). Or his powers have to somehow "know" that he's being struck with a broom meant for flying rather than just an ordinary broom.
Conversely, the foundations of all magic have to be unified and specifically target and circumvent Kryptonian invulnerability. So even when casting an enchantment to make a broom fly, there's a secret fine print in the spell saying, "Oh, and if you happen upon a Kryptonian soaked in a yellow sun, you will defy his invulnerability."
Rather, it makes much more sense that Superman is simply subject to physical laws (to the extent that his powers are likewise permissible in his world's physics) and to any breakages of those laws as well. A weapon enchanted to hit harder than physics allows, cut better than physics allows, etc. would have that impact upon him... but with his relevant powers intact. Summoning, say, a piano overhead... would defy physics, but the strike would fall completely within the boundaries of physics and be harmless to Superman, despite being summoned magically.
Attacks that are specifically magical in nature muddy the water. Magic that creates a fireball, for example, if done in a way that physics takes over after its creation, would be harmless to Superman... but if it included aspects of "burn when you shouldn't otherwise be able to burn" that kind of aspect would circumvent the physics of Superman.
This allows consistency for the physical rules of Superman AND for magic. While magic defies natural rules, it doesn't mean that it doesn't have rules of its own. You can't simply cast a spell to enchant a broomstick to fly, then decide that snapping such broom in two will cause the death of Voldemort. Rather, everything interacts with the broom as it was enchanted to do. Likewise, Superman interacts with magic as the magic is meant to do... rather than be some special exception to all magics that either his powers or magic somehow recognize.