I think it often is. Especially if you're using some pad from hell like the 360 one. A lot of people can master the pad and do just fine, sure. In terms of raw input, specifically the likelihood of unintentionally hitting a diagonal, a pad is at a significant advantage. Console ports correct for this, by the way. Play an unmodified arcade game on your PC. You're expected to hit a diagonal for a forward jump pretty much spot-on, not the relaxed inputs in something like SFIV for consoles that lets you slide into that motion with some forgiveness.
Those accidental inputs are why SNK created the click stick for their home platforms. It gives you the assured control of an arcade stick while allowing you to hold a smaller input device in your hand. They didn't go out of their way like that for no reason.
On higher tiers, I think this goes away. For SF games, there is history there so pad players are rare. I think the Tekken scene has lots of pad players, though, or at least I faced some locally at the arcade who brought their pads, and it seems like whenever I've seen a pad player over the years, it's usually been for that game. They must know what they're doing if they get that far. For lower tiers (I.E., 99% of us, even good players), having a lower chance of wrong inputs definitely affects the likelihood of winning, especially if the skill levels are close.