I'm pretty sure the answer is "Shigeru Miyamoto".
Gunpei Yokoi invented the NES controller, and it was great. He invented the SNES controller, and it was great. But Miyamoto had a major hand in the N64 controller, because it was basically being arranged around Mario 64.
The Virtual Boy, as you may recall, had twin D-pads, triggers, and four face buttons (a modest two for each thumb). I'm sure Yokoi would have given it twin analogs instead of D-pads if he had the budget (although he would've given the display more colors first), and left to himself, would've probably released a handheld with twin analog as soon as such a thing became feasible.
But then Miyamoto became the god of Nintendo, and Gunpei Yokoi was fired. And Miyamoto started talking about how controls had gotten too complicated (no doubt influenced by the N64 controller which was made to suit his own game), that controllers were scaring away the old people like himself, and that they needed to be simplified drastically. His concept for the GameCube controller was a freaking one button controller, with lesser so-called "satellite buttons" attached because people mocked his idea of a "win button". With the Wii Remote, he used waggle as an excuse to toss pretty much everything except for a D-pad and two buttons. At first, he didn't even want the analog nunchuck, that was just there to satisfy third parties and Retro.
The GameBoy Advance was obviously based on the SNES controller. Somebody decided that the SNES controller had too many buttons, and all signs point to that somebody being Miyamoto.
I don't believe the SNES controller had too many buttons. I think the GBA should have had four face buttons. And they should have been the four colors of the Super Famicom controller's face buttons.
