I don't think controllers are simple devices at all. The Wiimote was the simplest controller we've had in a while and it left many lacking when it came to available buttons. I don't count swiping to be substantial enough to draw my interest or even justify the inclusion of a trackpad. I can see menu navigation and a few other functions working with touch input but without an actual screen it seems limited.
No. It is exceedingly simple. Everyone knows and understands how to use a button. They are in a multitude of devices/appliances. Humans understand that pushing a button often nets a reaction. Its a simple interface design.
Analog sticks have a higher curve if you've never used one, but still simple in its design/effect. While not utilized in daily device interaction, it is easy enough to understand function.
Trackpads are no different. They allow for simple gestures most humans use without even knowing about it. They dont have to be finger gestures, but gestures with the hand, in general. From the turning of a doorknob to the key in a vehicle ignition, waving to friends, pointing, turning the page in a book - humans are conditioned for gesture action/reaction. This is perfect for video games.
You also sound like you are limiting what you think you can accomplish with a trackpad. It is most certainly not limited to "swipes". Theres a lot that can be done to make things interesting enough without looking... a few examples:
Swipe: in a FPS, the user holds the left trigger to pull a grenade. The right thumb slides left to right to "pull the pin". Now the grenade is cooking. Release to throw.
It can be used as a simple 180 swipe to turn around faster since analog is dedicated speed.
Circle: Hold a button/trigger and perform a circular motion to operate twist/handle/wheel devices.
Patterns: Perform specific movesets for X action. U, Z, C, N, L - any easily recognizable and performing pattern.
Just off the top of my head examples. Currently I am designing UI features and menu/inventory management for my game, PC (controller, ready) and I am looking at how I use everyday items in my house, out and about, my phone, car, utensils, etc - watching my wife, friends... trying to make an intuitive, "no instructions needed" robust yet self-explanatory system. Basing it off of how we condition ourselves to doing things without focusing attention directly, looking at, etc. It should just "click" and feel natural to navigate and use.
Again - the design allows for simple gestures - just enough to change things up without making it an overly complicated mess that compromises the rest of the controller.
Keep the player focused in the game by not having them look down. Keep it interesting and engaging by introducing elements people already understand.