Up your dexterity and you won't have this problem nearly enough. It's an intended game mechanic, and it forces you to make choices of what you're willing to live with.
As for the the topic, if you're not looking into the realism aspect of how a horse feels to ride, which shouldn't even be a thing since it's a fantasy rpg, then the controls are super responsive and you can do most of what you want to do without it feeling cumbersome. The only thing I dislike is that it is grouped in the item slot so you can make a mistake using flasks and other valuable items under pressure. There is a work around (on controller) but it still isn't simple since it requires more than one button press.
I still believe MGSV's horse is still the best overall though since it has some level of realism in how the horse gallops but is still easy to control, and it doesn't just appear out of thin air right in front of you. In Elden Ring, the way how you summon Torrent makes it an easy copout from tough situations that you didn't have in prior fromsoft games.