This phenomenon we see happen with modern Nintendo is what I call the NSMB effect:
DKC Returns & Tropical Freeze - Lack the charm and atmosphere of the original trilogy, I've seen people say they don't want to play it because it's similar to the movie Madagascar in tone rather than the more moody and serious tone of the first three.
NSMB U - Some people are turned off by the presentation to the point they aren't interested in playing it no matter how good the level design is, the music and visuals are soulless etc.
Yoshi's Woolly World - Described as bland and uninspired compared to the original SNES game.
Super Mario 3D World - Not a true 3D Mario, Galaxy was OK but hope the next game is more like Mario 64, however this kawaii cat suit game has more in common with the 2D games and doesn't have the epicness or immersion factor of the N64 masterpiece. People genuinely thought it was a 3DS game when it was revealed.
I'd argue all four/five games you mentioned are leagues away from what's going on with Sticker Star and Federation Force.
None of those games significantly changed the base gameplay of their prequels (SS) or are spinoffs that go in a significantly different direction from the main series, even by the standards of other spinoffs (FF). And while there are a handful of fans of the older games that don't agree with the changes, they all received a good-to-great critical reception from the general fanbase as well as reviewers and got decent/great sales (save for Tropical Freeze, but considering the sales of other Wii U games and how DKCR sold on the Wii, one can argue the Wii U's install base was responsible for that, rather than the game's quality).
SS in contrast has received decent reviews and sold well, but was also less well-received than its processors by reviewers and is greatly contested by the fanbase; and while the jury's still out for FF's post-release critical reception and sales, pre-release reception by fans has been predominately negative.
I can go on and on but I think you get the idea. My point is that this is different from your Banjo analogy, that example has more to do with the fact that colorful mascot games or stylized games with arcady gameplay don't sell to the Xbox and Playstation audience.
The Xbox series had Viva Pinata in regards to colorful mascot games, and that did fine; but other than that I can concede that point to you completely (lol Blinx the Time Sweeper).
Don't know why you listed PlayStation at all though, they've had a whole bunch of those colorful mascot games-Crash, Spyro, Jak, Ratchet, Sly, and LittleBigPlanet. All of those as well managed decent-to-great sales (even LBP, while it got off to a slow start at first sales picked up over time). Even the PS4's Knack got decent sales despite it's quality.
Concerning stylized games with arcadey gameplay, while I can't think of anything for Xbox, PS has Wipeout, another IP that managed decent sales.
Even if a traditional Banjo was guaranteed to not do well at all on the Xbox market, that doesn't mean something like Nuts and Bolts should had been made. A Banjo game may not have done blockbusters, but it still would had been a product the fans wanted at the very least. For many people, Nuts and Bolts was not that product, even if it was a good game, and the sales reflected that.
If they really didn't want to do a traditional Banjo (which wasn't even the situation at first--the teaser trailer for what would become N&B heavily implied it would be a traditional sequel like the first two games), that's fine, but Nuts and Bolts should had then been a new IP entirely and the Banjo IP should had been left alone.
This is why Bloodstained, Yooka Laylee and Mighty No 9 kickstarters are things and why Nintendo is funding Bayonetta 2 and possibly BG&E 2 as no other publisher wants to go anywhere near those games.
I'll give you the KS examples, but I doubt Sega scrapping Bayonetta 2 was because they thought it wouldn't sell. Bayonetta was actually one of their better-selling titles in addition to being critically well-received. Sega canning the game was more a result of the game being a statistic of several games they abandoned and the company drastically downsizing its IP focus in the then-wake of a huge financial loss. Also worth noting that while Nintendo funded the game, Sega still had publishing rights, so its not like they wanted to wash their hands of the IP completely.
This all goes back to the original point that Nintendo is a gameplay company first and foremost, I don't understand people who dismiss modern Nintendo games for aspects regarding personality and atmosphere when the gameplay is what really matters.
Given how a chief complaint for both SS and FF both pertain to their gameplay this last bit doesn't make sense to me.
Even if that wasn't, it's pretty obvious that while gameplay is good, earlier games strove for more than just good gameplay and that's why they are beloved. Streamlining the series down to just good gameplay is taking a huge chunk of what makes them special away and people aren't going to like that, to the point of making a decisive vote with their wallets.
Though like I said, given the gameplay complaints about both SS and FF, even the gameplay isn't even safe from this, so the "gameplay takes priority" argument doesn't even make much sense in this instance.
