They've never really tried since the N64 era. The marketing of that console was perfect for the Western audience, from "mature" software to ads themselves. Goldeneye was the game of the generation for older crowd interested in shooters, only for Nintendo to completely ignore them with the Gamecube and Xbox to swoop in and take them with Halo. Nintendo went as far as to advertise Conker's Bad Fur Day in Playboy. Crazy to think about today.
People forget how well the N64 fared against the Playstation in North America. It sold as many consoles as the SNES. As of March 31, 2001, N64 had sold about 20 million units in North America. At that point, Nintendo stopped supporting the console and started focusing on the Gamecube. Playstation was sitting at 31 million at the time. Keep in mind PS1 was released before N64, and had literally all of the third party support in the world. This resulted in software droughts that the PS1 simply didn't experience. While Nintendo's first party output was strong (Mario, Zelda, Mario Kart, Goldeneye) it couldn't compete with Sony's first party support (Gran Turismo, Crash Bandicoot) AND third party support (Final Fantasy, Metal Gear Solid, Resident Evil... the list goes on and on and on).
The idea that the PS1 dominated the N64 comes from worldwide sales (it KILLED N64 in Europe and Japan), and it continued to sell WELL into the life of the PS2, while the N64 died completely when the Gamecube released.
What's my point? The N64 sold well in North America on the strength of two things:
1. Compelling software
2. Strong marketing that appealed to ALL demographics
Their biggest problem? Cartridges. Yamauchi said fuck you to the third parties, so they said fuck you back and left Nintendo for Playstation and CDs. Could you imagine if Nintendo went with CDs and had Final Fantasy, Metal Gear and third party support they enjoyed from the SNES on the N64?
With the Gamecube, they hit 1 to an extent (lack of shooters hurt them, the Goldeneye/Perfect Dark audience moved away from Nintendo) and failed miserably with 2. Gamecube was the kiddy purple lunchbox with Mario's water gun and Celda. The Resident Evil games were too little too late; the damage was done. They had decent third party support for the first few years, but when you are lacking in 1 and 2, it's not enough. Doesn't help that PS2's third party support still completely shit on the Gamecube's. There was no Final Fantasy X, MGS2, Devil May Cry or Grand Theft Auto on the GC.
Wii was an anomaly. They hit 1 (Wii Sports/motion controls) and 2 (Wii Would Like To Play) so hard in such a different way that it hit the market like nothing anyone could have ever expected. They didn't need third party support for the Wii.
Now we reach Wii U, and we are once again at a Gamecube situation. I'd say they have even less compelling software and worse marketing than even the Gamecube, and the sales numbers show that. Obviously they tried to pull a Wii with 1 (the gamepad/Nintendoland?), but they didn't even try with the marketing. The first ads for the Wii U were those fucking dubstep ads. Compare that to the Wii Would Like to Play ads for the Wii, that sold everyone on the thing in 30 seconds. Terrible fucking job with the Wii U.
Why did I write this long post? What's my point? I very often see people argue that "Third parties coming to Nintendo wouldn't help anything, because the audience for Nintendo consoles aren't interested in those types of games." Well I point to the N64 and say "You're wrong. They just aren't trying hard enough, or at all." Here is the perfect Nintendo console:
1.
The compelling first party software of the N64. They all had the signature Nintendo charm and quality, but also revolutionary titles like Mario 64 and Ocarina of Time that put the entire gaming world on notice. Unique titles that had them step out of their comfort zone, like Goldeneye, Perfect Dark and Pokemon Snap. New IPs that spawned their own successful franchises for years to come, like Mario Party and Super Smash Bros. All not only some of the best playing games in the world, but also some of the best looking.
2. Strong marketing that appealed to all demographics, like the N64. N64 wasn't marketed and presented as a kiddy console. As a result, it wasn't known as a kiddy console. A game that involved shooting people in the fact was the third best selling game on the console, even beating out the revolutionary Ocarina of Time. It was not marketed exclusively to young kids and families, but instead a teenage and older audience. There were ads for the N64 during wrestling and in Playboy magazine. This gave the console an aura of "coolness" which sold younger audiences on it even though it wasn't being marketed DIRECTLY to them.
None of this cringeworthy shit.
3. A console built for third party developers, like the PS4. This point is the one Nintendo never hit, and explicitly stopped trying to hit starting with the N64. With the PS4, Sony went to the third parties, asked them what they wanted from their new console. After gaining this information, they built the PS4, which is essentially the dream console for third parties. This is evidenced by the unanimous third party support for the thing. Nintendo obviously does not share this philosophy, instead building the console themselves, completely internally, and
then presenting it to third parties. "So, what do you think? You'll make games for it, right?" That's not how it works today. Nintendo needs to look at the PS4, the current market leader, and
do that. Many say a Nintendo console that just does what the competition does is boring, or wouldn't sell well in the marketplace. It needs to be "unique". PS4 isn't doing anything unique or fancy. These are basic, fundamental traits that all video game consoles should have. Then you innovate ON TOP of that foundation.