It's a long story, but the executive summary is Playstation happened.
When it was still Nintendo and SEGA, SEGA tried to distance itself from Nintendo and its child friendly image by being the more edgy console for older boys. Black console, more shooters etc. While this worked very well for the Master System and Mega Drive / Genesis, it's also a bit of a risky demographic in that (especially at that time) gamers grew up and out of the hobby, so whereas Nintendo had the benefit that their demographic entered on their system and could stay in the Nintendo fold for longer, SEGA continuously needed new influx. Luckily, they were a bit of an arcade name, which fit well with the demographic.
However, their demographic was deteriorating in more ways than aging. SEGA went all crazy on hardware but did not have enough software development (or talent frankly) to justify all the addons. Meanwhile, the SNES proved a veritable powerhouse. You just needed the one console (and perhaps a multitap and if your were European an NTSC convertor), and you could play a crazy amount of classic games. Sega added in the Saturn which was a dud, and even the SEGA faithful were starting to feel burned on their SEGA products.
So we have an older demographic that stayed with SEGA because of their childhood but by and large was not completely crazy for the name, and SEGA needed new influx from casually interested, or perhaps Nintendo fans getting older, but this was console wars prime.
Then Sony entered. A name that was much more well-known in Europe and the US among the casually interested, and a big domestic champion in Japan, and it was targeting exactly the same demographic as SEGA. So the casuals flocked to Playstation--the exact same group SEGA needed to stay viable, as did some of the weary SEGA fans of old who were burned on SEGA but didn't want to go to teh kiddeh Nintendo. Nintendo fans meanwhile stuck to the fold, or if they also perceived it as kiddie would much rather jump to Playstation than to the bitter rival.
By the time Dreamcast launched, there was hardly a demographic left. Japan had three domestic champions to choose from, of which Nintendo and Sony were more popular. Europe was a smaller market and more Nintendo inclined; there were also more casuals who opted for Sony. When Xbox swayed the last of the American demographic, SEGA even if they weren't already in dire straits, had practically no-one left to sell their console to.