I don't think it's the console itself per se - but the content that such console provides. As per the console itself in this generation - Xbox doesn't look too hideous or too huge or ugly - as it's pretty minimal looking, clean box.
We can't really compare with something like iPhone - as the phone is seen as an object first and foremost. Functionality is almost secondary - as most of them touch base with doing internet, LINE, and other SNS functions etc. How cute it looks or premium it feels matter to a lot of people.
Consoles, although the design itself is important - the most important thing is the content that it provides. Xbox, provides a whole load of western games that Japanese don't care much about.
That's the bottom line... games. Xbox is full of western games, and that's the issue. Western "looking" games aren't just popular, nor big enough driving force. (Minecraft being an exception, but it's... more of lego looking, tbh.)
Unless they can bring in games that Japanese people love to play, they won't be successful in Japan. You can't force the public to buy something they don't really care much. And a lot of Japanese gamers don't care much (exceptions do apply) of western games filled with orc looking female characters, guns ablazing in a setting they can't really connect, dismissing some great gameplays - just like how westerners look at Japanese games are just weeb content filled with waifus, dismissing anything good about it too.