So far Cortana in it's current implementation makes the voice experience on Xbox far more frustrating, slower and less reliable. I'm not sure whether this is something they can reasonably fix before the planned Summer release. Here's my experience with Cortana so far:
When I issue commands to Xbox it is causing non-stop searches, chatter, and actions on my nearby Windows 10 phone, PC or tablet. For example "Hey Cortana Go Home," an essential and frequent Xbox command, will initiate GPS navigation on my Windows devices. Asking Cortana to play a song or artist would cause utter chaos as each device begins playing music. Most other commands will cause confusion on nearby Windows devices which then produces a Cortana response that gets picked up by the Xbox causing the Xbox to also get confused. Two possible solutions I can think of is 1.) to add an option to only activate a response when you direct Cortana to a device, "Hey Cortana Go to Xbox, Hey Cortana go to phone, Hey Cortana go to PC." Until you tell her where to listen to you she stays quiet. That would work for me. Option 2.) just let us choose the name of the assistant on each device. I don't know if the marketing department likes that solution, but it would work very well and personalize the digital assistant experience. A third and best solution would be that they make the whole service smarter and use any known inputs to only activate Cortana on the last device you interacted with.
As it stands right now even if I turn off those nearby Windows devices (a poor/unacceptable solution) the experience with Cortana is pure frustration. It's not a matter of words not being recognized as the words that display on screen are generally accurate. The major problem seems to be the cloud processing which is both unreliable and slow. When Cortana's servers become overwhelmed, something which happens way too frequently, the utility of Cortana on all Windows devices grinds to zero. "She" tells you some quip about not communicating with the internet or something silly. I get this response from Cortana about 85% of the time now. This might be mildly annoying when you're looking up facts on a web search, but when you're trying to do basic navigation like open YouTube, get back to the previous screen, or pause a video it becomes unbelievably frustrating. My final problem is more of a complaint towards Cortana itself which is that over the past couple years of using it I see many more features being added, but not more polish or reliability. When Microsoft delayed Cortana on Xbox I thought it was to polish the experience for consumers, but it seems the only focus was to just get a bad Cortana experience functional with headsets. It is obvious that Cortana is more powerful than previous voice controls and can do far more than anything we had previously, but it may also make the voice experience so unreliable and slow to the point of not being useful at all.
Hopefully they can surprise me and improve the speed and connection reliability before launch, but for now I just see this as potentially yet another step backward for Xbox One's utility.