Casual users are already on mobile. Plus, in terms of games, I'm pretty sure AppleTV will be compete with mobile phones, not consoles.
Yes - that doesn't restrict them to mobiles only though, as many of my friends will attest. This device is not a mobile phone and not with the same use cases. This is like saying the Wii was only competing with mobile games and not consoles - it misses the point. Most people don't care for the difference between the two as you are. They are interested in a device for the living room that will give them some games for party purposes or just a bit of fun, but importantly, a wealth of entertainment and internet services they already love - easily accessible for their TV. That makes it competitive in the lounge and not as a mobile device, whereas a phone is not something you share all the time and serve around a living room and TV. That will have an affect on consoles whether big or small dependant on its ubiquity and how well it caters to gaming in a palatable way for average folks with a passing interest.