It just degrades the whole package and experience of a 4k console. Why not make the UI render att 480p if it doesn't matter, should free up even more RAM, right? To me it's often the little details that make the difference between a good product and a great product.
Your 480p question maybe highlights a lot of the reasoning, because it isn't a hd or hd-ready resolution. It is an EDTV resolution, so consoles that only have hdmi for guaranteed hd-ready or better resolutions don't need to support EDTV - which is less than 720p.
So if we then ask: "well why not just do 720p, then? ". I would speculate that 1080p looks big enough (text scaling) even on the smallest screen option, because it is a super sample, it looks more detailed on hd-ready screens, and going the other way scales brilliantly on the vast majority of budget branded 4k tvs, too because it is a small 1:4 scaling from high quality starting point, and budget screens are rarely good enough to easily discern 4k and 1080p sources apart at regular viewing distances IMHO.
As for 1080p not being good enough, IMHO it is certainly good enough with the PS3 or PS4 UI, so if it isn't good enough with XsX/Xss/X1X/X1s then I suspect that's more to do with the UI, and not the resolution.
Why Xbox has stuck to 1080p for the UI I would speculate is actually to do with shared UI development with Windows 10. Unlike the PS4/X1 onwards that require encrypted hdmi connections that provide the consoles with EDiD info that contains resolution capabilities and the screen size to allow UI text scaling to be set appropriately, Whereas with Win10 PCs they still have to work with legacy unencrypted monitor connections that may not provide accurate or complete EDID info through all connection types, which can mess up the default windows 10 resolution selection - and potentially damage some monitors if the selected timings were out of range - as a result of the UI selecting a 4K resolution beyond that TV input's capabilities.