Is what is being said impossible, or is what is being said simply different than the traditional way of doing things? I am asking because, sometimes people have trouble seeing new possibilities when they have been in something for too long. Like, it's hard for a tennis player to bat properly in baseball, and vice versa.
There are variations that could be considered.
Imagine we wanted to create a very fast cache that could be written to directly (or even via VRAM, as caches are good for repeat reads anyway). We decide to make it bigger and more useful to pull stuff from the SSD or just commonly used stuff from VRAM. Let's make it 64mb or even 128mb. We also want to make it part of the die, to improve access times. The fastest way to do this is SRAM (which current caches are made using) which access nanosecond access times. 128mb might not sound like much, but there are 16,000,000ns in a single 16.6ms frame (60fps) and SRAM has around 50-100ns access times.
Congrats you have just created ESRAM which the Xbox One uses, which not only increases the die-size and heat profile of the chip, but was so hard to use, it was removed from the Xbox One X.
Developers like large, homogeneous hunks of RAM, even having two different memory profiles like the XSX has is worrisome (it's not a massive gulf, but you have to wonder why they did it.
I think a big part of what people are missing is, once you have the data in RAM, the next part is important, and it's that part where the XSX will shine. (Rendering).