I wonder why no other devs tried to go for the streaming/loading-as-you-lay approach?
Because it adds a whole new layer of complexity, notably because optical drives are slow and not guaranteed to always work at an optimal speed, so handling how to disguise load misses becomes a big part of it all. Metroid, for one, doesn't solve it in a very optimal way, adding doors everywhere and allowing their opening to lag quite a bit behind the player triggering it.