Okay, I'm gonna spitball for a bit (will admit some of this could be slightly overly optimistic on my part). I'm having a hard time picturing it's a proprietary SSD. The only case I can see them going this route would be making it similar to Intel's Optane memory. It's similar to RAM (I'm assuming in that it can provide random access? I know for sure the Persistent Memory modules do, I'm guessing the lower-tier ones just do it slower) and 32GB modules usually go for something like $60.
Guess if these PS cartridges are taking this route, they could source components at a large enough scale to design 64GB or even 128GB versions of that sort of thing for comparable prices or slightly higher (say up to $80). IIRC Optane Memory has reads of around 900 MB/s to 1400 MB/s (1.4 GB/s) and writes of around 145 MB/s - 300 MB/s.
Which is actually much slower than, say, the Gigabyte Aorus NVMe Gen4 SSD (5 GB/s read, 4.4 GB/s write), let alone GDDR6 or HBM2, but keep in mind stuff like Optane is meant to provide random-access and byte-addressable writes of RAM with non-volatile storage of NAND in one package. It would basically function like a large cache, and 64/128/256 GB capacities at prices between $60-$200 would probably be doable at large enough scales to get the pricing for components as low as possible and still be profitable while providing an SSG-style benefit with more robust NAND (3D Xpoint vs. traditional NAND), preferably over a dual PCIe 4.0 connection (if that would help bring even faster speeds than the numbers noted for the Optane; otherwise no reason to interface it over anything over 1x PCIe 4.0 lane tbqh).
I would prefer that over a SSG-style setup with the IC soldered to the board especially if the benefit for that wouldn't net any notable speed advantage. Plus with a replaceable solution, you can always upgrade. The benefits would be just big enough to make a noticeable impact on game performance, but not SO big that it shifts the development target for developers into wide gulfs of performance (which would create something more like a PC development environment). Otherwise if that sort of situation were to arise, it'd be better to just standardize to one spec integrated directly to the system itself.
If devs take advantage of any machine-learning capabilities the systems may have, they could just use a certain amount of any given cartridge's capacity as a constant to program against for specific game logic, assets etc., and then have other available amounts usable for algorithms generated through the machine-learning process. I could see PS5 (and Scarlet, if it's taking a similar approach) shipping with a 64GB "cartridge" of the aforementioned Optane-style memory pre-installed as the basic configuration, and they might offer larger-capacity ones optionally running up to 256GB, but those probably wouldn't be available at launch, just other 64GB versions.
And no, this wouldn't replace a traditional SSD; I'd expect PS5 to have an M.2 slot for optional drives, operating on some version of NVMe 1.3. If Sony did something like I'm picturing here, they're never going to be able to make a 'cartridge' of this stuff at 1TB, 2TB etc. sizes that is cost-effective, but they certainly aren't going with platter drive support unless it's external over some USB, either. Hence the M.2 slot for third-party SSDs.