Ok, so the "dreaded 3.5 GB wall" is in reality not a big deal. It all started with Shadow of Mordor on ultra textures running better on 980. But since the whole big debacle about that, Nvidia has released several drivers that have improved the situation. I've pretty much tried the game after each driver release and especially 350.12 made it much better. I run two Gigabyte GTX 970 overclocked in SLI.
The killer was when I upgraded from 8 GB DDR-1600 to 16 GB DDR-2133. Now Shadow of Mordor in 1440p with ultra textures doesn't stutter much at all. I played about 30 minutes and during that time maybe saw a stutter once (excluding ones right after the world has loaded the first time).
I'm guessing the game just is a memory hog overall. Even with a Titan X it seems to eat nearly 6 GB, which is what the developer recommends for ultra textures.
I could easily play Far Cry 4 at 4K DSR as long as I didn't use 4xMSAA, after which the game eats over 4 GB VRAM and starts getting small stutters at times. 2xMSAA is fine. GTA V also runs quite nicely at near max details (I think I just have motion blur off and advanced distance scaling at half) with 4xTXAA + SweetFx Lumasharpen filter.
So for one card I think you have to use settings that are beyond the GPU's processing speed to run into any real VRAM issues.
I have no idea why doubling RAM helps so much but it definitely does. Maybe 8 GB is too little nowadays, I don't know.