The main reason you're seeing this is because console engines are being more and more optimized for GCN (console) h/w which in turns means that they are less and less optimized for NV h/w (as these optimizations tend to go into the opposite direction these days) - only a handful of developers are spending any measurable amount of resources on PC version renderer optimizations, most just dump the code from console versions, add some LOD tweaking options and you're good to go.
If you reflect back a bit - Kepler was launched 1.5 years before the new consoles, and the most issues with it have started last year around this time - some 1.5-2 years after the new console launches, when devs have pushed the most out of the new console h/w. Same issues are manifesting themselves on Maxwell as well but to a lesser degree as Maxwell fixed a lot of Kepler's slow paths.
Polaris is unlikely to be a big change of GCN architecture and will probably bring improvements in vein of the usual previous GCNx updates. However I think that things will be a bit different going forward.
A. Because Pascal is basically Maxwell which means that the single NV optimization target will remain for the next year and this should help with PC optimizations where they are actually being implemented (supporting FL11_0 Kepler at this point is a pain in the ass for most multiplatform developers so this is essentially being dropped slowly but inevitably).
B. Because console optimizations are likely to be close to their peak at the moment and thus it's possible that GCN h/w won't receive as much uplifts from consoles as it did during the previous two years. The big unknown here is the influence PS4K/Neo will have on the scene though. It's possible that thanks to it we'll see a continuation of that trend on higher end of Radeon lineup (lower end is likely pushed to it's limits by PS4/XBO already).
Kepler, Maxwell and Pascal are all just optimizations of the same architecture. They aren't all completely new re-writes.
That's not accurate. Maxwell and Pascal are very close, that's true, but Kepler is a different architecture which is closer to Fermi (and even Tesla in some regards) than to Maxwell. Obviously no new architecture is a "complete re-write" as that would be very inefficient from R&D point of view. But the amount of changes which were put into Maxwell is actually very high. The next NV's arch which may have the same magnitude of refresh will be Volta, and then, yeah, something like Kepler-Maxwell can happen again, with Pascal GPUs loosing a lot of ground to the newer Volta GPUs.
I'm optimistic on this though as Pascal should be pretty good at handling most of GCN console code and because of this I think it's unlikely that there will be a lot of areas where radical improvements of Volta will bring appropriately radical performance gains when compared to Pascal. Volta's arch may actually be a bit too complex for the typical code it will run in 2017/18 as it's likely to push beyond DX12 FL12_1. But guessing this is always inaccurate.
One of the main concerns is whether Nvidia's cards have gotten worse COMPARED TO THEMSELVES, an issue that has been brought up in this thread. If this were merely a matter of drivers, we could simply install the different drivers from those time periods and verify whether or not newer drivers are gimping the hardware.
They didn't. This isn't an issue of drivers or gimping, it's an issue of the general change in the average rendering code practices which is happening because of new console h/w.