TL;DR - Build resolutions based on total pixel count. Use your max DS res as a baseline to determine best AA (for the most part). Then, you can find a max count, and use this formula to use whatever AR you want:
a = squareroot(total pixels) and then AspectX(a) = ResX, AspectY(a) = ResY.
At the upper limit, nudging the total pixel count up, and determining a res that way will allow you to wring the most out of your IQ (but will likely require a windowed mode). The same is true nudging downward.
The Idea:
So, I often find it tough to get the right image quality out of a game, because I am hitting the vram wall. The card is will to slog through the frames, but its not allowed to thanks to a framebuffer that is just too large for the vram. This is nothing new. And surely alot of folks have the vram headroom, but the card hits its limit in power. The same will still apply.
But working around this brought a few ideas to mind.
First there are the variables. Similar to photography, where you balance what you have available (iso, shutter speed, aperture, etc.) with what you want aesthetically (exposure, motion, DoF), with IQ in games and for screenshotting, you have a number of things to balance, really only towards either or both of two ends: FPS and Aspect Ratio. You have your vram, your GPU power, your resolution, and your AA.
I am clarifying a way to balance these things most optimally, mainly by approaching things with 'total pixels' in mind. Its not the horizontal resolution that determines vram usage, but the total of course. Obviously 3840 x 1634 is lower res than 3840 x 2400. (~6mil vs ~9mil)
The idea is really just that you find a good 'upper limit' baseline, and tweak from there. When tweaking, two more variables arise, downsampling limit, and fullscreen/windowed (or unlocked) capability in the game. Most games seem to be able to have some way of forcing into a windowed mode and run beyond your downsampling limit, especially with SRWE and Stein3x's recent find of SoftTH.
Obviously with the downsampling limit, you can't surpass it to gain extra IQ. But in some instances, you may want to step down from it, but for instance, only icrementally, or in a different AR.
So, its not really a very complicated concept, I just plan to outline the different ways that referencing total pixel counts helps to max out a game, or moreover your system in general, and streamline the 'trial and error' method.
First you take your max downsampling resolution (as it is universally runnable), and determine the total pixels. For instance, mine is 3840 x 2400, which is about 9 million. Select this resolution and see what you can do (which is the usual SoP). Then you'll have some sort of situation with the game
There are basically three:
1. With Antialisng, you've either maxed it out, or you max out a certain level. You want to try and squeeze out some more IQ
2. The above, but you want to squeeze better FPS out.
3. The above, but you want to try a different aspect ratio.
4. You can tell that you are very close to running the next AA level, thanks to how your vram usage looks.
In any case, you use your max DS as a baseline to see where to look for your true 'max pixel IQ'.
Now, as an example, in NFS:MW, I can run the game at ~9mil pixels (16:10 4k) with SSAA 2x. Only SSAA is offered, and if I try 4x, it is a horrible slideshow or crash. But performance at 2x is pretty good, and not capping my vram, so I want to find what more I can do.
AA is usually very intensive, like MSAA, SSAA, or TXAA, or light, like SMAA or FXAA. The latter are post AA, and done much differently, thus easier to run and carries little impact on vram (compared to the former). So, from your should be able to determine what your general level of AA capability is from your max DS res, and won't need to fiddle with the AA afterward. I've found that usually the best AA you can do at your max is the basis for the best IQ you can get, rather than lowering the res to manage higher AA levels - unless you are in situation 4. In that case, it is helpful to have a many different 'total pixel' DS reses that step down from your max, in order to notch in quickly whether or not you can in fact get, say, 8xMSAA without trading much actual res. (its a little murky becuase it is hard to quantify, as far as I know, how much vram the AA pass is taking).
If you're going beyond your max DS res, you have to run the game windowed. That part is covered in other posts. From there, rather than trying to jump the X and Y res up bits and see what you can get away with, work from total pixel increments. I went from 9 to 10mil. The quick way to get your res is to find a variable to plug into your aspect ratio, i.e. 16(x):10(x) to get your total res. Its probably apparent, but effectively you're looking for the square root of your total pixels dived by the product of your aspect ratio, i.e. 10mil/160 = 62,550, and the root of that is 250. Then 16(250) * 10(250) = 4000 x 2500. Thats my max res at 16:10. At 16:9 its 4224 x 2385.
You can push further by smaller total pixel increments, but I'd suggest minimum increments of 250,000, and even then the difference may be quite small. 1mil is a noticeable difference, both in IQ and performance.
So its also quickly apparent how this is helpful when, as I have been doing with Crysis 3, you're juggling different aspect ratios.
I definitely recommend trying out different ARs. It's cool stuff.
I hope that wasn't too much of stating the obvious, but I've found its refined my IQ a good bit, and made the process of finding a sweet spot for games much quicker and more exact.