• Hey, guest user. Hope you're enjoying NeoGAF! Have you considered registering for an account? Come join us and add your take to the daily discourse.
  • The Politics forum has been nuked. Please do not bring political discussion to the rest of the site, or you will be removed. Thanks.

Checkerboard rendering, standard rendering, and upscaling

Liabe Brave

Member
Jan 23, 2007
4,509
3
1,140
Lawrenceville, GA
There's a lot of confusion about the new technique of checkerboard rendering (CBR). The most pervasive is that it's a form of upscaling, taking a set of rendered pixels and doubling them to create the final image. While this idea is inspired by a step in the real process, it's a misunderstanding and doesn't capture the entire method. This leads to bad estimates of the costs, benefits, and possible applications of CBR.

I'll try to describe what CBR actually does, and compare it to standard rendering and upscaling. The difficulty with this plan is that I have no training as a computer scientist! My knowledge derives completely from reading on the web, and from discussion on GAF. Therefore, please take the following explanation with a huge grain of salt. I know there are very knowledgeable folks here, so I'd be happy to make revisions wherever I've made mistakes. I expect that there are mistakes.



The typical basis of computer graphics for games is polygonal models or meshes. These describe the objects in a scene, with smooth curves approximated by flat shapes joined together. As a base example we'll work with throughout, imagine a piece of debris being tossed by an explosion. Two frames in sequence might show something like this:


Time is advancing from left to right

These models are mathematical, giving them immense precision. But the screens they're displayed on have a limited number of pixels. Details in the polygonal scene will thus not exactly match this grid. Some pixels may contain only a sliver of a polygon; others will overlap multiple polygons. Since each pixel can only be one solid block of color, the renderer must decide what the best value is. This process of turning a 3D scene into a 2D grid of pixels is called rasterization.


The details inside each pixel can't be displayed

At the most basic level, each pixel's value is determined by checking which polygon covers the center of the pixel. Polys closer to the camera take precedence. Here's three scenarios, showing the polygon positions in the top row, and the final pixel value below them.



In the first case, no polygon covers the center, so the pixel is "see through", and the final value comes from the skybox behind everything. In the second case, the pixel is set to green. Note that there is no blending of pixel color--not even a hint of brown in the green. In the third case, both polys cover the center but the pixel is set to the color of the nearest one. Note that the fact that the green polygon covers more of the area is ignored.

Repeated for all pixels, the result for our example might end up looking something like this:



This is a start, but there are more influences to a scene's appearance than the raw color of the objects in it. Therefore, rasterization continues with many more iterative steps of different kinds. It's an extremely complex process, and I couldn't fit all the details even if I knew them. But here's a vastly simplified diagram of some stages. Each one often includes multiple substeps.


The reason for skipped stage numbers will become apparent.

1. Textured. Artists create flat drawings of detail that are wrapped around the models. This stage gives the same result shown before.
Our example shows the debris above some rocks, in front of a blue skybox.

2. Lit. Multiple sources of light may shadow or color the scene, adjusting pixel values again.
Our example has orange light from the offscreen explosion, also casting a glow on the debris.

3. Shaded. "Shaders" is a catchall term for methods that adjust the results from previous stages. They can do a great many different things, increasing realism or stylizing the image.
Our example adds motion blur to the flying debris.

10. Particles. Very small elements may skip the computationally expensive prior stages, and be added at the end.
Our example has flying sparks from the explosion.

12. UI layer. The user interface is almost always added last, as a 2D overlay on the image.
Our example shows the corner of a waypoint marker.



The calculations needed to determine every final pixel value are complex and time-consuming. CBR was invented as a way to cut some of the cost, without reducing the number of pixels in the final image.



Each frame begins with two separate rendering paths. (Very often CBR is conceived as having these occur sequentially, but I don't know of any technical reason they can't be done in parallel up to a point.)

Stages 1 to 3 are identical to standard rendering, but done only for every other pixel. In each successive frame, which pixels are rendered is switched back and forth. The result of this path is sent to Stage 7, and also forms the basis for the second path during the next frame.

4. ID buffer. Each object or polygon in the prior frame has a unique identifier (only some are shown).

5. Vectors. The ID of the polygon(s) associated with each pixel is tracked as it moves, creating a vector. Everything that moves is tracked, but I've shown only a few vectors to make things less cluttered. (Motion may also be tracked back two frames instead of one.)

6. Projected. The renderer uses the tracked origin and motion to move the old pixels into new positions.

7. Merged. The results from each path, which are opposite sets of pixels, are combined to make a whole frame. Note that the same number of pixels has been rasterized as in standard rendering.

8. Confidence. All the pixels from the second path have a confidence level, indicating likelihood of error due to erratic or extreme motion. (Only some confidence levels are shown.)

9. Adjusted. Pixels with low confidence levels are compared to their immediate neighbors, and their values adjusted by compromise with those adjacent pixels. This is the only step of CBR that uses a method like upscaling. Only some pixels are affected, and fewer as motion gets more predictable or takes up less of the screen. At 100% confidence, CBR becomes literally identical to standard rendering.
Our example outlines pixels that were adjusted due to low confidence.

Stages 10 and 12 are the same as with standard rendering.



CBR is one way of making rendering cheaper, but it's very complex and was only developed within the past few years. Previously, a common way to reduce render cost without reducing scene complexity was simply to render fewer pixels. Of course, the smaller frame then has to be blown back up to the correct size for the display. This is upscaling.



Stages 1 to 3, and Stage 10 all take place exactly as with standard rendering, simply at a lower resolution.
Our example assumes a 9:10 ratio, similar to a 900p:1080p upscale, or a 1800p:2160p one.

11. Upscaling. The frame is enlarged, so each original pixel now sits over portions of multiple pixels. A compromise method blends them and selects final values.
Our example shows the precise results, not an approximation or "what if" scenario. The method used is bilinear, which is due to extremely low computing cost on modern hardware.

Stage 12 is the same as with the other methods.



Here are the final frames of each method side-by-side.



As you can see, both checkerboard and upscaling deviate from the "real" results of standard rendering. But despite the fact that they cost about the same in rendering time, it's clear checkerboarding gives much more accurate results. This may be more obvious in an animated comparison.



Hopefully this helps people understand what CBR does. I'll do my best to answer questions, and make alterations to fix my errors and improve clarity.

Thanks for sticking with this very long and complex post!
 

Liabe Brave

Member
Jan 23, 2007
4,509
3
1,140
Lawrenceville, GA
So Standard/Native > Checkerboarding > Upscaling if I'm looking at the pictures right?
Yes. CBR is a tradeoff, giving up some accuracy in exchange for quicker rendering. But it's not just better than upscaling in absolute terms, but also by efficacy. That is, for the same reduction in render costs, you get much better results.
 

GearFourth

Junior Member
Feb 20, 2016
7,029
1
340
So Standard/Native > Checkerboarding >>>>>>>>>> Upscaling
Ftfy. When I compared native 4k to checkboarded from 1800p to 2160p or 1400 and up both looked really identical to me to native from normal viewing distances it's impressive. Traditional upscaling is nowhere close to checkboarded results.
Anyways great thread OP
 

rokkerkory

Member
Sep 2, 2013
16,509
617
490
Do you guys think CBR will be prevelant in the future? Esp as we get more powerful h/w? I don't see displays going to 8k anytime soon. Mass market wise that is.
 

desmax

Member
May 6, 2014
2,555
0
0
Wow, that's very well explained; thanks.

I'm hoping nVidia or Amd implements CBR based scaling. I know you can scale to any resolution you want on PC, but gaming in 4k can be a bit too expensive to the GPU, let alone 8k in the future.

Wouldn't have a problem using it if needed
 

Liabe Brave

Member
Jan 23, 2007
4,509
3
1,140
Lawrenceville, GA
Do you guys think CBR will be prevelant in the future? Esp as we get more powerful h/w?
The ID buffer that improves the accuracy of CBR motion projection can be done in software. However, there appears to be native hardware support for it in PS4 Pro. While Pro is using an updated Polaris design, Cerny has said there are higher features included. If this is one of them, then AMD's upcoming Vega cards may also have hardware support.

I'm hoping nVidia or Amd implements CBR based scaling. I know you can scale to any resolution you want on PC, but gaming in 4k can be a bit too expensive to the GPU, let alone 8k in the future.
I get what you mean, but part of why I made this thread was to clear up what CBR is. And it isn't any kind of scaling, at all. You don't "use CBR to scale" or have "CBR based scaling". CBR is, as the name says, a type of rendering. It has a limited fallback in cases of error which uses the same sort of method as scaling. But the technique as a whole is very, very different.

I don't mean to harp on your word usage specifically. Even very knowledgeable folks like Digital Foundry have been sloppy with their terminology when it comes to CBR.
 

desmax

Member
May 6, 2014
2,555
0
0
I get what you mean, but part of why I made this thread was to clear up what CBR is. And it isn't any kind of scaling, at all. You don't "use CBR to scale" or have "CBR based scaling". CBR is, as the name says, a type of rendering. It has a limited fallback in cases of error which uses the same sort of method as scaling. But the technique as a whole is very, very different.

I don't mean to harp on your word usage specifically. Even very knowledgeable folks like Digital Foundry have been sloppy with their terminology when it comes to CBR.

Sorry, I just wanted to put it in a way that would be easy to understand.
Granted, I did used to think of CBR as a scaling method, but your post made me understand it in more detail.
 

Alo81

Low Poly Gynecologist
Jun 24, 2012
13,139
1
710
steamcommunity.com
It would be interesting if the GPU manufacturers were able to do something similar to DSR and implement a more universal checkerboard rendering style into games.
 

McSpidey

Member
Feb 25, 2005
1,063
0
0
Australia
I'm all aboard on any effort to clarify for the masses that checkerboard is a rendering technique, not an upscaling method.
 

HTupolev

Member
Aug 13, 2012
5,331
1
0
1. Textured. Artists create flat drawings of detail that are wrapped around the models. Each pixel becomes colored based on those images (using the "majority vote" or "compromise" methods mentioned above).
Our example shows the debris above some rocks, in front of a blue skybox.

2. Shaded. Materials have properties other than their base color: reflectivity, smoothness, and so on. Shaders capture these traits (and many others), and adjust pixel values.
Our example makes the metal debris shiny, and the rocks rough.

3. Lit. Multiple sources of light may shadow or color the scene, adjusting pixel values again.
Our example has orange light from the offscreen explosion, also casting a glow on the debris.
I think I would have left out #2, for both simplicity and accuracy. "Shading" is a pretty broad term, and when developers use it to specifically refer to a rendering stage, it typically comes after lighting to finalize results. Data that would define objects as being shiny or whatever would typically be part of the "texturing" process, more or less.

Good and well-presented general overview, though.
 

Liabe Brave

Member
Jan 23, 2007
4,509
3
1,140
Lawrenceville, GA
I think I would have left out #2, for both simplicity and accuracy. "Shading" is a pretty broad term, and when developers use it to specifically refer to a rendering stage, it typically comes after lighting to finalize results. Data that would define objects as being shiny or whatever would typically be part of the "texturing" process, more or less.
Okay. If it wouldn't be trouble, could you present a little more detail about what shading is actually used for? My understanding was that PBM are driven by shaders, and of course there's the term "fur shader". None of that seems like it would make sense to apply after lighting. I am very likely misunderstanding things. Perhaps you can clear up my confusion.
 

Inuhanyou

Believes Dragon Quest is a franchise managed by Sony
Jul 26, 2014
26,221
2
0
America
I've heard that 1800p checkerboard is actually lower accuracy than 1440p native but also takes less rendering resources...

Something about quadrupling 900p frames in a checkerboard pattern and getting that result..its very confusing.
 

Liabe Brave

Member
Jan 23, 2007
4,509
3
1,140
Lawrenceville, GA
I've heard that 1800p checkerboard is actually lower accuracy than 1440p native but also takes less rendering resources...

Something about quadrupling 900p frames in a checkerboard pattern and getting that result..its very confusing.
There's basically no way on earth that 1440p native upscaled to 1800p is more accurate to 1800p than 1800c is. Some people might prefer the soft look, but even the very best upscale blurs everything on screen.

As for the second part, that is confusing but doesn't sound very much like CBR. The combination of four frames sounds a bit like "geometry rendering", where 4xMSAA samples are used to resolve edges at the higher size, but leaving textures as upscaled. One of the modes of Bound does this, for instance, since it doesn't have highly detailed surfaces that'd look better with CBR.
 

HTupolev

Member
Aug 13, 2012
5,331
1
0
Okay. If it wouldn't be trouble, could you present a little more detail about what shading is actually used for?
An example usage, in the context of a lighting pipeline, would be in the second geometry pass in a pre-pass deferred system.

That approach is to allow the use of extremely slim g-buffers (i.e. because you're trying to make the game play nicely with the Xbox 360's eDRAM size), and such g-buffers might not even include diffuse/albedo color. So, when the deferred lights are being drawn, the results literally don't take into account the color of the objects, just things like normal maps and roughness. So, in a second pass of the scene geometry, you'd provide the diffuse/albedo texture, to allow the final shade of the surface to be calculated from the accumulated lighting data. That could be considered part of shading.

My understanding was that PBM are driven by shaders, and of course there's the term "fur shader". None of that seems like it would make sense to apply after lighting. I am very likely misunderstanding things. Perhaps you can clear up my confusion.
"Shaders" in general are extremely broad. Vertex shaders can be used to animate geometry, a post-process shader might apply bloom to a scene. You might even run particle physics as "a shader." These days, a "shader" is basically any task that gets run on the programmable ALUs in a GPU.
 

Tripolygon

Member
May 6, 2012
4,407
6,394
1,160
NYC
Brilliant explanation, for months people kept arguing over what it was prior to pro launch despite other efforts to explain the process.
 

Liabe Brave

Member
Jan 23, 2007
4,509
3
1,140
Lawrenceville, GA
An example usage, in the context of a lighting pipeline, would be in the second geometry pass in a pre-pass deferred system.
Okay, that makes sense. I tried my best to make the "standard rendering" flow as schematic as possible, so that it could apply to both forward and deferred renderers. It seems that might be a simplification that just can't be reconciled.

"Shaders" in general are extremely broad. Vertex shaders can be used to animate geometry, a post-process shader might apply bloom to a scene. You might even run particle physics as "a shader." These days, a "shader" is basically any task that gets run on the programmable ALUs in a GPU.
This also clears up a lot. Tomorrow I'll make some revisions to the OP to hopefully tighten stuff up.
 

Tallshortman

Member
Jul 31, 2008
6,446
1
770
Thanks OP, it's readily apparent that checkerboarding is pretty impressive. Looking at several games in person, it looks fantastic and worlds better than 1080p upscaled to 4k. It's very close to looking native.
 

pottuvoi

Banned
May 17, 2010
3,821
34
780
It would be interesting if the GPU manufacturers were able to do something similar to DSR and implement a more universal checkerboard rendering style into games.
In a sense the nvidia MFAA is similar as it uses 2 frames to double MSAA sample amount.
http://www.anandtech.com/show/8526/nvidia-geforce-gtx-980-review/6
With different sample patterns and forcing of shading of subsamples it should be quite close to CB.

Do not really like vendor specific and/or limited rendering types.
It's usually better to expose programmability and give developers access to all parts of hardware, so they can create method that fits their game.


Checkerboarding is just one of many methods of reducing cost of rendering and increasing overall quality.
We have seen some in Ps3 era and now with better programmability and lot of research, there will be more varied attempts and increase in quality.

Perhaps best method currently is the one seen in Ratchet and Clank.
 
Jan 27, 2011
7,638
25
660
Nevada, USA
So anytime somebody says checkerboard rendering is an upscaling method they don't know what they are talking about and I should direct them to this thread.

Gotcha.
 

pottuvoi

Banned
May 17, 2010
3,821
34
780
So they can eventually get rid of the artifacts when in motion?
The method is pretty much fully programmable, you can choose how you render/shade things and how you combine them.
So, yes it should be possible. (Perhaps trough full size idbuffer to get perfect edges on geometry.)

As I said before we will see many different approaches on the problem and I doubt CBR is the one that will be the best for the cost/quality.
 

GHG

Member
Nov 9, 2006
25,112
33,458
1,845
Love the OP. Thanks for taking the time to make it.
 
Well done, OP!

So they can eventually get rid of the artifacts when in motion?
I have not implemented CBR myself, yet, but the method seems to be very clever
considering fast motion. For, those pixels moving very fast (like, velocity
too high for the given timestep) will just be upscaled instead of trying to
fix the back integration. Hence, there will be a resolution drop for those
pixels. Given a racing game, the outer pixels of the screen should look a lil
more blurry than, which doesn't seem to be a big issue because you focus at the
middle where the pixel won't move that fast, hence, having a higher resolution
given CBR. Might be interesting to change kernel size based on the velocity of
the pixels, such that the blur goes down continuously while approaching the
center (given it's a racing game). Don't know if that makes any sense. But I
think this could give some interesting 2d blur, but may look crap considering
the game to be 3d. Hmm.
 

Rodelero

Member
Dec 13, 2009
1,475
0
0
Absolutely great post - a good few full on gaming websites could do with featuring this in my opinion. Its impressive that someone who clearly isn't entirely comfortable with the graphics pipeline has produced something so informative. All the same, for the sake of clarity it would be good to clean some of this up.

Here are the issues that jump out at me:

The point made earlier about the shader section is valid. It seems odd to separate texturing, shading and lighting when they are really all part of the same shading process. The pixel shading is in charge of colouring those pixels, typically utilising texturing and lighting calculations. No need to go into too much detail, this stuff can be far more complex, but it's not really the focus of your piece. Given that you're already simplifying the meshes section so much, it seems fine to combine the issue of pixel colouring into one part.

I'd probably remove the part about particles - it seems a little redundant. I think it suffices to have that separate UI section at the end to illustrate that the checkerboard or scaling processes arent necessarily applied to everything on screen.

In regards to rasterisation, the majority vote comment is incorrect. In the most basic scheme, a triangle will contribute a pixel if the triangle contains the middle of the pixel. Whether that ends up being the pixel you see most typically depends on whether another triangle contributes a pixel which is nearer the camera, in which case that will be used instead.
 

KOCMOHABT

Member
Nov 14, 2013
519
0
280
kosmonautblog.wordpress.com
Yes. CBR is a tradeoff, giving up some accuracy in exchange for quicker rendering. But it's not just better than upscaling in absolute terms, but also by efficacy. That is, for the same reduction in render costs, you get much better results.

not exactly the same reduction in rendering cost, since you have to check against the old buffer by reading the velocity buffer and then probably sample the neighborhood to reject further false pixels, too. The good thing is that basically all engines output a velocity buffer anyways, so that part comes for "free".


Not very expensive still, but a lot more expensive than simple bilinear upsampling, which only uses one texture lookup (bilinear is free on GCN) and is more cache friendly.

If the way OP describes it is correct then it's basically temporal supersampling, but instead of oversampling a single pixel it just samples the total resolution over 2 frames.

It's interesting then how AA is handled in these games, it's probably some form of SMAA.

I think TAA on top of checkerboard might further increase artifacts, I wonder how the devs go about that, especially since there is no correct per-pixel velocity map available. Would love to see how a naive TAA slapped on top of the upsampled image works. Preferably with continuous integration.

Either way it's interesting to find out how much this affects cached results and temporal integration for other effects. I guess for most of the stuff that uses temporary reprojection like ambient occlusion and dither effects for volumetrics etc. it's no problem since these effects work at low resolutions anyways. For fullscreen temporal reprojections the last frame got a bit more unreliable however.
 

dynamitejim

Member
Jun 6, 2004
1,748
0
0
38
This seems to be brought up mainly for 4K, why not use it for 1080p or 1440p? Is simple scaling better at lower resolutions?
 

Rodelero

Member
Dec 13, 2009
1,475
0
0
This seems to be brought up mainly for 4K, why not use it for 1080p or 1440p? Is simple scaling better at lower resolutions?

It's more that it's a relatively new idea than anything. Killzone Shadowfall and Rainbow Six Siege both use it, I believe.
 

Soren01

Member
Jan 26, 2013
978
77
615
Brazil
Of course. It was already used in multiple shipped PC games.

I think we've already seen this in some games. It makes sense since you are trying to extrapolate even more data from each rendered sample.

Aside WD2, which other games use this technique on PC?
 

King_Moc

Banned
Sep 19, 2010
11,648
8
810
England
Good explanation op.

As anyone with a Pro and a 4K TV should be able to attest, the results are way better than with upscaling. It's really impressive tech.
 
Feb 24, 2008
9,648
1
920
So Standard/Native > Checkerboarding > Upscaling if I'm looking at the pictures right?

Yes. You can consider checkerboarding as a smart, algorithm driven upscaling, so of course it will be normal than 'raw' upscaling.


That said, I only tried checkerboarding in QB and man, it really cuts back visual quality, so consider the example image with a few pixels from OP as a visual help to explain how it works.
 

Bamboo

Member
Sep 11, 2015
612
0
0
Wow, this is great stuff, OP. I don't know much about the processes myself, but I'm always eager to read about it. Especially if it's explained in a way that I understand. Thanks!
 

kanuuna

Member
May 25, 2013
2,079
0
475
Nice illustrations. Gives a good view of the finer details lost with checkerboard rendering.
 

ElFly

Member
Sep 3, 2006
16,567
0
0
that's interesting and a good way of presenting it but I have a question:

what kind of artifacts does this create? because the projection process seems ripe for errors even in some simple cases

the first that comes to mind is panning/rotating the camera in some way. say you press the joystick to the side, your mans is starting to look to the right, and pixels you have no idea what they contain start to move from outside the screen

so it seems to me every time you turn around, there are a bunch of pixels on the border of the screen that necessarily have 0% confidence and just get upscaled? does this look weird in practice? a small blurring of the border every time you look around?
 

LukasTaves

Member
Dec 14, 2011
9,869
3
715
Yes. CBR is a tradeoff, giving up some accuracy in exchange for quicker rendering. But it's not just better than upscaling in absolute terms, but also by efficacy. That is, for the same reduction in render costs, you get much better results.
Well, it's usually not at the same cost, reconstruction techniques tend to be significantly more expensive than just running the game at a lower res.

The pro however, has dedicated hardware for keeping track of primitives which might be the heavier part of the process, so in that case it might be real close indeed.
 

BradleyLove

Member
Oct 18, 2008
1,417
7
780
Thanks OP, very informative. Any chance you can add something in to explain what resolutions are derivable using CBR? For example, I believe you can't native 1440 and then checkerboard to 4K.
 

Durante

Member
Oct 1, 2006
48,836
1
0
peter.metaclassofnil.com
Thanks OP, very informative. Any chance you can add something in to explain what resolutions are derivable using CBR? For example, I believe you can't native 1440 and then checkerboard to 4K.
Basic checkerboarding as discussed here always renders exactly half as many pixels as the target resolution. This makes it inconvenient to express using the "p" notation, since there's really no equivalent lower resolution which is easily derivable or has a convenient pixel number.

Personally, I've used "c" notation for that before, as in 2160c = 4k checkerboarded.

When games want/need to render even fewer pixels, many of them seem to do checkerboarding and then traditional upscaling on top of that.
E.g. rendering at 1800c and then upscaling to 2160p.
(which is actually shading roughly as many samples as 1273p)
 

riflen

Member
Feb 12, 2013
2,402
0
0
that's interesting and a good way of presenting it but I have a question:

what kind of artifacts does this create? because the projection process seems ripe for errors even in some simple cases

the first that comes to mind is panning/rotating the camera in some way. say you press the joystick to the side, your mans is starting to look to the right, and pixels you have no idea what they contain start to move from outside the screen

so it seems to me every time you turn around, there are a bunch of pixels on the border of the screen that necessarily have 0% confidence and just get upscaled? does this look weird in practice? a small blurring of the border every time you look around?

This technique will produce artefacts. You can see an example here on Digital Foundry's recent Knack video.
One reason why this entire technique is reasonable is that almost everyone using a PS4 Pro is going to be connecting it to a 60 Hz sample and hold display and be seated several feet away. The motion blur inherent in the display will do quite a good job of disguising these artefacts. The motion resolution is low enough anyway that you can't casually observe them.