Post processing AA, which is an effect generally applied after a frame has been rendered. This is stuff like FXAA and TXAA. It looks at the rendered frame data and, using smoothing algorithms, attempts to clean up jagged edges on the image. Post processing AA has the smallest impact on performance.
MSAA/SSAA are AA implementations that are generally part of the rendering process itself. MSAA/SSAA are brute force methods of AA and, on average, can be very expensive. SSAA in particular is very expensive as it renders scenes at higher resolutions than what you're actually playing, and downsamples the data in an effort to clean up aliasing. MSAA is less expensive, but can still have a large impact on performance.
Quality is mixed across the board. Ideally the best form of AA simply in terms of image quality is SSAA. It's basically saying "let's render everything at fucking stupid resolutions so we have a lot more data, then shrink it down". Unsurprisingly, it's also expensive as fuck. MSAA is a "results are mixed" case. Often it works really well if at a cost, but the advent of modern rendering engines has caused a clash. MSAA doesn't always play nice with aliasing on shaders, and going into next gen we'll be seeing a lot more deferred rendering, which is very difficult to get working nicely with MSAA (see: Battlefield 3).
This is why a lot of engines are opting for post processing AA solutions, like FXAA, because it's generally inexpensive. Results are mixed, but a lot of people hate it, because the idea of applying AA as a post processing solution, after a scene has been rendered with jaggies intact, is a recipe for image blur as that's the laziest method. There's a lot of games out there where turning on FXAA is like smudging vaseline over the screen. The jaggies are gone, but the cost of sharp image quality is very high. But a lot of this comes down to the algorithms and methods used, and programmers are trying to find methods that implement both post-AA and render-AA together. TXAA High in Arkham Origins combines 8xMSAA with post processing for it's own thing. Metro: Last Light has a post-AA built into the engine that cannot be turned off, and looks great with SSAA sharpening the image. SMAA is also a pretty nice post AA that keeps the image sharp while trying to clean up jaggies.
Thanks for this, been sorta iffy on how AA worked for a while now.
I've been playing Batman:AO on my mid range PC with FXAA and I don't really mind it too much. Better than nothing if you can't do MSAA.