I've had better luck using it with applications that lack exclusive full screen mode, like PCSX2.
For whatever reason, regardless of process priority, emulation or video settings, the application starts dropping frames after about 5 minutes of playing any game. It's not in-game slowdown or performance issues - PCSX2 reports a solid 60 fps framerate, and uncapping the framerate launches it into 200+ fps. The only way to correct the issue is to pause the emulator, minimize the window, and then maximize and unpause.
Game Mode partially alleviates the issue when enabled. PCSX2 automatically "recovers" shortly after the issue starts, but it's still annoying to have rock-solid 60 FPS games drop half of its frames when nothing strenuous is happening in-game.
Otherwise, yeah, I'd leave that shit disabled.
It reserves those cores/threads regardless of any activity that may or may not be present on them, so you will always have less cores available to the game itself. From what i know, the number of threads reserved depends on the total amount of threads on the system, but it will always be a net loss of total number of cores/threads available to the game as some are reserved for background processes.
This isn't huge for older games which aren't as threaded, but newer titles, especially those on DX12 (like BF1) or Vulkan are usually heavily multi threaded when it comes to CPU usage, so game mode is straight taking away some of those available resources, hence lower performance.
Great post, and helps make sense of why DX12 and well-threaded DX11 games (especially on Nvidia hardware) suffer from Game Mode.