Nah, it shouldn't. I even lowered my resolution, which reduced the VRAM usage greatly. Then increased that poolsize value so it was using almost all my VRAM (with a little bit of headroom left over). Still got stuttering in the same point as in that video @ 0:48s.now this shouldn't be happening, your 680 with (at least) 2 GB of VRAM should be plenty to handle almost the entire area of a chapter with only a couple of loading spots, I have a modest 660 Ti 3 GB and it uses 2200-2400 MB VRAM in Infinite, only stutters a bit while loading for a split second in a couple of pre-defined places throughout the chapters, rest being silky smooth... so that about stuttering happening every 50 meters is definitely something wrong in your end not the game itself, delete XEngine.ini and let the game create a default one, tweak again the poolsize value for your VRAM.
I only have 1GB available (5850+5970) but making all that extra VRAM available, it should be loading textures ahead as much as possible, but no matter what that poolsize value was, it always stuttered. Pretty obvious that the only read-ahead that's being used is triggered at certain points and isn't dictated by how much video RAM one has available.