Everyone gets the patches, regardless of what system they're on - they're patching the base game code (and potentially other files, if they've changed the file layout to accommodate 4K assets), and just like any other patch, everyone gets those, unless they're staying offline and avoiding updates entirely.
What's optional are the 4K asset packs, which download automatically on an X, and on older systems only if you opt in via the option in Settings.
In Quantum Break's case, I'm guessing what happened is that in order to separate out the 1080p/4K versions of prerendered cutscenes/etc, they had to completely rip apart and re-organise the game's file structure, which due to the way patching works on Xbox (it's done on a per-file basis, with any file that's changed downloading the new version) means you're stuck basically re-downloading the entire game when you get the patch (luckily files are downloaded in-place, and you don't need additional space for the patch to download before installing).
That said, the patch for QB on a regular XB1 or One S without the 4K option checked should only be about 45-50GB or so, so I'd double-check you don't have the option enabled.
My One S QB install size is 47.5 GB, while my One X install size is 102 GB.
If you're not sharing a drive between the two systems, you should probably head to Settings -> System -> Storage -> Make more space -> Shrinkable games. That'll let you remove redundant data that isn't needed on the X (IE, files that are only necessary for the One S version, such as 1080p copies of prerendered cinematics). Quantum Break goes down to 83.6GB, so you're saving nearly 20GB, and there might be other games that have substantial savings as well (when I first upgraded, Rise of the Tomb Raider was another ~20GB, IIRC, and I had a few other games with more modest savings).
