Because your brain sucks at it. Most of your brain is specialist hardware for doing one thing or another. It's fantastic at language and facial recognition but abysmal at mathematical operations. High level reasoning, planning, maths etc are not as computationally intensive as you might think, while getting something to stand upright or parse a sentence is much harder than you might think. Yet we are designed to do the latter, so it seems trivial to us, while we are not designed to do the former so it seems really hard or complicated. This is known as "Moravec's paradox".
Hmmm... the idea that the brain has 'specialist hardware' is a misnomer I feel. It characterizes the brain as having these prebuilt innate areas like a visual area, a language area, a face area, etc, etc.
The reality is that the neocortex is more like a cloud computing center. General purpose massively parallel computing network.
The reason that humans so consistently develop visual centers, language, centers, facial recognition areas, is because overall, the brain is wired up very consistently to its sensory modules and the various connections to the body (similar to how the make up of the body itself is very consistent from human to human). And that the environment that it interfaces with is very consistent as well (certainly has many more fine grained similarities than we'd intuitively feel - access to family, access to language, access to edges of high contrast and low contrast and colours, etc).
With that said, as remarkable as the brain is, it's not an idealized system for processing all types of information.
Especially information that requires massive operations and iteration - it's far better in processing patterns that can be heirachically catergorized.
As a result, it does things like facial recognition well enough... but iterating in mathematical operations poorly.
Additionally, mathematics tends to require abstraction that ends up at the higher levels of the heirachy... and for most people require way more stacks of the neural pyramid (and thus processing power) than is optimal for the problem.
But remarkably, even numbers and maths can be somehow catergorized for efficient sorting and traversal by this neural structure - as evidenced by savants and other high functioning autistics and people with conditions that nonetheless express a remarkable degree of intelligence in some field.
But despite this, I have a feeling that the brain is still not an efficient number crunching system; even in brains that have through whatever reason and circumstance been wired up for such a task... at least not for standard mathematical operations.
