The limitation enforces the need to prioritize and make decisions on how you prefer to focus your mage. Compared to older Infinity games with which the system is adopted, Pillars cuts a lot of the chaff and tries less to have sever situational spells to rely. But yea, choosing between direct damage and the ability to terrify, weaken, corrode, or whatever can feel a bit awkward with limited slots.
As mentioned, you do get new grimoires from enemies, the idea being you have different "decks" to switch between(or another avenue to procure new spells for yourself). I found I never bothered swapping grimoires and instead just jammed my single book with whatever spells seemed most effective as more unlocked with progression. Most spells are all useful, some only a bit moreso than others, so 4 per category level is plenty to succeed really with a bit of diversity(have a mix of buffs, cripples, and damage and you will be good).
