Give it some time, the first few levels are a bit dull.
Later on it does get very hectic with little breathing space (you'll have some adventurers just entering the tower and the countdown for the next wave has already started) and it can be scary to watch as it all falls apart and you don't have time/resources to make it right (even little things like one adventurer taking too long on a floor allowing the others to skip it and it completely messing up your attack plan). Though I do hear even further on you might have to grind an easy level to unlock the level ups for your guys as you'll need the upgraded units.
Though I guess it doesn't get any faster, just more action packed so it doesn't appear as slow paced. It still really eats up time.
Plus you've got to love a tower defence game where you're actually defending a tower.
