In practice they generally slowed things down significantly, for multiple reasons. The first is simply the extra calculations needed, but that's only a small part of the impact - turning on bump maps may also force you to use a more complex lighting model than if you weren't using them, which also slows things down (sometimes drastically). The final thing is N64 specific - the graphics processor is microcoded and the programmer can choose which set of ucode they load into it - depending on the features you need you may end up having to load ucode that's more capable, but also slower.
So it's more a case of not just being inherently slower, but also defeating certain optimizations you might otherwise be able to use to speed things up.