If the only metric you care about is future votes for the Democratic Party, then sure, let it ride, let him do incalculable damage to the people, the country, our global standing, hell, to the office he holds.
But for most of us, who aren't ourselves politicians, that's not the metric we're interested in. We want to mitigate harm as much as possible. To that end, even a somewhat more effective Pence would be better than Trump, because he's not going to wantonly smash things. The damage he can do is more likely to be the sort that we can repair later. He's not going to, like, nuke North Korea because Kim Jong Un said something mean about him or whatever.
More than that, there's an empirical question here: has Trump committed impeachable offenses? If the answer is yes, he should be impeached. Letting him get away with it sets a horrifying precedent for the future.
Edit:
Also worth noting, this has nothing to do with "hating" Trump, except to the extent that hating him is correlated to how absolutely unsuited he is to his office.