One of the things I enjoyed doing after finishing the book was circling back and trying to imagine what happens if you strip the Count out entirely. I've always felt the three principle villains (and Caderousse, who I don't feel can rightly be called a villain so much as a co-conspirator) all end up failing to get what they were after anyway.
Caderousse is easy; he was already in ruin before the Count returned. He had the least impact on Dante and the Count really had the least impact on him; he mostly left Caderousse to his own fate and he made his own bad choices regarding the diamond.
M. de Villefort's wife was going to kill Valentine, and very likely Noirtier as well. He'd still lose a child and father, and without the Count supplying the poison she probably would have botched the murder. If that came to light, he'd likely lose his position (the very thing he conspired against Dantes to obtain) for not finding a murderess right under his nose (which is mentioned several times). It's been a year, but I also seem to recall her having tried to poison Valentine before with a different poison but it wasn't effective?
Fernand conspired against Dantes for the love of Mercedes, but he never really had it; she loved Dantes all her life. He'd still have his title and wealth since the account of Ali Pasha would be unlikely to come to light, but the thing he ruined Dantes for, he never obtained in the first place.
Danglars is trickier; he conspired against Dantes out of jealousy over his good fortune; being made captain of the Pharaon, his future prospects and his family. But he flees to Spain after framing Dantes, so he never gains the Pharaon. He settles in Paris as a wealthy baron but he's still an unsophisticated miser with a title everyone knows he bought. And in the end, his wife was cheating on him with Lucien and his daughter fled incognito, so he failed to obtain a family as well. He probably would have kept on as he was, but he never really obtained what he ruined Dantes for either.
You could argue the Count is the catalyst for their actions, but I think things still would have ended up badly for those individuals, which is why Dumas chose to end the book with "Wait and Hope;" God / karma / the universe usually balances things out in the end, maybe not as dramatically or as harsh but all the conspirators end up losing or never having what they were after. But then, we wouldn't have such an awesome book, would we? So while the book wants to be a revenge fantasy while eventually deciding "revenge is bad," it feels like a natural shift as the Count sees he's not the instrument of divine retribution he thought and that even his best laid plans had flaws (namely Maximilian and Valentine). I think that's why the book gets away with it, and I think Dumas spent enough time inside the head of the principle characters to show that despite what they did to Dantes they never really got what they wanted in the end.