But... he did? He was on the verge of taking over the entire wizarding community. He had the entire Ministry under control in the final book. Before that he murdered a number of people and continued to murder people at Hogwarts. I thought the final book drove the point home when the trio infiltrated the Ministry to find the Horcrux and it was under this fucked up Nazi like rule. Even in the film version David Yates was inspired by Brazil, as a couple of shots during the Ministry sequence were lifted from it (which I found to be very cool).
What I was trying to say was that despite this, the only people really putting any effort into taking Voldemort's power down was the Order. Had the Ministry, instead of being a concentration of pussies, collaborated with the Order could have yielded some much better results than they got. Instead they waited until the fucking Ministry was overtaken and Harry and Dumbledore pretty much had to handle that shit on their own.
With some help, obviously.