Milo is not being prevented by the government from speaking. Universities absolutely have the right to pick and choose who to they allow to speak, and people can respond in turn and try to apply pressure with their own speech. Also, there's no basis in feelings, it is based on factual observation. Milo planned to dox people on stage, hundreds of people. Waiting to see if it happens again after it has already happened, and not acting unless he is found guilty of a crime.
Finally, I am not advocating for a criminal having their speech limited. However, I am advocating that if someone exploits free speech to commit crimes, that no longer should be a right. Free speech is only a right until you start using it to reduce the rights of others. Just like if you used a firearm to commit a crime, your second amendment rights could reasonably be restricted.