Just saw this on Netflix. It left me really disturbed and thoroughly creeped out. I'll skip through the obviously great stuff - the acting, the dialogue, the cinematography, the music - all fantastic. What's bothering me about it is the moral structure of the film, and how it works. There is no modern, rational voice in this film, no modern interpretive device. Spoilers, of course.
The moral structure is very religious, old-school Christian. God is real, Satan is real. The wilderness is full of Satan's literal influence. The father is a heretic because he disagrees on points of interpretation of the Bible. His sin of pride in his beliefs causes him to drag his family out into the countryside, away from the protection of the plantation. The family's faith is weakened by the difficulties of life on their own, making them more vulnerable to Satan's havoc. The black goat is literally Satan, the rabbit is the witch's familiar. The children are literally influenced and corrupted by Satan, and he uses them, aided by the witch(es), to help tear the family apart. Caleb regains his faith and offers his life up to Christ, experiencing ecstasy as he leaves his body. In the end, Satan leaves alive the family member of most use to him - a nubile virgin - and induces her into his coven, along with all the other wicked women he's seduced.
I guess the way it works best is to show the thought processes of a people from a bygone era, frightened people in a wild country, who clung to their religion because they didn't have much else of an explanation for life's harshness and difficulties.