I was going off a bit about this in the underrated 2015 games topic, but it makes me sad that I think the reason the reviews were so mixed is because Fatal Frame 5 is a Fatal Frame game. What I mean by that is, instead of modernizing itself, FF5 changes things to its own formula but doesn't really... "Modernize" itself in the direction of keeping up with industry trends and expectations. Almost all the complaints in negative reviews for the game are things that are true of other entries in the series as well. It's a sad realization, but I think the mainstream doesn't like old-school horror games anymore, even though they haven't existed in years really, and Fatal Frame never achieved the popularity of Resident Evil or Silent Hill, so it can't get away with, "Going/Sticking to the Roots," like those series can.
I think there's some mentality that all games need to be held to some gold standard and we can't go back to certain things, and while I somewhat understand I think games are big enough that we can have old-school games along with the new. I think FFV will find its audience, and maybe it is a niche, but I think many took the mixed reviews as to say this game is a bad FF game, when that's not true at all, it's a solid entry in the FF series, just critics apparently aren't that big on FF anymore it seems.
FFV is't the best entry in the series, nor the worst (though I very much enjoyed what I even consider the worst entry in the series), but it's a Fatal Frame game, that much is true, and I guess if you like FF or not is going to help/hurt your opinion on this game. All the Fatal Frame games are quite slow pace-wise, involve back-tracking and returning to areas, they are quite atmospheric (though I've never personally been too scared by them outside a few moments, but enjoy them and like their stories and find the combat fun), and they keep to a lot of old-school design philosophies. They have changed slowly over time, but are basically the only on-going horror series that hasn't "lost" itself, though apparently by not doing this people then criticize it for not changing. Can't please everyone, apparently.