Oftentimes (but certainly not always), I find that the visual style is closely tied to the setting and theme of the game, so sure, those elements together influence whether I'll be interested in it.
It's not just that I've grown less interested in certain Japanese art styles, but it's also that often these games use certain common stereotypes, character motivations, settings and themes that I don't find as interesting, engaging or humorous in those particular combinations.
I'm also not a huge fan of how sci-fi and fantasy worlds are depicted in most western games (all media really), often drawing inspiration from the same few sources and using the same techniques (grossly generalizing here, most monsters are anthropomorphic demons with rocky skin and spiky carapace armor, most aliens are anthropomorphic lizard cat people with scaly skin, wielding assault rifles, and most space ships are ugly grey hulking factories with sharp angled corners), so that can also put me off.
On the other hand, there are times when "bad" graphics (less sophisticated or developed or very uneven art style, amateurish, cheap modeling and animations etc.) don't really bother me that much if there's something else appealing to me there, and I think that's mostly because the lack of expertise, talent or a developed style on the visual side can often really only mean the lack of experience or a troublesome development process (so it can also entail lousy writing, questionable gameplay mechanics etc.), so it doesn't have that additional baggage of, say, a "moe styled" game, and if the setting, themes or certain gameplay mechanics appeal to me, even better.
