Lmao no way. That can't be it
That'd be sooooooo out of character if Kazuma just disowned Haruka over that.
I think you're missing the point.
Kazuma let Haruka go and be an idol, because that's the kind of person he is, right? Then those guys are questioning the parenting skills of girls who become idols. Kazuma bristles because they're indirectly criticising him.
At least, that's how I interpreted what Gacha-pin said.