The quality of the TV is infinitely more important than it's resolution. There are plenty of shitty 4k TVs. I think the contrast ratio, color, motion handling, and input latency of high end 4k tv's is definitely worth the upgrade. The high end TV's just all happen to be 4k now, but they are leaps and bounds better than high end TV's from 5 years ago even. 4k itself is nice, but the overall picture quality enhancements of high end sets, paired with good HDR make top of the line 4k TVs almost unbelievable to watch.
I bought a Sony 850D 4k hdr TV in 2016. I kept it for a month, but it sucked. I tried to like it, but it's an IPS panel with really shitty contrast. The picture looks good in bright scenes, but anything at night in dark scenes was bloom city, completely unacceptable for an (at the time) 1300 dollar TV.
I returned the Sony for a KS8000 from Samsung. Wonderful TV, I'd probably still use it to this day if it had not been stolen from my house by burglars. Jokes on them though, they didn't take the one connect box, which renders it useless. The only problems I had with it were a little bit of blooming in dark scenes, black bars in movies turning greyish in HDR content, and motion handling was pretty poor for low frame rate content. This could be compensated for with VERY LIGHT motion smoothing, but for 30 fps video games the motion smoothing jumped the input lag up to 100+ ms. As a result, you just kind of had to get used to 30 fps games being blurry/smeary.
I now have an LG B7 OLED. Easily the best TV I've ever owned. I watched Pacific Rim UHD on it last night, and HOLY FUCK. Just such a beautiful thing to experience. I've had it for 7 months now, and I'm still impressed with it every day. Putting on any HDR UHD movie just brings a big grin to my face every time. Games look amazing too, and supporting PC games at 120fps take it to a whole nother level.
Edit: to answer the question about comparing them to my 1080p TVs. I had a samsung from 2011, and a vizio from 2010. Even the Sony 850D smoked the ABSOLUTE SHIT out of both of them. I have a 1080p TV in the bedroom, and it's fine. But now that I'm used to the picture quality of modern high end displays, having an old 1080p set in my home theater would feel like I have a weird Fisher Price TV. Like the difference between the Wii U tablet and the Switch. I feel like people don't bring the actual build of new TVs up enough. All of my 1080p sets had these 2 inch thick bezels. Seems like a weird thing to complain about, but all of the 4K TV's I've owned have milimeter thick bezels. They just look like a floating screen. And hell, the OLED TV itself is about as thick as two CDS stacked on top of eachother.