I can see the difference, but it's not worth the investment imo.
That's kind of the issue here. People regularly talk about "investing" in 4K but that's not what you're investing in.
TV technology has improved dramatically in the last few years with consumer OLED TVs and huge improvements made with LCDs.
All of that technology is limited to 4K panels. Nobody is selling decent 1080p display any longer - they are cheap, bargain bin products with lower quality panels and chipsets.
When buying a new TV, it's not about investing in 4K, it's about investing in good panel technology. It just so happens that those panels are 4K. I think the improvements in panel quality and the introduction of HDR make a larger difference but 4K is a very nice addition.
Beyond the rendering resolution, I also feel that the benefit of 4K is the elimination of visible pixel grid at normal viewing distances. On a 1080p display, you can very easily see the pixel grid, I've found. On a 4K display, unless it's absolutely massive and you sit rather close, the grid effect is basically eliminated.
Basically, I don't think it's fair to say "it's not worth investing in 4K" because that's not what you're really investing in when buying a new TV today. That's just part of the package.