Having worked in home theater when HD started for several years, plus when the blu-ray/hd-dvd war started I used to check every single television that I'd hook up.
I honestly could never really tell a difference at 40" or below. Easily able to see difference at 46" or above.
Unless you plan to use television as second screen for computer or whatever, anything 42" and above, get 1080p.
With 4k, I've only seen first hand experience with 84" versions and can easily see the difference from 1080p so I don't know how 4k will play out at smaller sizes.