I use my 50" Plasma for most PC gaming, though if I'm going to play long stretches of SWTOR, I generally pull up to my monitor that's sitting nearby. That's the only game I've seen image retention/burn in start to become a problem.
Otherwise the only drawback I've discovered is console games look like ass when I play them now. And I'm not exaggerating...the PC IQ has me totally spoiled to the point of almost non-enjoyment when I boot up a console.
Yep, IR would be a big concern for me. Can easily imagine the wife just leaving it on the desktop or something one day and coming home to a burned in clock/date.