There are some good definitions and examples pointed out here, the STALKER games being the classics. And bear in mind that the games mentioned have GOOD things about them, and people like them, it's just that they tend to have their own variety of idiosyncrasies.
In ARMA II with DayZ yesterday, I was in a building with 3 doors. I stood in front of one with the <> icon, scrolled the mouse wheel to bring up a menu (with such options as open door and close door), and tried to open the door...which apparently resulted in a different door 6 feet behind and/or to the side of me opening. I was reading a getting started guide and it was saying that sometimes one might simply give up trying to loot something because it was so difficult to get the cursor on it. I'm not sure whether the above is specific to the mod or an artifact of ARMA II itself, however.
Overall, it could consist of odd animations or odd/clunky control schemes, possibly combined with strange bugs of the "the game is good, but you'd better have 20 quick-saves" sort. Yes Bethesda is known for that sort of thing as well, though I would say Skyrim is their best yet in terms of a fairly polished user interface and a game that is easy to get into.