There shouldn't be any shame but depending on the game you might be missing a lot by playing on the difficulty.
For example, quite a few action games go in automatic mode on the easiest difficulties: you simply mash random buttons and the game will do the combos for you - this kinda defeats half the purpose of playing the game to start with IMO. You are actually doing yourself a disservice by playing those games on the easiest settings. But hey, that's completely up to you. Some people prefer to play that way, otherwise the mode wouldn't even exist.
But I'm definitely a proponent of playing on higher difficulties, because a significant number of games just become more interesting on higher difficulties. A good example would be Crysis: the game becomes more realistic on the highest difficulty: enemies speak Korean instead of English, you need to physically pick up weapons and ammo, you can't drive and fire a jeep's machine gun at the same time, you have less indications on the HUD saying stuff like "Warning! There's an enemy/grenade over there" etc... This all adds to the experience and makes you more attentive to what's going on around you.
Adding to that, I noticed that when playing some games on their easier difficulties (that sometimes include normal mode actually), some people will just brute force their way through (because they're not punished for it) and will miss out on some really cool mechanics because of it.
So yeah, there are definitely benefits to avoiding playing things on easy but at the end of the day just choose the mode you'll have the most fun with. As long as you don't go and tell people "that game sucks it's was too easy when I played it on easy mode" there shouldn't be any issues.