This is definitely something that needs to be taken on a case-by-case basis, I don't think either extreme is 100% correct, and even on individual games one solution isn't going to suit everybody. I wish Wii VC games gave you the option of turning filtering on or off... it's "always on" (TG16 games) or "always off" (everything else). Some TG16 games look awesome with the filtering, totally natural... some look like a blurry mess. Some SNES games look great untouched... and some I load up directly after playing a nice TG16 game and wish the same filter was applied. I find with most games I'll just stick with the original chunky pixel art, though.
I think what the problem is with many games is that the filtering makes it look unnatural, even though no filtering makes even the most beautiful pixel art look blocky and ugly.
3D games, I more often than not find that super-high resolutions and AA/AF make some PS1/N64 games look once again quite unnatural. I actually found this out years ago when I stopped buying console versions of games and buying the PC versions instead. While GTAIII on PS2 might have been a low-res chugging mess, GTAIII on PC at high resolutions, with AA/AF, at 60fps, made the game look less like another world and more like a video game. Suddenly you notice characters are made out of like 5 polygons and one big low-res texture. It was nice, but it sort of pulled you out of the ambiance.
One crime is having 3D characters rendered at super high resolutions running around a pre-rendered background originally drawn at a 320x240 resolution (PS1 FF games), or vice versa (Paper Mario 64). It just looks silly. Final Fantasy VII is a good example actually... when you're just walking around it looks terrible because your character and the NPCs are super sharp and clear, running around on a jagged mess. Then you go into battle, where the backgrounds become 3D, and it looks great.
On the other hand, sometimes it just looks superb. Sin & Punishment on N64 looks like someone puked on my TV when played on an actual N64... played at high resolution with filtering, it's incredible. R-Type Delta on PS1 at 1600x1200 with all the filtering turned on... oh my, that's the 3D side scroller of my dreams.