Till I was 22, I firmly believed in "musical segregation", i.e. not listening to any genre of music that isn't related to the genre I am into (which was and still is black and prog metal). However, I have since abolished this foolishness (a girl triggered this change, obviously...) And I now firmly believe that it is quite eye-opening and exciting to "listen outside the box". You'd be surprised. So, no, I wouldn't say that your taste is slipping. I would say that your taste might be breaking out of its confinements. I'd never believed to find a rap/hip-hop group that I'd be into. And then I discovered Eric B. and Rakim. Also, even though I am a metal dude, one of my favourite bands is Ghost. And they are more of a rock band (one of the bigger ironies is that many metal dudes don't like rock all that much). Also, quite a lot of classical composers are just brilliant. Alexander Borodin, Gabriel Fauré, Erik Satie, Gustav Holst, ...
Even artists like Depeche Mode or Simian Mobile Disco are cool.