Doesn't make you a bad person, though I know some who would think it does. Aesthetic appreciation is not morality.
I wouldn't invoke morality, but to me, someone for whom music is like oxygen, I think of people who say they don't like music in a slightly pitying way, a bit like somebody who's colourblind - functional but incomplete because they have a lack which means they're missing out on the full spectrum of human experience. Whatever style of music you're into - and I like just about all of them in differing degrees - anybody who likes any kind of music in a serious as opposed to shallow and superficial way will know (although they may struggle to put it into words) just how much it enriches your inner, emotional life - even what some might call spiritual life: other than hardline Islamism there's no religious tradition without its music.
You can go through life and exist fairly contentedly I should think in what I would call a fairly mediocre, humdrum, so-so way without ever having been in love, for example, but I would say that if you're thereby spared some of life's worst suffering, you also miss out on potentially its greatest joys. Likewise, to me, though further down the emotional scale, with music.
