That person sounds like a tool. Be the bigger person dude. If you own it, then they can't make you feel bad, and they end up looking like the bigger loser.
I used to get embarrassed about my interests too, but now once I start talking about it to people or putting it into perspective for them, they tend to actually seem slightly interested or at least stop taking digs.