Empathy is not something you feel on command. It’s only something that you feel towards someone of value to you, in some context. You don’t feel empathy for their sake. You feel empathy as a natural and logical outgrowth of the fact that you value this person — and, therefore, you want to understand their point-of-view. If you don’t experience empathy for those you care about, then, in essence, you don’t sufficiently care about those you care about. Right there: the contradiction is your own. It’s not that you’re “selfish.” It’s really quite the opposite; you’re not paying enough attention to those whom you supposedly value and want in your life.