Not having “a heart” or “a burden” for something doesn’t release me from Scriptural commands about it, and might be an indication of how my heart needs to change. If what delights God doesn’t delight me, it’s a sad commentary on me, not a “get out of obedience free” card. I can claim to have “a passion” for another kind of ministry or Christian activity; that doesn’t mean I get to ignore the ones that are harder for me, even though I can focus on areas in which I’m equipped.
I don’t need to feel called to love widows and orphans. I can read.
“I don’t really have a heart for the poor,” is an admission of depravity, and ought to be a prayer request, not an excuse for a greedy, insular lifestyle.