I think at some point you have to just stop feeling bad for being relatively rich in the world. You don't have to be a jerk about it like O'Leary, but really you do what you can and call it a day.
Also, I think there is a discussion about foreign aid in here and it being useless, but I would urge you to read the 2014 Gates Annual Letter:
http://annualletter.gatesfoundation.org/