All of this sounds made up. After the first time it happened, people would make sure that fucker had money the next time they even thought about inviting that person to lunch.
If it happened again? You're no longer allowed to accompany us.
Meh, I don't mind paying for someone's lunch once in a while if they don't have the money. It's the continued occurrence of it and the lack of any payback that makes this wrong, along with OP's total passiveness.