First, it's not really XP. It's your position in the REGIONAL leaderboard backed by MMR. Whether you move up or down is dictating by the the position of everyone around you.
You might be rank 50, at the 50th percentile and go to bed. Wake up and play a round in the morning, win, and still rank down if a WHOLE bunch of people moved above you overnight.
Second, your penalty IS based on average team MMR. The team's average levels are taken into account, and the points taken from the losers and given to the winners is based on that
Losers higher than their team's average level by logical extension must sacrifice more points than players lower than their average team levels. Winners higher than their average team levels will receive less points than players lower than the average team level.
It's unavoidable if you want the penalty to be fairly applied. If everyone on the losing team and winning team got the same bonus/penalty, people of a higher rank would be pushed into TOO HIGH a rank for defeating lesser teams and people of a lower rank would be pushed into TOO LOW rank for losing to better ranked teams.