Yes that is how averages work. The point is more that looking at averages can avoid or diminish responsibility, and in that case likely isn't the right metric.
If 80% of the problem is caused by 20% of the people, then average really isn't the best way to discuss the problem.