Then it's not a fair question. You're not comparing 40k vs 20k, you're comparing 40k vs literally my family dying (like the hypothetical train diversion thing), that's fear mongering and not a valid argument.
The risk does not go up for my family because of self-driving cars. That's innate to the 40k vs 20k numbers.
So the proper question is: if your family was killed in an accident, what would be your reaction if it was a human driver vs AI? For me:
human driver - incredibly mad because it was probably preventable
AI - still mad, but supportive of self-driving improvements because it probably can be patched
The first would make me bitter and probably anti-driving, whereas the second would make me constructive and want to help people understand the truth of how it works. I'm still mad in both cases, but the second is more constructive.