I am someone still somewhat bothered by ethnicity-swapping (though not really for any of the reasons you described), but here's an annecdote:
When I first started engaging with the Hannibal franchise, I started with the Mads Mikelson TV series.
The character of Jack was played by Lawrence Fishburne.
Then, I watched the old movies, and shocker - Jack is a white guy.
Yet, I didn't care that Jack was black in the reboot. The only conclusion I could draw was that it didn't annoy me because I had always known Jack as black.
Now, I could be totally wrong about this, but I think a lot of people get bent out of shape because it's distracting above all else.
I couldn't care less about Jack being black or white, he's a side character in a movie I'll watch once in my life. Yet, I was thinking about race-swapping in the middle of the movie.