I just don't understand why people are attached to it
Because words aren't racist, people and opinions and sentiments expressed by them are.
When the term ricing is used for so long and 95%+ of people don't know where it came from and have zero negative connotations associated with it, your argument (from their perspective) sounds like this:
"Don't use the term, by its obscure origin you didin't even know about it is racist"
"I've been saying it for years without any racism intended nor perceived"
"bro it's been you are being a fucking racist for years"
And then you are surprised by the negative reaction...