That's a thought-terminating cliche that people often use to dismiss legitimate criticism.
In this particular instance I think you've made a good case that broadening the definition is a good thing, but I really hate the implication no use of language is ever wrong, but rather just "evolution", which is implied to be least be a neutral process if not actively beneficial.
I've seen people defend literal typos as "language evolution" and get massively upvoted while anyone who dares to disagree is mocked. A typo isn't language evolution at all unless it becomes popular. Otherwise, to continue the biology metaphor, it's just a language mutation, and like biological mutations, typos are harmful to communication far more often than they're helpful.
Another example a bad use of language is how words and phrases are co-opted for political purposes. "Woke" is an obvious recent example. "Welfare" is an older much much more egregious example, where just the mere spelling of the word makes the original meaning clear, and that meaning is unequivocally positive, yet most people think "welfare" means government assistance to poor people. Or take "pulling yourself up by your bootstraps", which went from an example of something that's literally impossible to something people are unironically told they should be doing. This sort of thing is language evolution, but it's not neutral. It's done with an political agenda. It impoverishes our language our language and stifles honest communication.