1000% this. I live in the bible belt and am a big burly bearded bastard so people "quiet part out loud" at me with supersonic speed (1). I'd almost make a mortgage payment if I had a buck for every time someone said some ridiculous shit then I got in trouble for "getting political" aka politely and calmly engaging with the statement just made directly to me.
I didn't make it political. The person saying trans folk should "wear the right clothing" made it political. I believe the word you're looking for is "uncomfortable," and if you don't want it to get uncomfortable maybe tell HIM not to get political. If he says it, I have a right to respond - and silencing my speech but not his is an explicit endorsement of his speech.
(1) It has literally happened in like 5 sentences or less between even me and a stranger multiple times. "Hey what's up" "nothing much started a new job" "cool, I haven't worked in a bit but I worked at target for a bit" "why? Target funds ANTIFA TERRORISTS"