Native speaker here, never had any issues with this or any other common homophone in English. I mean, yes, spelling mistakes did occur, and we now have autocorrect to further screw things up. But I never “learned the rules” and still got everything consistently correct from a rather early age.
“Learning by ear” is just another excuse for laziness and/or ignorance. Pick up books, read extensively, and like anyone else not marinating in cultivated ignorance, you too can utilize the language effectively and correctly.