Anyone who has ever heard a 128kbps mp3 side-by-side with a 320kbps (or really anything above 192kbps in my experience) version can tell you that bitrate definitely matters. The better audio equipment you play it through, the more noticeable it is.
It definitely becomes inaudible at a certain point, but back in my CD ripping days, I'd scoff at anything below 192kbps