I mean, at a certain point this kind of thinking becomes like the MPAA's math around thinking every person downloading a movie from a streaming service was a lost sale.
Yes, this would mean a massive expansion of translated audiobooks without the labor that traditionally would have gone into creating them.
But we don't have translations for the majority of audiobooks in the majority of languages because the costs of that labor has historically outweighed the benefits of a potential expanded audience in niche languages for the long tail of audiobooks.
Personally, I'd rather live in a world where there's broad accessibility to information for all people regardless of their native languages, rather than one in which humanity tears down its own tower of Babel to artificially preserve the status quo.