A lost phone is a much bigger issue to me. A lost card is trivial - call my bank, they cancel the old one, issue a new one, it's here in 2 days.
A lost phone? Shit, gotta go to the cell store, get a new sim, but only after I've gone home and grabbed an old phone, and spent a few hours setting it up. Including things like restoring 2FA stuff, which is a pain, at best.
Oh, you don't have spares lying around like I do? Then you're paying full price at the cell store, anywhere from $300+ (or getting a shitty low end phone, if your cell provider sells one).
And then I'd bet most people don't have proper backup/sync of their data, so would lose things like photos, downloads, etc.
Losing a phone is clearly far more of a hassle than losing a credit card or two.
I have to carry my "wallet" (little more than a money clip), for things like my ID, access cards, etc, anyway.
Using a phone to pay is far less convenient, and it's too many eggs in one basket, which is already an issue with phones.
I've also never had a credit card randomly reboot and get stuck in a bootloop. And in 30+ years of carrying them, I've had exactly one magnetic stripe failure - many years ago - and the stripe is rarely used any more.