Uh, each service only has access to your public key, not the private one that stays with you. It's less risky than a regular password.
Even with U2F hardware keys where the server-side stores the encrypted key (to allow for infinite sites to be used with a single hardware key), it's only decryptable on your key and thus isn't that useful for someone who has compromised a service.