Your Https connections are also symmetric, so that's a silly thing to dislike.
The handshake and key exchange are asymmetric, and used to establish a symmetric session key.
ZigBee encryption is fine for the use case, because you're only adding devices you know are being added. You inherently trust that your physical ZigBee device is the device it claims to be.
There's potentially an opportunity to hijack the key exchange between devices at network join, but you'd have to approve the listening device to your network in the first place.