You can pin the pod to a specific node and pass through the USB device path and that will work. But the whole point of k8s is redundancy and workloads running anywhere.
Plus for IOT networks like zigbee and zwave, controller position in your house is important. If your server is more centrally located that may not be a concern for you.
I've heard of some using a USB serial over Ethernet device to relocate their controller remotely but i haven't looked into that. Running this one off rpi for the controller just made more sense for me.