I've never noticed an appreciable performance hit, but I also don't generally swap much. Most of the time on a desktop/workstation I'm surprised to see a gig or 2 in swap. Nvme drives are pretty fast.
If you are actually using swap space on a regular basis it might be worth it to upgrade RAM or use a dedicated drive for swap if necessary.
I remember btrfs having swap file issues but the details are fuzzy, these days I use zfs on my nas and ext4 everywhere else.