I have no clue what could be causing that. I'd start looking into each link in the chain and making sure it's working.
But any halfway reasonable config should be able to handle audio playback, no matter how lossless. Audio-only just doesn't achieve datarates that would choke up... Anything.
Essentially, benchmark file transfers, transcoding, etc. Make sure each step of how it works is in fact working. Check drive SMART health... Whatever you can think of.
Also logs. No need to read through thousands of lines, but looking at the lines time stamped around when the issue occurs is always a good idea. FFMPEG logs, JF logs, client player logs, does SMB or whatever network drive protocol you're using have logs? If it does, check em.