I use Jellyfin. I think in your use case, each user would be setup have their own library. You can enable or disable library on a per user basis as will as a per client basis.
Downside is that the default web interface isn’t great as a music player. It does the job but it’s not great.
Other hand, multiple music-first clients exist for a lot of different platforms. Odds are good you can find a client that suits how you listen to music.