Emby, Jellyfin, and Plex will all detect connection speed, adjust quality settings, and transcode the media to playback without buffering.
I wouldn't recommend Plex. They've been steadily moving away from self-hosted private media servers and towards just serving comercial content to you.
I myself run Emby as I'm rather fond of their development team and their attitude towards privacy. It does require payment for 'emby premier', ie the installable client apps and transcoding features, but it has single payment lifetime licenses as well as monthly.
Jellyfin is a popular open source option that is built on a fork of Embys older open source code before they went closed source.