Excellent! I've just started my own investigations in the same direction, so please keep us posted on your progress.
Have you looked at Friendica? It requires more of a server than what I've got, but it looks pretty good to me.
A couple of days ago, I set up a GoToSocial instance. It's still alpha software, but it looks pretty good so far and is a lot less resource intensive than Mastodon. It's also server-only, but the clients I tried all worked with it just fine.
I've been able to follow people on regular Mastodon instances, but haven't interacted outside my instance beyond mere following.
It was easy to get going on "bare metal" (no Docker), although I had to start over once because of a documentation gotcha. (The instructions as-is will leave you with an old version instead of the newest version.)