In order to make it into a Discord or Zoom competitor you would need to solve far higher bandwidth things like HD video and low latency audio, and both of thouse are fundamentally very different things for a server to handle as compared to high latency short text messages.
A large number of Discord servers just use text.
For video, maybe integrate into something that already exists, like Jitsi? Instead of trying to build one single app that handles everything, maybe it would be nice to have a suite of apps that all work together and can all use the same login.
A lot of video conferencing systems are already mostly peer-to-peer, at least for enterprise apps. Skype was originally peer-to-peer too. NAT traversal is usually provided by STUN servers. There's some issues like that (for example it reveals the user's IP addresses) but you could proxy everything through a TURN server to solve that.
Peer to peer is the best way to implement end-to-end encrypted communication.
Having said that, very large groups can benefit from a client-server model, like what Zoom does.