I came across this issue on my own discord server, the system kinda encourages you towards those higher security levels without really being especially clear about what it will do to the user experience.
One thing I would clear up though:
I think both sides in the OP are correct here.
Yes, the server admin sets the security level that triggers those requirements.
But it's also true that the server/admins do not get your phone number, that private information is only kept within discord's verification system. It is not sent to the server admins.