Definitely agree for a single install. If OP has a bunch of these installs to do, then editing an install USB to configure networking and enable sshd might be worth the effort. Do the install over ssh and hope the machine starts up as desired, but even then, if it doesn't just magically appear on the network, he's going to need a monitor to see where the startup failed.
Raspberry Pi's disk imager will let you pre-configure networking, accounts, and ssh, so you just write the image to an SD card, plug it in, and go. That's a great solutions for systems usually meant to be headless and removable media. If OP's client hardware allows, he could plug in the M2 or SATA drive meant to be the server's startup, install Deb there, and. transfer to the server hardware. That's definitely more work that just swapping the keyboard & monitor, but it accomplishes OP's stated goal. (Otherwise, a lot of this thread follows the linux meme of "How do I [X]?" "[X] is dumb, do [Y] instead.")