Id imagine all games rely on at least the X server running to handle the display.
I know back in the day you could do some cool stuff with framebuffer, but I don't know if you'd get 3d acceleration today even if you installed the drivers, because they probably need a bunch of libraries that are packaged as part of DEs/WMs
If you just want the experience of launching graphical stuff from the CLI, that can be done. You'd still install all the packages for your chosen display server and WM/DE, then you can write a small bash script that launches a desktop session and starts your program, then closes the desktop session after you exit the program.