That's what the "language" part of "Large Language Model" means. It processes, predicts and generates language.
Language does not mean "text". It's not "Large Text Generator". The core definition of the word language is "communication".
An LLM isn't (always) trained exclusively on text. And even those that are become greater than the raw sum of its parts. What that means is that the model can learn context not in the raw text itself.
The chat part just feeds it back the last couple messages for context
Partially true. There's more to it though.
It doesn't understand anything.
And neither does antivirus, but it still does its job.