It's also trained on data people reasonably expected would be private (private github repos, Adobe creative cloud, etc). Even if it was just public data, it can still be dangerous. I.e. It could be possible to give an LLM a prompt like, "give me a list of climate activists, their addresses, and their employers" if it was trained on this data or was good at "browsing" on its own. That's currently not possible due to the guardrails on most models, and I'm guessing they try to avoid training on personal data that's public, but a government agency could make an LLM without these guardrails. That data could be public, but would take a person quite a bit of work to track down compared to the ease and efficiency of just asking an LLM.