That doesn't sound realistically threatening to me.
The problem is that the vast majority of Matrix users are on the same server. So if you have a public or private room, most likely one of them will join your room from said server, thereby collecting and uploading all your room data to said server. Because of this, their servers probably contain the metadata of 99% of the Matrix-verse. Which is then very interesting to, and highly susceptible to, dragnet operations from various 3-letter agencies.
Realistically it's not a huge threat, and a huge step up from some other platforms, but it is undesirable for sure.