Matrix also does have a pretty big problem with meta data. By default it stores a ton of meta data (at least the reference server implementation does) and I am not sure if this is even a solvable problem without redesigning the protocol. When opting for an alternative to Signal, XMPP is probably the better choice.