Yes, the "to address" cannot be encrypted as it is necessary to deliver the mail, the "from address" are needed to send a notification when the "to address" doesn't exist.
Technically, the "from address" probably can be encrypted, like in signal; but I think it is required in the current email standard.