I would disagree. How you dress, how you present, how you act, these are all things that speak without words. I would call those speech.
Truthfully, though, if I were coining the phrase, I would have said "freedom of expression" and "freedom of thought" instead of "freedom of speech" and "freedom of worship". Both of those are broader categories that encapsulate the concepts FDR articulated. After all, what is worship but conceptualizing the deep thoughts about where the universe came from, and finding a community of like minded folks?