“The magic is not that the model can learn math problems in English and then generalize to new math problems in English,” says Barak, “but that the model can learn math problems in English, then see some French literature, and from that generalize to solving math problems in French. That’s something beyond what statistics can tell you about.”
It is not magic and all this "it's magic" discourse is IMO counter-productive. When a model does something interesting, people need to dig on what it's doing and why, for better models; and by "interesting" I mean both accurate and inaccurate (enough of this "it's hallu, move on!" nonsense).
And it's still maths and statistics. Yes, even if it's complex enough to make you lose track of. To give you an example, it's like trying to determine exactly the position of every atom of oxygen and silicon in a quartz crystal, to know how it should behave - it should be doable if not by the scale.
Now, explaining it: LLMs are actually quite good at translation (or at least better than other machine-based translation methods). Three things might be happening here:
It converts the prompt into French, then operates on French tokens.
It operates on English tokens, then converts the output to French tokens.
It converts the logical problem itself into an abstract layer, then into French.
I find #1 unlikely, #2 the most likely, but the one that would interest me the most is #3. It would be closer to how humans handle language; we don't really think too much by chaining morphemes ("tokens"), we mostly handle what those morphemes convey.
It would be far, far, far more interesting if this was coded explicitly into the model, but if it appeared as emergent behaviour it would be better than nothing.