The idea is that your passwords are stored on the phone. You want a separate long random password for each account, so it's unfeasible to remember them. It's also a big pain to type every one such password on a screen keyboard. Thus, the password and the phone are the same factor.
I have avoided having important passwords on my phone because of this, but some people use their phones more heavily than I do. My more important accounts are only accessed via my laptop, using a TOTP phone app as 2nd factor. I rarely take the laptop out of the house.