While true, it's a bit more nuanced than that. They can absolutely have requirements in the contract that will put you on site. For instance, they can have you being the one to set up the conference room for the morning meeting. They can also categorically say that their VPN access is only for FTEs.
But as an independent business negotiating a contract, you just haggle these terms away. It's still a good idea to document expectations, including work hours and locations.