It feels very much like most stuff that's likely to be developed for it will have the feel of "museum exhibit at home" or AR-ified iOS app.
The inability to use any controller is going to lose them a lot of latency and precision sensitive usecases. It is very Apple to make it totally standalone, but it's going to cost them a fair bit.
A lot of real time remote control usecases will be impossible for latency issues alone, it won't be a good solution in most multiuser environments (both due to no relative tracking, but also cost and hygiene issues for shared devices), it won't be great for bringing into public spaces (poor long range tracking, etc) or small spaces (limits gestures), hand tracking camera position means you have to hold your hands up and mostly open (accessibility issues), etc.
Even if the hardware can do more, Apple won't give developers access to more.