In theory, yes, you could make a mess, and any firmware is supposed to be certified to allow the device to be used.
In practice, this has been a convenient excuse to keep a whole chip with a separate OS in every smartphone, and it is very difficult to isolate from the rest of the system (see Graphene OS efforts).
I say all firmware should be opensource. Whether you're allowed to change them or not is a separate question... for now.