I basically run my house IoT setup as you desire. My smart switches are a mix of Tasmota (open source firmware, running totally locally) and ZigBee (an open protocol for IoT interoperability). The whole lot is controlled by a NUC running home assistant. My doorbell camera also streams directly to the server.
Home Assistant basically acts to glue everything together, and provides nice, easy to use GUIs. It can also bridge between networks. It's easy to have all your IoT things on an isolated network, with no internet access. Only the HA install can see both networks.
I've also been careful of WAF (Wife Acceptance Factor). If the internet goes down, almost everything keeps working. If the NUC dies, the switches still work as dumb switches. The bulbs all default to full brightness neutral colour.