I thought it had something to do with the interface having an IP (...) LinkLocalAddressing=ipv4
I'm not so sure it is about the interface having an IP... I believe your current LinkLocalAddressing=ipv4 is forcing the interface to get up since it has to assign a local IP. Maybe you can set LinkLocalAddressing=no and ActivationPolicy=always-up and see how it goes.