I suspect the delay would still be longer than a Youtube like implementation which may need to switch transcodes multiple times, but that's probably unrealistic at this point anyway.
Transcoding everything to AV1 could be a solution too, since high resolutions can look quite good at low bitrates, so you could limit it to 5mbps or 10mbps for any resolution and be done with it. But I'm not sure Jellyfin supports that, and at least from the UI it doesn't give you particularly fine grained control over resolution/bitrates. Perhaps having a secondary library of just AV1 transcodes that you handle manually (perhaps even using a software encoder) could be an option for some.
The client side is also an issue, with not that many devices supporting hardware decoding (although I've found it's fast enough in software with most modern smartphones at least).