The thing you're not accounting for is that work that primarily involves thought, which is what "office drones" are doing, aren't productive in the same way that physical or service jobs are.
Looking off into space thinking is part of the work. People average about four hours of productive work in an eight hour day.
The thing you can't do is get rid of half the people and then expect the other half to magically be eight hours productive per day. Businesses keep trying and weirdly it just tanks their output.
AI is not the panacea that so many people think it is. Do you feel happy when you need help with something you bought and you get an AI trying to offer you helpful articles or tips? I don't. Do you want the same level of service from the entity that controls where your paycheck gets deposited or fixed your HSA contributions?
If you definition of work is butts in chairs typing, office workers don't do too much work. But that's a very naive definition of what most office workers are actually doing.