Interesting - I didn't bother to set the X-Real-IP headers until now and this might speed up my instance too. Thanks!
Then I wondered: what if the program is "smart" and throttles it by itself without any warning to the admin if it thinks that an ip address is sending too many requests?
The word you're looking for is "Rate Limit(ing)" and according to the documentation you could also disable it completely.
But I guess the cleanest and most secure solution would be to just set the headers on the reverse proxy.