IMAP is a mess. None of the major email services or clients properly implement the protocol and pretty much all of the major email services have a proprietary replacement of their own with IMAP as an afterthought. That's why so many of the best email clients these days only work (or only have all features enabled) with Gmail or Office365.
For the user mostly it's just slow. It can literally take ten seconds just to check if there's any mail and that's if there are no new messages. When there are messages it takes much longer.
It's not slow because the servers are slow, it's slow because IMAP sucks. Too many requests and the requests are not really in a format that works well for the actual database format used by clients and servers.
For developers there are bigger problems, it's incredibly difficult to write an IMAP client that even works at all with all with all email servers.
JMAP fixes all of those issues. It's still not perfect, I think a perfect protocol would use Activity Streams, but it's definitely the best (open) email protocol available right now.