At least for windows, you can ctrl-z that away and it'll handle your mouse fumble. Explorer also highlights the files after a copy so if that doesn't work (and it was a copy action), just delete them immediately.
I haven't used *nix for daily stuff in years but I'm sure the same abilities are there, surely.