That's a fair point. But I suppose even with leisurely activities, it could help you improve your other activities, mental faculties and functioning, depending on how you do it or what that activity is. Playing sports is maybe fun but it also improves your ability to work in a team, which is important in a workplace. Depending on the videogame you're playing, it can also be educational and improve temperament. I learned lots of history and geography playing Hearts of Iron. Also funny enough, at the time, I thought I might be wasting time trying to get past a difficult section in Call of Duty: World at War over and over for at least one hour. But for some reason I thought to myself I won't be a quitter. Playing that game ironically developed my perserverance.
I think it depends what that "time wasting" activity is if you could reflect on how that would make a positive impact on you. Albert Einstein plays violin when he needs time to think of solutions to a problem. And if you know how he came to the idea of theory of general relativity, it does sound more creative than scientific. He did say imagination is more important than knowledge.