Voyager 2 went with a different trajectory specifically to fly by the outer planets. Voyager 1 went with a more aggressive gravity assist from both Jupiter and Saturn to gain the speed necessary to leave the solar system. So it's not only that it takes decades to get that far, but also the launch window of when different planets are aligned to make the mission feasible.