Are these cats native (or naturalised) to your local ecosystem? If wherever you live has had cats for a hundred years or so, the local wildlife would have adapted to them. Otherwise, cats can damage the local ecosystem.
Do you rely on the cats to suppress vermin (rats, squirrels, small birds, etc.)? Even if your cats aren't actively killing them, their mere 'patrolling' can drive these pests away. But if you keep them indoors, you lose this protection.
Are there any local predators that are particularly good at catching cats?
If your answers are yes, yes and no, then let your cats out. If they are no, no and yes, keep them in as far as possible.