Because Libertarians don't care about people's rights (in the modern US usage at least). For example, without government legalization of slavery would be the default since nothing is stopping it. Libertarianism would say if you can afford to buy a person, and they ended up in slavery because they weren't good enough or whatever, then it's fair that you should be able to purchase them.
Libertarianism is the philosophy of Ayn Rand. They're the people who want to remove government so they can fuck children and exploit people. They don't want liberty. They want authority, but by rich people not voters.
What you are talking about is Anarchism. Government shouldn't be telling people how to live their lives but should provide protections and assistance to allow them to live the best they can.