I'm not sure how well docker-in-docker would work via portainer. Maybe it does, I've not tried it.
I would just do it from a folder you set up yourself and drop the docker-compose.yml in it, and go. If you want to share your dockercompose I can see if I notice a problem. I remember having to get over a couple issues at the time, but it's been a while and can't remember them offhand.