Welcome to Incremental Social! Learn more about this project here!
Check out lemmyverse to find more communities to join from here!

Are you reusing one postgres instance for all services?

I have many services running on my server and about half of them use postgres. As long as I installed them manually I would always create a new database and reuse the same postgres instance for each service, which seems to me quite logical. The least amount of overhead, fast boot, etc.

But since I started to use docker, most of the docker-compose files come with their own instance of postgres. Until now I just let them do it and were running a couple of instances of postgres. But it's kind of getting rediciolous how many postgres instances I run on one server.

Do you guys run several dockerized instances of postgres or do you rewrite the docker compose files to give access to your one central postgres instance? And are there usually any problems with that like version incompatibilities, etc.?

Shimitar ,
@Shimitar@feddit.it avatar

This is one of the annoying issues with docker, or better, on how docker is abused in production.

The single instance/multiple databases is the correct way to go, docker approach mess up with that.

Rewriting docker files is always a possibility but honestly defies the reason why docker is used by self hosters.

Also beware that some devs will shunt you out of support if you do, specially the apps that ships docker files by default.

Go bare metal if possible, that way you have full control. Do docker for testing up stuff quickly and be flexible at cost of accepting how stuff is packaged by upstream

sardaukar ,

The official Postgres Docker image is geared towards single database per instance for several reasons - security through isolation and the ability to run different versions easily on the same server chief among them. The performance overhead is negligible.

And about devs not supporting custom installation methods, I'm more inclined to think it's for lack of time to support every individual native setup and just responding to tickets about their official one (which also is why Docker exists in the first place).

MrMcGasion ,

That's a big reason I actively avoid docker on my servers, I don't like running a dozen instances of my database software, and considering how much work it would take to go through and configure each docker container to use an external database, to me it's just as easy to learn to configure each piece of software for yourself and know what's going on under the hood, rather than relying on a bunch of defaults made by whoever made the docker image.

I hope a good amount of my issues with docker have been solved since I last seriously tried to use docker (which was back when they were literally giving away free tee shirts to get people to try it). But the times I've peeked at it since, to me it seems that docker gets in the way more often than it solves problems.

I don't mean to yuck other people's yum though, so if you like docker, and it works for you, don't let me stop you from enjoying it. I just can't justify the overhead for myself (both at the system resource level, and personal time level of inserting an additional layer of configuration between me and my software).

Lifebandit666 ,

I agree to a certain extent and I'm actively using Docker.

What I've done is made an Ubuntu VM, put Docker on it and booted a Portainer client container on it, then made that into a container template, so I can just give it an IP address and boot it up, then add it to Portainer in 3 clicks.

It's great for just having a go on something and seeing if I wanna pursue it.

But so far I've tried to boot and run Arr and Plex, and more recently Logitech Media Server and it's just been hard work.

I've found I'm making more VMs than I thought I would and just putting things together in them, rather than trying to run stacks of Docker together.

That said, it looks like it is awesome when you know what you're doing.

sardaukar ,

It's kinda weird to see the Docker scepticism around here. I run 40ish services on my server, all with a single docker-compose YAML file. It just works.

Comparing it to manually tweaking every project I run seems impossibly time-draining in comparison. I don't care about the underlying mechanics, just want shit to work.

MrMcGasion ,

I think that my skepticism and desire to have docker get out of my way, has more to do with already knowing the underlying mechanics, being used to managing services before docker was a thing, and then docker coming along and saying "just learn docker instead." Which is fine, if it didn't mean not only an entire shift from what I already know, but a separation from it, with extra networking and docker configuration to fuss with. If I wasn't already used to managing servers pre-docker, then yeah, I totally get it.

sardaukar ,

I used to be a sysadmin in 2002/3 and let me tell you - Docker makes all that menial, boring work go away and services just work. Which is want I want, instead of messing with php.ini extensions or iptables custom rules.

MrMcGasion ,

Maybe I'll try and give it another go soon to see if things have improved for what I need since I last tried. I do have a couple aging servers that will probably need upgraded soon anyway, and I'm sure my python scripts that I've used in the past to help automate server migration will need updated anyway since I last used them.

Moonrise2473 ,

I have everything in docker too, but a single yml with 40 services is a bit extreme - you would be forced to upgrade everything together, no?

sardaukar ,

Not really. The docker-compose file has services in it, and they're separate from eachother. If I want to update sonarr but not jellyfin (or its DB service) I can.

skittlebrau ,

I care about the underlying tech in the things I deploy, but the reality is that I lack the time to actively do this in practice.

Ideally I would set everything up manually, but it’s just too hard to keep up with 30+ projects and remembering how/why I set everything up, even with documentation. Docker Compose makes my homelab hobby more manageable.

summerof69 ,

What overhead are you talking about? You don't need a dozen of instances of a database. You can create one, with or without docker, and configure any service to use it. The idea of docker and docker compose is that you can easily start up the whole env. But you don't have to.

nik282000 ,
@nik282000@lemmy.ca avatar

I keep each service separate as far as DBs, if something breaks or get a major upgrade I don't have to worry about other containers.

ad_on_is ,
@ad_on_is@lemmy.world avatar

I use the provided databases in the docker-compose file, since some services require a specific version and I'm too lazy to investigate whether it works on my existing service or not.

DeltaTangoLima ,
@DeltaTangoLima@reddrefuge.com avatar

I run Proxmox with a few nodes, and each of my services are (usually) dockerized, each running in a Proxmox Linux container.

As I like to keep things segregated as much as possible, I really only have one shared Postgres, for the stuff I don't really care about (ie. if it goes down, I honestly don't care about the services it takes with it, or the time it'll take me to get them back).

My main Postgres instances are below - there's probably others, but these are the ones I backup religiously, and test the backups frequently.

  1. RADIUS database: for wireless auth
  2. paperless-ngx: document management indexing & data
  3. Immich: because Immich has a very specific set of Postgres requirements
  4. Shared: 2 x Sonarr, 3 x Radarr, 1 x Lidarr, a few others
bjoern_tantau ,
@bjoern_tantau@swg-empire.de avatar

Asked the same question a little while ago. See https://swg-empire.de/post/625121 for more opinions.

I ended up putting Nextcloud and Lemmy on one service. No problems so far, resource usage and performance are great. But I've not been running it for very long.

matto ,

Not so long ago I had the same question myself, and I ended up setting 1 Postgress instance and 1 MySQL instance for all services to share.
In the long run, I had so many version and settings incompatibilities across services that moved back to one DB per service that is tuned specifically for it.
Also, I add a backup app to all my docker compose files that have a DB in it. This way, backups happen periodically and automatically.

richmondez ,

Which db backup app do you use if you don't mind me asking?

tritonium , (edited )

You don't need a db backup app... bind mount the data to a location then just stop the container and have borg take the backups. You can do this with all your containers.

/docker/postgres

/docker/postgres/data

/docker/postgres/compose.yml

And do that with every container. Easy as fuck to backup and restore them.

matto ,

Sorry! I know it's been ages, but this is what I use:

marcos ,

Do not run databases in Docker unless you know really well what you are doing.

It's completely possible to run them correctly in Docker. But it's far from trivial, and if you need to ask this, it means that you probably won't be able to.

foggy ,

Nothing worse in Linux communities than gatekeeper answers like this.

It's fine to point out that something's challenging to someone who may be a novice, but to suggest it's above them? Eat it. At the very least, provide a resource and let them confirm for themselves.

cm0002 ,

Cybersecurity communities too, there was one guy on [The Other Site] I saw awhile back who, whenever somebody asked a question about what they should do to secure X or Y or if Z security product was better than V because they just did general IT, would always default to something along the lines of "If you don't know, don't bother its above you and you should shell out $$$ to an actual firm otherwise you'll be shelling out $$$$ to another firm to clean up your mess"

Surprise surprise, when I googled his username (The fact I was even able to do this isnt a great sign for a "security professional" IMO lmao) he actually owned one of those "Databreach Triage" firms...yea...I'm sure there was no conflict of interest whatsoever lmaoo

richmondez ,

This right here... This whole community is about learning to do things for yourself. It might be after been given resources to learn you do decide its too much for you but people should be given the chance to discover that themselves.

foggy ,

Shout out to hack the box.

If you're a noob or a veteran in any branch of IT looking for a good cybersecurity community/platform...

Most of us IT folk check the box of "knowledge peaks and valleys". They're the first community I've found that seems to actually respect the idea that someone might know way more about XSS and SQL injection in react apps than some other guy knows about binary exploits through packet disassembly, and that both of them are fucking experts and neither of them are lacking for not knowing what the other knows.

Bookmeat ,

If you don't care about down time then feel free to consolidate all your apps to one postgresql instance. If you want to do maintenance on your DB it may impact all your other applications.

Shimitar ,
@Shimitar@feddit.it avatar

My database instances downtime is only when the server itself is rebooting. Never had a single downtime in 20+ years beside that.

sardaukar ,

You've never had to run migrations that lock tables or rebuild an index in two decades?

Shimitar ,
@Shimitar@feddit.it avatar

Why would that have blocked all my databases at once? That would affect the same database I was migrating, not the others.

sardaukar ,

Yes, it would cause downtime for the one being migrated - right? Or does that not count as downtime?

Shimitar ,
@Shimitar@feddit.it avatar

Yes it counts indeed... But in that case the service is down while its migrated so the fact the database is also down does it count?

I mean, it's a self hosted home service, not your bank ATM network...

andrewalker ,

I have a single big Postgres instance, shared among immich, paperless, lldap, grafana, and others. I only use the provided docker-compose as inspiration and do my own thing. It's nicer to back up a single database (plus additional volumes, but still).

atzanteol ,

I only use the provided docker-compose as inspiration and do my own thing

This is the correct way to look at it. Most applications that provide a docker compose do so as a convenience to get started quickly. It's not necessarily what you should run.

sunaurus ,
@sunaurus@lemm.ee avatar

If I have several backends that more or less depend on each other anyway (for example: Lemmy + pict-rs), then I will create separate databases for them within a single postgres - reason being, if something bad happens to the database for one of them, then it affects the other one as well anyway, so there isn't much to gain from isolating the databases.

Conversely, for completely unrelated services, I will always set up separate postgres instances, for full isolation.

ptz ,
@ptz@dubvee.org avatar

I used to, but now I just have one big one (still in Docker) that's sized and tuned to handle all of my applications.

I've only had one version compatibility issue, but that was because I was on pgSQL 13 and the updated version of one application needed 15. Upgrading that didn't affect any other applications. If it had, I would have just broken that one application out to its own stack-local Postgres.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • selfhosted@lemmy.world
  • incremental_games
  • meta
  • All magazines
    •