yopp
Oh fuck #postgres. In 18 they made a change in data path from `/var/lib/postgresql/data` to `/var/lib/postgresql/`.

EDIT:

I was wrong, it's not postgres change, it's change made in "official" docker image, which in fact is not official and maintained by "the PostgreSQL Docker Community". My apologies to pg people.

They "the PostgreSQL Docker Community" (not affiliated with Postgres in any way) did exactly zero checks so users will shoot themselves in a leg in two ways:

  • kinda benign: after 17 → 18 upgrade it won’t start
  • fatal: user mounted docker volume at the old pathContainer restarts and data gone (that’s me)

    • What the actual fuck. Why on earth you couldn’t do a simple fucking check that old directory exists and has data, but new directory is empty, and then start with directory with data spitting warnings that default has been changed and you need to do something about that.

    This is utter incompetence and giving exactly zero fucks about your user base

    Mar 23 at 11:00amWeb
    Show threadRégis Haubourg1d ago
    @alex this is me migrating tonight to 18. Had to switch to a full pgdump for 12 To..
    Show threadAndreas Scherbaum1d ago
    @alex What is your environment or OS?
    Show threadTomas Vondra1d ago
    @ascherbaum @alex Yeah. I wonder what packages are being used?
    Show threadyopp1d ago

    @tomasv @ascherbaum nothing special: dockerized environment, official docker image. I’ve deployed one of the services with pg18 but used default mount path to data. So I was unplesantly surprised to find empty database after 18.2 to 18.3 image upgrade. Old container was gc’ed because both pg and app healthchecks passed. Which is technically correct: app rolled bootstrap migration and went up in blank state and pg was indeed available. But data is gone. It was expected that there could be issues with 18, so it was first deployment with not that important data.

    Another another lesson that observability as useful as your risk assessment. Also lesson in trust in vendors: no matter how consistent in their changes they were in past, they absolutelly will do stupid things. That on me. I could’ve paid more attention that’s also on me.

    But doing change of this magnitude so careless and without any proper discussion of what can go wrong for downstream users is on the people who made this damn pull request merged.

    Worst part this change is purely about aesthetics and addresses exactly zero issues: pg sill gives no shit about operational complexity of major version upgrades. While I respect people around pg, database itself is an example of software that is plan hostile to users. It’s never joy, it always different from of pain.

    Show threadAndreas Scherbaum1d ago

    @alex @tomasv What does "official docker image" mean?

    If you refer to this:

    https://hub.docker.com/_/postgres

    This is kind of the official "Docker Postgres Image", but this is not maintained by the PG project.

    Blaming Postgres for this is the wrong audience.

    postgres - Official Image | Docker Hub

    The PostgreSQL object-relational database system provides reliability and data integrity.

    Show threadyopp1d ago

    @ascherbaum @tomasv

    🤦 thanks for pointing this out.

    My apologies for pg people. I stand corrected, it's maintained by "docker community" (what the fuck "official" means then)

    Show threado/1MS\o ⌨️🐧 | #WeAreNatenom1d ago

    @alex I hope you know, that a dump/restore is required to migrate data from any previous PostgreSQL major release.
    https://www.postgresql.org/docs/current/upgrading.html

    Also, with PostgreSQL 18, initdb creates new databases by default with enabled data checksums.
    Based on this change, an in-place migration with pg_upgrade is only possible, when database running the old PostgreSQL version is also created with data checksums.

    18.6. Upgrading a PostgreSQL Cluster

    18.6. Upgrading a PostgreSQL Cluster # 18.6.1. Upgrading Data via pg_dumpall 18.6.2. Upgrading Data via pg_upgrade 18.6.3. Upgrading Data via Replication …

    PostgreSQL Documentation
    Show threadTomas Vondra1d ago
    @db_geek @alex No, dump/restore is not required. pg_upgrade exists for a long time.
    Show threado/1MS\o ⌨️🐧 | #WeAreNatenom1d ago

    @tomasv @alex Yes, I know.
    I'm not sure if many people in the self-hosting sector are truly aware of some complexity when dealing with PostgreSQL DBs.

    My main intention was the advice, that it is not possible to mount the dabase files from a previous PostgreSQL major version without a migration with a new PostgreSQL major version.

    Also using pg_upgrade is, when running PostgreSQL in a container, not so easy.
    One possible way can be to use this container image:
    https://github.com/tianon/docker-postgres-upgrade