When pg_upgrade Passes Checks but Crashes During Upgrade: A PostGIS--PROJ Library Conflict

📅 March 04, 2026 ✍️ PGCare

PostgreSQL upgrades with pg_upgrade are usually predictable and safe, especially when the pre-upgrade consistency check passes. However, in environments that rely on extensions like PostGIS, external library dependencies can still cause unexpected failures.

During a PostgreSQL 13 → 16 in-place upgrade, we encountered a situation where the pg_upgrade --check step completed successfully, but the actual upgrade process crashed PostgreSQL backends. The root cause turned out to be a runtime library conflict between multiple installed PROJ versions used by PostGIS.

This post walks through the investigation and how the issue was resolved.

<!--more-->

Environment

Upgrade scenario:

  • PostgreSQL 13 → 16
  • In-place upgrade using pg_upgrade
  • Linux (Rocky Linux / RHEL based)
  • PostGIS installed
  • Multiple PROJ libraries present on the system

Installed PROJ versions:

  • proj82
  • proj94
  • proj95

PostGIS depends on the PROJ library for coordinate transformations, and mismatched runtime libraries can cause instability.

Pre-Upgrade Validation

As recommended, the upgrade process started with a consistency check.

/usr/pgsql-16/bin/pg_upgrade -b /usr/pgsql-13/bin -B /usr/pgsql-16/bin -d /media/data/pgdata/data -D /media/data/pgdata/16 --check

The check completed successfully.

Typical output looked like:

Checking cluster versions                     ok
Checking database user                        ok
Checking prepared transactions                ok
Checking for presence of required libraries   ok

Since all checks passed, the upgrade was started.

Upgrade Crash

During the actual upgrade execution, PostgreSQL backends began crashing with SIGABRT.

Using coredumpctl helped inspect the failure.

coredumpctl gdb postgres

The stack trace showed:

#0 raise()
#1 abort()
#2 __libc_message()
#3 malloc_printerr()
#4 _int_free()
#5 osgeo::proj::common::UnitOfMeasure::~UnitOfMeasure()
#6 __run_exit_handlers()
#7 exit()
#8 proc_exit()
#9 PostgresMain()

The key indicator was:

libproj.so.25

This revealed the crash originated in the PROJ library, not PostgreSQL itself.

What This Means

The crash happened during proc_exit, when PostgreSQL cleans up backend processes.

A destructor inside the PROJ library attempted to free memory incorrectly, triggering:

malloc_printerr
_int_free

This is usually caused by:

  • double-free memory
  • heap corruption
  • incompatible shared library usage

Root Cause

The system had multiple PROJ versions installed simultaneously:

  • proj82
  • proj94
  • proj95

PostGIS had been compiled against one PROJ version, but the runtime linker loaded another version during execution.

This mismatch caused memory corruption during backend shutdown.

Identifying Which Library Was Loaded

To confirm which PROJ library PostgreSQL was using:

cat /proc/<backend_pid>/maps | grep libproj

Example output:

/usr/proj95/lib64/libproj.so.25

This confirmed the runtime library path being used by the PostgreSQL backend.

Checking PostGIS Version

SELECT postgis_full_version();

Example output:

POSTGIS="3.5.1" PGSQL="160" GEOS="3.12.1" PROJ="9.3.0"

This reveals which PROJ version PostGIS expects.

Fixing the Library Conflict

The solution was to ensure PostgreSQL always loads the correct PROJ library version.

1. Configure the dynamic linker

Add the correct library path:

echo "/usr/proj95/lib64" > /etc/ld.so.conf.d/proj95.conf
ldconfig

2. Verify runtime libraries

Confirm that PostgreSQL backends load the correct library.

cat /proc/<pid>/maps | grep proj

3. Ensure correct PostGIS build

Install the PostGIS package compiled for PostgreSQL 16.

yum install postgis35_16 proj95 proj95-devel

This ensures PostGIS and PROJ are compatible.

Why pg_upgrade Did Not Detect the Problem

pg_upgrade --check verifies:

  • catalog compatibility
  • extension presence
  • shared libraries

However, it does not execute extension code paths.

The crash occurred during runtime cleanup (proc_exit), which only happens when backend processes execute normally.

Because of this, the issue was invisible during the check phase.

Best Practices

1. Audit extensions before upgrades

SELECT extname, extversion FROM pg_extension;

2. Verify spatial dependencies

SELECT postgis_full_version();

3. Avoid multiple conflicting library versions

Having several versions of the same library increases the risk of runtime conflicts.

4. Validate runtime libraries

Check which shared libraries PostgreSQL loads during execution.

cat /proc/<pid>/maps

5. Ensure extension packages match the PostgreSQL version

Example:

postgis35_16

Conclusion

This upgrade failure was not caused by PostgreSQL itself, but by a PROJ library conflict introduced by PostGIS dependencies.

Key takeaways:

  • pg_upgrade --check cannot detect runtime library conflicts
  • PostGIS introduces additional external dependencies
  • Multiple library versions on a system can cause subtle runtime crashes
  • Always verify PostGIS and PROJ compatibility before major upgrades

Carefully validating the entire dependency chain is critical when upgrading PostgreSQL environments that rely on spatial extensions.