PostgreSQL WAL Full on OpenShift (CP4I)

4 min readApr 18, 2026

In production environments running on OpenShift (especially IBM Cloud Pak for Integration / CP4I), one critical issue you might encounter is:

> 💣 **PostgreSQL fails to start due to a full WAL (Write-Ahead Log)**

This can lead to:

Database downtime
API/services becoming unavailable
Pods stuck or crashing

This article walks through a **safe, step-by-step recovery process** without risking data loss.

Initial Symptoms

Database pod not ready

oc get po -n cp4i | grep db

management-dfde3ed0-dfde3ed0-db-1                                0/1     Running            3 (56d ago)        119d

WAL is 100% full

oc rsh <db-pod>
df -h

[root@OCPDC ~]# oc rsh management-dfde3ed0-dfde3ed0-db-1
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
sh-5.1$ df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         100G   39G   62G  39% /
tmpfs            64M     0   64M   0% /dev
tmpfs            13G   82M   13G   1% /etc/hostname
tmpfs            13G   82M   13G   1% /etc/passwd
/dev/sda4       100G   39G   62G  39% /run
tmpfs           8.0G     0  8.0G   0% /dev/shm
/dev/rbd2       110G   58M  110G   1% /var/lib/postgresql/data
/dev/rbd3        28G   28G  1.9M 100% /var/lib/postgresql/wal
tmpfs           8.0G   20K  8.0G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs            32G     0   32G   0% /proc/acpi
tmpfs            32G     0   32G   0% /proc/scsi
tmpfs            32G     0   32G   0% /sys/firmware
sh-5.1$

/var/lib/postgresql/wal   28G   28G   0   100%

⚠️ What NOT to Do, never run

rm -rf /var/lib/postgresql/wal/*

Logs show recovery failure

No space left on device

oc logs management-dfde3ed0-dfde3ed0-db-1 --previous

{"level":"info","ts":"2026-02-20T14:28:04.758354024Z","logger":"postgres","msg":"record","logging_pod":"management-dfde3ed0-dfde3ed0-db-1","record":{"log_time":"2026-02-20 14:28:04.758 UTC","process_id":"33","session_id":"69986f74.21","session_line_num":"5","session_start_time":"2026-02-20 14:28:04 UTC","transaction_id":"0","error_severity":"FATAL","sql_state_code":"53100","message":"could not write to file \"pg_wal/xlogtemp.33\": No space left on device","backend_type":"startup","query_id":"0"}}

Root Cause

The issue is not just “disk full” — the real problem is:

> ❗ WAL cannot be cleaned because PostgreSQL still considers it required

Common causes:

inactive replication slots
* archiving backlog/failure
* unfinished recovery process

Solution

Step 1 — Verify WAL PVC

Issue Pod : management-dfde3ed0-dfde3ed0-db-1

oc get pvc -n cp4i | grep dfde3ed0-db

root@OCPDC ~]# oc get pvc -n cp4i | grep dfde3ed0-db
management-dfde3ed0-dfde3ed0-db-1                           Bound    pvc-d0e0dd95-9df7-4a01-82a1-d54991912b86   112Gi      RWO            ocs-storagecluster-ceph-rbd   <unset>                 290d
management-dfde3ed0-dfde3ed0-db-1-wal                       Bound    pvc-19937875-235d-485a-b989-7672d766b876   28Gi       RWO            ocs-storagecluster-ceph-rbd   <unset>                 290d

oc get pvc management-dfde3ed0-dfde3ed0-db-1-wal -n cp4i -o yaml

Step 2 — Resize WAL Storage

oc edit pvc <wal-pvc> -n cp4i

oc edit pvc management-dfde3ed0-dfde3ed0-db-1-wa -n cp4i

spec:
  resources:
    requests:
      storage: 30G   👈 Change this

To this

spec:
  resources:
    requests:
      storage: 100G   👈 To this or just increase as your prefered

Step 3 — Wait for resize to complete

oc get pvc <wal-pvc> -n cp4i -o yaml

oc get pvc management-dfde3ed0-dfde3ed0-db-1-wal -n cp4i -o yaml

status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 28Gi
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2026-04-18T05:36:53Z"
    status: "True"
    type: Resizing
  phase: Bound

Done

status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 94Gi
  phase: Bound

Step 4 — Restart the database pod

oc delete pod <db-pod> -n cp4i

oc delete pod management-dfde3ed0-dfde3ed0-db-1 -n cp4i

Step 5— Check if instance is fenced

oc logs management-dfde3ed0-dfde3ed0-db-1

{"level":"info","ts":"2026-04-18T05:38:19.785372194Z","msg":"Instance is fenced, won't start postgres right now","logger":"instance-manager","logging_pod":"management-dfde3ed0-dfde3ed0-db-1"}

oc get cluster management-dfde3ed0-dfde3ed0-db -n cp4i -o yaml | grep -i fenced

[root@OCPDC1 ~]# oc get cluster management-dfde3ed0-dfde3ed0-db -n cp4i -o yaml | grep -i fenced
    k8s.enterprisedb.io/fencedInstances: '["*"]'

This means all instances are locked

Step 6 — Unfence the instance

oc annotate cluster management-dfde3ed0-dfde3ed0-db -n cp4i k8s.enterprisedb.io/fencedInstances-

oc delete pod management-dfde3ed0-dfde3ed0-db-1 -n cp4i

Step 7 — Monitor status

[root@OCPDC1 ~]# oc get po -A | grep -i management-dfde3ed0-dfde3ed0-db-1
cp4i                                               management-dfde3ed0-dfde3ed0-db-1                                 1/1     Running     0                   24s

[root@OCPDC1 ~]# oc rsh management-dfde3ed0-dfde3ed0-db-1
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
sh-5.1$ df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         100G   39G   62G  39% /
tmpfs            64M     0   64M   0% /dev
tmpfs            13G   82M   13G   1% /etc/hostname
tmpfs            13G   82M   13G   1% /etc/passwd
/dev/sda4       100G   39G   62G  39% /run
tmpfs           8.0G  1.1M  8.0G   1% /dev/shm
/dev/rbd3       110G   59M  110G   1% /var/lib/postgresql/data
/dev/rbd2        93G   28G   65G  30% /var/lib/postgresql/wal
tmpfs           8.0G   20K  8.0G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs            32G     0   32G   0% /proc/acpi
tmpfs            32G     0   32G   0% /proc/scsi
tmpfs            32G     0   32G   0% /sys/firmware
sh-5.1$

/var/lib/postgresql/wal   93G   28G   65G   30%

Key Insights

WAL is not the main data store — it is a log of changes
WAL full is a **symptom**, not the root cause
PostgreSQL will not delete WAL if it is still needed