Tolerate read-only database in API and content worker heartbeats

[lucasc017]

Summary

• API and content workers now gracefully handle a read-only database (PostgreSQL SQLSTATE 25006) by skipping heartbeat writes instead of crash-looping
• Workers continue serving read traffic (collection downloads, EE pulls, API GETs) during the read-only window and resume normal heartbeats once the database becomes writable
• Adds _is_read_only_db_error() helper to detect read-only transaction errors across the psycopg __cause__ chain

Context

Galaxy.ansible.com's zero-downtime deployment temporarily switches the database to read-only. After pulpcore commit 3e8183bcc ("Mark a failed heartbeat as a catastrophic failure"), heartbeat write failures became fatal — causing API and content pods to crash-loop during this window even though they could still serve read requests.

Three crash sites are fixed in each of the two affected entrypoints (pulpcore/app/entrypoint.py and pulpcore/content/__init__.py):

1. Startup — AppStatus.objects.create() INSERT fails → set app_status = None, retry on next heartbeat cycle
2. Heartbeat — save_heartbeat() UPDATE fails → log warning, skip this cycle
3. Shutdown — app_status.delete() DELETE fails → log info, continue cleanup

The task worker (pulpcore/tasking/worker.py) already handles heartbeat failures gracefully and is unaffected.

Fixes: AAP-80698

Test plan

• Unit tests for _is_read_only_db_error() helper (pgcode, sqlstate, no-cause, wrong-code, nested-cause)
• Unit tests for API worker heartbeat (read-only skip, real DB error still fatal, retry create when app_status is None, recovery when DB becomes writable)
• Unit tests for API worker run() cleanup (tolerates DB error on delete, skips when no app_status)
• Unit tests for content worker heartbeat (read-only asave_heartbeat skip, read-only acreate skip)
• Existing heartbeat tests still pass (InterfaceError, OperationalError)
• Manual verification: switch pgbouncer to read-only endpoint in stage — API pods should remain up

🤖 Generated with Claude Code

[mdellweg]

From an earlier discussion on chat: "The heartbeat is only successful if the timestamp in the database was updated, because other components will read that timestamps and draw conclusions from it."

This approach is unsafe.

Also there is consideration for zero downtime upgrades in a live database:
https://pulpproject.org/pulpcore/docs/dev/learn/plugin-concepts/?h=zero#zero-downtime-upgrades

[Funi1234]

Hey @mdellweg
I Think the gap in the current approach is that there's no upper bound on how long a worker will tolerate read-only mode, so in theory it could run indefinitely with a stale timestamp.

Would it work to add a grace period and track when the worker first hits a read-only error, and if it's been read-only for longer than API_APP_TTL, crash anyway?
That way the heartbeat rule is preserved: after API_APP_TTL seconds without a successful write, downstream components would already consider the worker "missing".

If API_APP_TTL is too short for this window, could we make a new variable?