Skip to content

{184305174}: Close displaced DB_AM_RECOVER handles in __ufid_add_dbp#6064

Draft
chands10 wants to merge 1 commit into
bloomberg:mainfrom
chands10:accum
Draft

{184305174}: Close displaced DB_AM_RECOVER handles in __ufid_add_dbp#6064
chands10 wants to merge 1 commit into
bloomberg:mainfrom
chands10:accum

Conversation

@chands10

@chands10 chands10 commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

When __ufid_add_dbp is called for a file that already has an entry in the ufid-hash, the existing handle is displaced by the new one. The previous code cleared the added_to_ufid flag on the old handle but did not close it, causing a file descriptor leak.

Root Cause:

  • During replication, dbreg_register log records cause __dbreg_do_open to be called, creating a new DB_AM_RECOVER handle for the file
  • __ufid_add_dbp replaces the previous handle in the ufid-hash but only clears its added_to_ufid flag without closing it
  • The displaced handle remains in gbl_db_open_list holding an open file descriptor
  • Each subsequent dbreg_register for the same file displaces and leaks another handle
  • When delfiles deletes old file versions, these leaked handles keep the inodes alive, causing files to appear as "(deleted)" in lsof

Fix:

  • When displacing an existing DB_AM_RECOVER handle, update the ufid-hash to point to the new handle first, then close the old handle
  • The close removes the old handle from gbl_db_open_list, releasing the file descriptor

When __ufid_add_dbp is called for a file that already has an entry in
the ufid-hash, the existing handle is displaced by the new one. The
previous code cleared the added_to_ufid flag on the old handle but did
not close it, causing a file descriptor leak.

Root Cause:
- During replication, dbreg_register log records cause __dbreg_do_open
  to be called, creating a new DB_AM_RECOVER handle for the file
- __ufid_add_dbp replaces the previous handle in the ufid-hash but
  only clears its added_to_ufid flag without closing it
- The displaced handle remains in gbl_db_open_list holding an open
  file descriptor
- Each subsequent dbreg_register for the same file displaces and leaks
  another handle
- When delfiles deletes old file versions, these leaked handles keep
  the inodes alive, causing files to appear as "(deleted)" in lsof

Fix:
- When displacing an existing DB_AM_RECOVER handle, update the ufid-hash
  to point to the new handle first, then close the old handle
- The close removes the old handle from gbl_db_open_list, releasing the
  file descriptor

Signed-off-by: Salil Chandra <schandra107@bloomberg.net>
@chands10

chands10 commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

The DB_AM_RECOVER handles get leaked — not the files themselves directly. The chain is:

  1. __dbreg_do_open creates a DB_AM_RECOVER handle for a file and adds it to the ufid-hash via __ufid_add_dbp
  2. Later, a regular handle opens the same file (e.g., via bdb_open_more_tran during schema change reload)
  3. __ufid_add_dbp finds the existing entry, replaces ufid->dbp with the new handle, clears added_to_ufid on the old handle — but never closes the old handle
  4. The old DB_AM_RECOVER handle is now orphaned: it's not in the ufid-hash anymore, but it's still in gbl_db_open_list and still holds an open file descriptor

The file itself is fine — it's still on disk, still referenced by the new handle. The problem is the orphaned handle holding a duplicate file descriptor. Later, when delfiles removes old file versions and unlinks the files from disk, the orphaned handle's file descriptor keeps
the inode alive. That's what shows up as (deleted) in lsof — the directory entry is gone but the inode can't be freed because a leaked handle still has it open.

@roborivers roborivers left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cbuild submission: Success ✓.
Regression testing: 4/685 tests failed ⚠.

The first 10 failing tests are:
truncatesc_offline_generated [db unavailable at finish] **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
phys_rep
skipscan **quarantined**
sc_downgrade [timeout] **quarantined**
phys_rep_tiered [timeout]
phys_rep_tiered_nosource_generated [timeout]
phys_rep_tiered_firstfile_generated [timeout]
truncatesc [timeout] **quarantined**
reco-ddlk-sql [timeout] **quarantined**

@chands10 chands10 marked this pull request as draft July 8, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants