From 198b7b5c892f94066ed0d5f46eaa0882b1bc63d0 Mon Sep 17 00:00:00 2001 From: David Schilling Date: Sun, 11 Oct 2026 06:19:31 +0200 Subject: [PATCH] MDEV-41450 Purge coordinator loops forever in purge_sys_t::close_and_reopen() and leaks THD mem_root (22+ GB in < 2 min) for tables with an indexed virtual column On a Galera node, a TOI DDL (e.g. DROP TABLE, ALTER TABLE) on a table with an indexed virtual column can BF-abort the InnoDB purge coordinator. The purge coordinator THD (system_thread != NON_SYSTEM_THREAD) holds an MDL_SHARED/MDL_EXPLICIT ticket on every table that a purge batch opened via open_purge_table(). When the TOI DDL requests an exclusive MDL on such a table, wsrep_handle_mdl_conflict() finds the purge coordinator as the granted lock owner and, since MDEV-38218 removed the "wait for system threads" branch, calls wsrep_abort_granted(), which sets THD::killed on the coordinator THD. Nothing ever resets this flag: the purge coordinator is not a user connection and has no statement to abort. From then on every open_table() in open_purge_table() fails with "query execution was interrupted", so purge_sys_t::close_and_reopen() (retry loop introduced by MDEV-39261) spins forever. Every iteration allocates a new TABLE_LIST in the coordinator's thd->mem_root, which is never freed, so the mariadbd RSS grows by hundreds of MB per second until the process is OOM-killed. The node never purges again and does not shut down cleanly. The combination that triggers this exists since MDEV-38218 and MDEV-39261 were both merged: 10.11.17, 11.4.11, 11.8.7, 12.3.2 and later. 10.6 is not affected (MDEV-38218 was not applied there). Fix, three parts: * wsrep_handle_mdl_conflict(): do not BF-abort a server-internal background thread that holds explicit MDL. Such a thread has no wsrep transaction to roll back and never resets THD::killed. Instead the requesting (TOI) thread waits for the lock to be released, as it did before MDEV-38218. This is the part that prevents the kill. * open_purge_table(): pass MYSQL_OPEN_IGNORE_KILLED. Purge has no statement whose interruption THD::killed could signal; a stale kill flag on the purge THD must not make every subsequent open_table() fail. This makes close_and_reopen() able to make progress even if the purge THD got killed for some other reason. * trx_purge_close_tables(): call reset_thd() instead of close_thread_tables() so that thd->mem_root is freed together with the tables. Nothing else in the purge coordinator's mem_root outlives its open tables, so the TABLE_LIST allocated by every open_purge_table() call is released at the end of each batch and each close_and_reopen() retry, bounding the memory use even if the retry loop is entered. Reproduced with a 2-node Galera cluster (docker, mariadb:11.4.13 and mariadb:12.3.3): one node runs a constant UPDATE/DELETE load on an InnoDB table with an indexed virtual column while the other node repeatedly executes DROP TABLE / CREATE TABLE on a second table with an indexed virtual column. Within about 40 s the first node's purge coordinator has killed = KILL_QUERY_HARD and its memory/sql/THD::main_mem_root in performance_schema grows by ~20 MB/s until OOM. With this patch the same workload shows "MDL conflict, waiting for system thread" in the wsrep debug log, the DDL completes and main_mem_root stays below 1 MB for the duration of the test (verified with builds of both 11.4 and 12.3). --- sql/sql_class.cc | 7 +++++-- sql/wsrep_mysqld.cc | 13 +++++++++++++ storage/innobase/trx/trx0purge.cc | 9 +++++++-- 3 files changed, 25 insertions(+), 4 deletions(-) diff --git a/sql/sql_class.cc b/sql/sql_class.cc index e744bbd5f8fd7..5a7ef672a471f 100644 --- a/sql/sql_class.cc +++ b/sql/sql_class.cc @@ -5168,8 +5168,11 @@ TABLE *open_purge_table(THD *thd, const char *db, size_t dblen, DBUG_ENTER("open_purge_table"); DBUG_ASSERT(thd->locked_tables_mode < LTM_PRELOCKED); - /* Purge already hold the MDL for the table */ - Open_table_context ot_ctx(thd, MYSQL_OPEN_HAS_MDL_LOCK); + /* Purge already hold the MDL for the table. A kill flag on a purge THD + (for example from a Galera BF abort) is never reset, and purge has no + statement to abort; it must not make every subsequent open_table() fail. */ + Open_table_context ot_ctx(thd, MYSQL_OPEN_HAS_MDL_LOCK | + MYSQL_OPEN_IGNORE_KILLED); TABLE_LIST *tl= (TABLE_LIST*)thd->alloc(sizeof(TABLE_LIST)); LEX_CSTRING db_name= {db, dblen }; LEX_CSTRING table_name= { tb, tblen }; diff --git a/sql/wsrep_mysqld.cc b/sql/wsrep_mysqld.cc index dc9809ad5263e..097176103aa0c 100644 --- a/sql/wsrep_mysqld.cc +++ b/sql/wsrep_mysqld.cc @@ -3626,6 +3626,19 @@ void wsrep_handle_mdl_conflict(MDL_context *requestor_ctx, // Granted thread is either TOI or applying wsrep_handle_granted_bf(request_thd, granted_thd, ticket, key); } + else if (granted_thd->system_thread != NON_SYSTEM_THREAD && + granted_thd->mdl_context.has_explicit_locks()) + { + /* Granted thread is a server-internal background thread holding an + explicit MDL, e.g. the InnoDB purge coordinator holding MDL_SHARED + on a table for the duration of a purge batch. Such a thread has no + wsrep transaction to roll back and never resets THD::killed, so a + BF abort would leave it killed for the rest of the server lifetime + (MDEV-41450). It releases its locks on its own; the requestor waits, + as it did before MDEV-38218. */ + wsrep_mdl_log(WSREP_MDL_DEBUG, "MDL conflict, waiting for system thread", + request_thd, granted_thd, ticket, key); + } else { // Granted thread is not brute-force thread, we can abort it diff --git a/storage/innobase/trx/trx0purge.cc b/storage/innobase/trx/trx0purge.cc index 19cec8f269e26..147d02ab76a3c 100644 --- a/storage/innobase/trx/trx0purge.cc +++ b/storage/innobase/trx/trx0purge.cc @@ -48,7 +48,7 @@ Created 3/26/1996 Heikki Tuuri TABLE *open_purge_table(THD *thd, const char *db, size_t dblen, const char *tb, size_t tblen, MDL_ticket *mdl_ticket) noexcept; -int close_thread_tables(THD* thd) noexcept; +void reset_thd(MYSQL_THD thd); MDL_ticket *get_mdl_ticket(TABLE *table) noexcept; /** Maximum allowable purge history length. <=0 means 'infinite'. */ @@ -1106,7 +1106,12 @@ static void trx_purge_close_tables(THD *thd, purge_table *last_entry, node->tables.clear(); } - close_thread_tables(thd); + /* Close the tables and free thd->mem_root, which holds the TABLE_LIST + allocated by every open_purge_table() call. Nothing else in the + coordinator's mem_root outlives its open tables, and without this + every batch (and every close_and_reopen() retry) would leak one + TABLE_LIST per opened table. */ + reset_thd(thd); for (auto mdl : mdl_tickets) {