MDEV-41450 Purge coordinator loops forever in purge_sys_t::close_and_reopen() and leaks THD mem_root (22+ GB in < 2 min) for tables with an indexed virtual column

[djschilling]

• The Jira issue number for this PR is: MDEV-41450

Description

On a Galera node, a TOI DDL (e.g. DROP TABLE, ALTER TABLE) on a table
with an indexed virtual column can BF-abort the InnoDB purge coordinator.
The purge coordinator THD (system_thread != NON_SYSTEM_THREAD) holds an
MDL_SHARED/MDL_EXPLICIT ticket on every table that a purge batch opened
via open_purge_table(). When the TOI DDL requests an exclusive MDL on such
a table, wsrep_handle_mdl_conflict() finds the purge coordinator as the
granted lock owner and, since MDEV-38218 removed the "wait for system
threads" branch, calls wsrep_abort_granted(), which sets THD::killed on
the coordinator THD. Nothing ever resets this flag: the purge coordinator is
not a user connection and has no statement to abort. From then on every
open_table() in open_purge_table() fails with "query execution was
interrupted", so purge_sys_t::close_and_reopen() (retry loop introduced by
MDEV-39261) spins forever. Every iteration allocates a new TABLE_LIST in
the coordinator's thd->mem_root, which is never freed, so the mariadbd RSS
grows by hundreds of MB per second until the process is OOM-killed. The node
never purges again and does not shut down cleanly.

The combination that triggers this exists since MDEV-38218 and MDEV-39261
were both merged: 10.11.17, 11.4.11, 11.8.7, 12.3.2 and later. 10.6 is not
affected (MDEV-38218 was not applied there).

Fix, three parts

• wsrep_handle_mdl_conflict(): do not BF-abort a server-internal
  background thread that holds explicit MDL. Such a thread has no wsrep
  transaction to roll back and never resets THD::killed. Instead the
  requesting (TOI) thread waits for the lock to be released, as it did
  before MDEV-38218. This is the part that prevents the kill.

• open_purge_table(): pass MYSQL_OPEN_IGNORE_KILLED. Purge has no
  statement whose interruption THD::killed could signal; a stale kill flag
  on the purge THD must not make every subsequent open_table() fail. This
  makes close_and_reopen() able to make progress even if the purge THD got
  killed for some other reason.

• trx_purge_close_tables(): call reset_thd() instead of
  close_thread_tables() so that thd->mem_root is freed together with the
  tables. Nothing else in the purge coordinator's mem_root outlives its open
  tables, so the TABLE_LIST allocated by every open_purge_table() call
  is released at the end of each batch and each close_and_reopen() retry,
  bounding the memory use even if the retry loop is entered.

How it was found

We hit this on a 3-node Galera cluster running 12.3.3 while migrating
~3500 databases (each with a table that has an indexed VIRTUAL generated
column) from a 10.6 cluster with mariadb-dump / mariadb (DROP/CREATE
TABLE as TOI DDL, then bulk INSERT). One node, which only applied write
sets, grew from 8.7 GiB to 38.5 GiB RSS (plus 7 GiB swap) in about a minute
and was OOM-killed; the other two nodes stayed at ~4 GiB. perf record -e page-faults on the affected node attributed 99 % of the page faults to
open_purge_table() on an ib_tpool_worker thread. The reproducer below
then showed the purge coordinator THD with killed = KILL_QUERY_HARD right
after a wsrep_abort_granted() whose granted side is logged as
granted 0:LOCAL : NULL, i.e. a background THD without a query.

Reproduction

2-node Galera cluster (docker, mariadb:12.3.3 and mariadb:11.4.13):

• Node 1 runs a constant UPDATE/DELETE load on an InnoDB table with an
  indexed virtual column (so that purge goes through open_purge_table()).
• Node 2 repeatedly executes DROP TABLE / CREATE TABLE on a second
  table with an indexed virtual column.

Within about 40 s node 1's purge coordinator has killed = KILL_QUERY_HARD
and its memory/sql/THD::main_mem_root in performance_schema grows by
~20 MB/s (12.3.3: 854 MiB after 46 s; 11.4.13: 755 MiB after 49 s) until
the container is OOM-killed.

With this patch (built from this branch on 11.4, and the same patch on
12.3.3) the same workload logs MDL conflict, waiting for system thread
(with wsrep_debug=1), the DDL completes, and main_mem_root stays at
0.16 MiB for the full 170 s run on both versions. The reproducer script
(local-purge-galera-probe.py) is attached to MDEV-41450.

Control runs with the same workload on unpatched official images:

• mariadb:10.6 (10.6.28): the TOI thread waits (BF thread waiting for BACKUP), purge is never killed, no leak.
• mariadb:10.11.16 (MDEV-38218 but not yet MDEV-39261): the purge
  coordinator is killed (MDL conflict-> kill in the log) and stays killed,
  but without the retry loop there is no leak; instead index entries of the
  virtual index on that node remain unpurged until restart. Part 1 of this
  fix also covers that case.

Testing

• Built on top of 11.4 (RelWithDebInfo, -DWITH_WSREP=ON, no new
  warnings in the touched files) and on 12.3.3; runtime verification with
  the reproducer on both builds, see above.
• No MTR test is included: the failure needs a Galera TOI DDL to arrive
  while the purge coordinator holds the MDL on the same table. I did not
  find a debug sync point on the purge side that lets this be scheduled
  deterministically from a .test file. Suggestions welcome.

Release Notes

Fixed an unbounded memory leak and purge stall on Galera nodes: a TOI DDL
on a table with an indexed virtual column could BF-abort the InnoDB purge
coordinator, after which purge stopped working and mariadbd memory grew
until the process was killed.

How can this PR be tested?

See "Reproduction" above; run the attached reproducer against an unpatched
and a patched build.

PR quality check

• I checked the CODING_STANDARDS.md file and my PR conforms to this where appropriate.
• For any trivial modifications to the PR, I am ok with the reviewer making the changes themselves.

Licensing

I am contributing this under the terms of the BSD-new (3-clause BSD) license.

[CLAassistant]

All committers have signed the CLA.

[gkodinov]

LGTM. I'd backport to 10.11, but this is for the final reviewer to decide on.

Please stand by for the final review.