Skip to content

bulk: fix deadlock in recursive directory expansion - #8318

Open
DmitryLitvintsev wants to merge 1 commit into
12.1from
8062/12.1
Open

DmitryLitvintsev wants to merge 1 commit into
12.1from
8062/12.1

Conversation

@DmitryLitvintsev

Copy link
Copy Markdown
Member

Motivation:

Issue #8062 reports that recursive bulk requests (e.g. DELETE with expandDirectories=ALL) on large directory trees stay in RUNNING state forever, with "Available permits for directory listing" dropping to 0. Increasing bulk.limits.dir-list-semaphore only delays the hang and, at very high values, exhausts the JVM memory.

A directory listing task holds a dir-list permit while iterating over the directory and, for each subdirectory found, blocks acquiring a permit for the child listing. Once the number of directories being listed concurrently reaches the number of permits, all permits are held by tasks waiting for permits and nothing progresses. Since the semaphore is shared by all requests, a single large request also blocks directory listing for every other request. A similar cycle exists for an initial directory target holding an in-flight permit while waiting for a dir-list permit.

Modification:

Replaced the dir-list semaphore with a new non-blocking BulkTaskThrottle, shared by all requests. Directory listing tasks are registered as running immediately (so that the transition to directory processing still waits for them) and queued in the throttle; a queued listing is launched as soon as a permit is available, and every released permit launches the next queued listing, regardless of which request it belongs to. Submission never blocks, so a listing task can queue its subdirectories while holding its own permit.

A listing cancelled while queued returns its permit without running. A listing rejected by the executor fails the request instead of silently skipping the subtree.

Added "Directory listings waiting for permit" to the service info output. Added unit tests for BulkTaskThrottle and for BulkRequestContainerJob expanding a directory tree wider than the number of listing permits (this test hangs without the fix), including cancellation during expansion.

Result:

Recursive bulk requests on large directory trees complete with the default bulk.limits.dir-list-semaphore, which now only limits the number of concurrent directory listings sent to PnfsManager.

Ticket: #8062
Acked-by: Tigran
Target: trunk
Request: 11.2, 12.0, 12.1
Require-book: no
Require-notes: yes

Motivation:
-----------
Issue #8062 reports that
recursive bulk requests (e.g. DELETE with expandDirectories=ALL) on
large directory trees stay in RUNNING state forever, with
"Available permits for directory listing" dropping to 0. Increasing
bulk.limits.dir-list-semaphore only delays the hang and, at very high
values, exhausts the JVM memory.

A directory listing task holds a dir-list permit while iterating over
the directory and, for each subdirectory found, blocks acquiring a
permit for the child listing. Once the number of directories being
listed concurrently reaches the number of permits, all permits are
held by tasks waiting for permits and nothing progresses. Since the
semaphore is shared by all requests, a single large request also
blocks directory listing for every other request. A similar cycle
exists for an initial directory target holding an in-flight permit
while waiting for a dir-list permit.

Modification:
-------------
Replaced the dir-list semaphore with a new non-blocking
BulkTaskThrottle, shared by all requests. Directory listing tasks are
registered as running immediately (so that the transition to
directory processing still waits for them) and queued in the
throttle; a queued listing is launched as soon as a permit is
available, and every released permit launches the next queued
listing, regardless of which request it belongs to. Submission never
blocks, so a listing task can queue its subdirectories while holding
its own permit.

A listing cancelled while queued returns its permit without running.
A listing rejected by the executor fails the request instead of
silently skipping the subtree.

Added "Directory listings waiting for permit" to the service info
output. Added unit tests for BulkTaskThrottle and for
BulkRequestContainerJob expanding a directory tree wider than the
number of listing permits (this test hangs without the fix), including
cancellation during expansion.

Result:
-------
Recursive bulk requests on large directory trees complete with the
default bulk.limits.dir-list-semaphore, which now only limits the
number of concurrent directory listings sent to PnfsManager.

Ticket: #8062
Acked-by: Tigran
Target: trunk
Request: 11.2, 12.0, 12.1
Require-book: no
Require-notes: yes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant