Skip to content

[feat][PES][jobhistory] add task-diagnosis api to classify failed tasks into linkis or underlying cause - #1095

Merged
Alexkun merged 3 commits into
dev-2.1.2-webankfrom
dev-2.1.2-task-diagnosis
Aug 27, 2026
Merged

[feat][PES][jobhistory] add task-diagnosis api to classify failed tasks into linkis or underlying cause#1095
Alexkun merged 3 commits into
dev-2.1.2-webankfrom
dev-2.1.2-task-diagnosis

Conversation

@v-kkhuang

Copy link
Copy Markdown

What is the purpose of the change

Background/Problem:
When a Linkis task fails, the jobhistory record only stores the raw error code, error desc and metrics. It does not indicate whether the failure originated from the Linkis framework (scheduling, entrance, engineconn) or from the underlying compute engine/resource (YARN container killed, OOM, file/permission errors). Ops engineers must manually inspect logs to root-cause each failure, which is slow and hurts troubleshooting efficiency.

Purpose of Change:
To address this problem, this PR adds a task-diagnosis RESTful API in the jobhistory service that automatically classifies a failed task into a diagnosis type (LINKIS / UNDERLYING / UNKNOWN) and the failed stage (Scheduling / Execution), driven by configurable error-code ranges and keyword rules parsed from the task's error code, error desc and metrics.

Value/Impact:
After the change, ops and upper applications can call a single API to get a structured diagnosis result, distinguishing framework issues from engine/resource issues. This speeds up ticket routing and enables AI-driven ops diagnosis, while the feature is off by default and degrades gracefully on exception.

Related issues/PRs

Related issues: close #1094
Related pr:none

Brief change log

  • Add TaskDiagnosisService interface and TaskDiagnosisServiceImpl to classify failed tasks by diagnosis type and failed stage
  • Add DiagnosisResult entity carrying diagnosisType, failedStage, errorCode, errorDesc and engine/yarn context
  • Add TaskDiagnosisConfiguration with a hot-reloadable enable switch and configurable error-code/keyword rule sets
  • Expose the GET /jobhistory/task-diagnosis endpoint in QueryRestfulApi with permission check and exception-safe fallback

Checklist

  • I have read the Contributing Guidelines on pull requests.
  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible
  • If this is a code change: I have written unit tests to fully verify the new behavior.

@v-kkhuang v-kkhuang added the enhancement New feature or request label Aug 21, 2026

@Alexkun Alexkun left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@Alexkun
Alexkun merged commit 7fb0826 into dev-2.1.2-webank Aug 27, 2026
14 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants