Skip to content

[FLINK-40416][runtime] Add built-in text AI functions - #4512

Draft
haruki-830 wants to merge 1 commit into
apache:masterfrom
haruki-830:feature/text-ai-functions
Draft

[FLINK-40416][runtime] Add built-in text AI functions#4512
haruki-830 wants to merge 1 commit into
apache:masterfrom
haruki-830:feature/text-ai-functions

Conversation

@haruki-830

Copy link
Copy Markdown
Contributor

What is the purpose of this pull request?

This PR implements FLINK-40416 by adding six built-in text AI functions to Pipeline transforms:

  • AI_CLASSIFY
  • AI_TRANSLATE
  • AI_SUMMARIZE
  • AI_SENTIMENT
  • AI_EXTRACT
  • AI_MASK

Brief change log

  • Add Calcite definitions and Janino/runtime dispatch for the six text AI functions.
  • Use English built-in prompt templates and return structured responses as VARIANT.
  • Define NULL semantics: NULL input or a NULL model response returns NULL.
  • Raise a function-specific error for a non-NULL invalid JSON response.
  • Validate model references and text-generation/embedding capabilities.
  • Preserve UDF-first behavior when a UDF has the same name as a built-in AI function.
  • Extend the Dummy model, E2E coverage, and English/Chinese documentation.

Verifying this change

This change added tests and was verified as follows:

  • Runtime parser, Janino, and function tests passed on both Flink 1.20 and Flink 2 profiles: 42 tests per profile.
  • Composer MiniCluster tests with the Dummy model passed on both profiles: 4 tests per profile.
  • Added regression coverage for same-name UDF precedence.
  • Updated AiFunctionE2eITCase; the Docker-based Pipeline E2E test was not run locally.
  • Spotless, Checkstyle, and diff checks passed.

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? English and Chinese docs

Was generative AI tooling used to co-author this PR?
  • Yes (OpenAI Codex)

Generated-by: OpenAI Codex

@github-actions github-actions Bot added docs Improvements or additions to documentation composer runtime e2e-tests add-ons labels Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

add-ons composer docs Improvements or additions to documentation e2e-tests runtime

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant