feat(search): route /api/es/raw & /api/es/search through phase-aware SearchAPI (#36396)

[fabrizzio-dotCMS]

Proposed Changes

Fixes #36396 · addresses the /api/es/raw phase-bypass finding in #35752 (and extends the same fix to /api/es/search)

Both backend Elasticsearch REST endpoints in ESContentResourcePortlet called deprecated, ES-hardcoded ContentletAPI methods that bypassed the phase-aware router introduced in #35609. They always read from Elasticsearch regardless of the active OpenSearch migration phase — could not validate Phase 2/3 read routing (caused the false TC-028 failure in QA #35752) and would break entirely once ES is decommissioned in Phase 3.

Endpoint	Was	Now
POST /api/es/raw	esapi.esSearchRaw() → hardcoded ES	phase-aware ContentletAPI.searchRaw() → SearchAPI
POST /api/es/search	esapi.esSearch() → hardcoded ES	phase-aware ContentletAPI.search() → SearchAPI

Both now observe whichever engine the active phase selects (ES in phases 0–1, OS in phases 2–3) and keep working after ES is decommissioned in Phase 3. The FIXME(OS-cutover) is gone.

/api/es/raw — neutral response (documented shape change)

Routes through searchRaw() and serializes the vendor-neutral ContentSearchResponse with the dotCMS default ObjectMapper (no ES wire format). This endpoint is a low-level query/aggregation API used via curl; the ES Search backend portlet UI uses /api/portlet/es-search/render, not this endpoint, so the shape change is safe.

Field	Notes
hits	object with hits[] (each: id, index, sourceAsMap, score, fields), totalHits, hasError
scrollId	scroll id, or null
tookMillis	query time in ms
aggregationTree	neutral aggregation tree, keyed by aggregation name

/api/es/search — phase-aware with full backward compatibility

Unlike /api/es/raw, /api/es/search has a live consumer: the dot-es-search Angular portlet (Results/Raw/Aggregations/Suggestions tabs) parses the ES-wire shape of the esresponse field. To route through the neutral SearchAPI without breaking that portlet or external clients, the ES-wire shape is preserved at the edge:

• Legacy ES-wire adapter (ESContentResourcePortlet#toLegacyEsJson) rebuilds esresponse from the neutral DTO: took, hits.total.value, hits.hits[]._id/._index/._score/._source, aggregations (ES-native typed keys e.g. sterms#content_types), and suggest. Isolated in the resource so the shared neutral DTO stays clean; removable at R7. The contentlets array is unchanged (built from Contentlet objects).
• Velocity back-compat accessors on ContentSearchResponse (getHits/getScrollId/getTookInMillis/getAggregations/getSuggest) and TotalHits (getValue) so $dotcontent.raw(...) / $ESContent templates using ES-style property syntax keep resolving. All @JsonIgnore or same-value merges → the /api/es/raw neutral JSON shape is unchanged (guarded by AggregationDomainTest).
• suggest is now a neutral field on ContentSearchResponse, populated from both ES (from(Suggest)) and OS (term/phrase/completion union) → the Suggestions tab has parity across engines.

Testing

• Integration (run locally, Postgres 5437 + ES 9207):
  • ESContentResourcePortletTest — 9/9 (incl. test_search_esresponse_preservesLegacyEsWireShape pinning the ES-wire esresponse contract)
  • ContentSearchToolTest — 9/9 (Velocity search()/raw() hit+timing accessors, and a legacy property-syntax test proving the compat accessors resolve)
• Unit: AggregationDomainTest — 12/12 (pins the neutral /api/es/raw JSON; confirms the compat getters + suggest field did not change it).
• ./mvnw compile -pl :dotcms-core — BUILD SUCCESS (Java 25); openapi.yaml regenerated for both endpoints.

Checklist

• /api/es/raw routes through phase-aware searchRaw(); neutral response documented
• /api/es/search routes through phase-aware search(); ES-wire esresponse preserved via adapter (no portlet/client breakage)
• Velocity back-compat accessors; neutral /api/es/raw JSON shape unchanged (guarded by test)
• suggest carried on the neutral DTO from both ES and OS
• FIXME(OS-cutover) removed on both paths; works in Phase 3 with ES decommissioned
• openapi.yaml regenerated and committed
• Integration + unit tests green locally

Part of the OpenSearch migration epic #35476. Builds on #35609.

🤖 Generated with Claude Code

[claude]

Claude finished @fabrizzio-dotCMS's task in 7m 33s —— View job

---

Rollback-Safety Analysis Complete

• Read docs/core/ROLLBACK_UNSAFE_CATEGORIES.md
• Diffed 88286804...c8f93986 against every category
• Checked for DB migrations / ES mapping changes — none found
• Verified the /api/es/search legacy-wire adapter (toLegacyEsJson) vs. /api/es/raw's lack of one
• Confirmed the ESContentTool#raw() viewtool return-type change (H-8) predates this PR (from feat(opensearch): extract vendor-neutral SearchAPI and phase-aware router #35609, unchanged here)
• Posted rollback-unsafe finding as a PR comment
• Added label AI: Not Safe To Rollback

Result: 🟡 MEDIUM — M-3 (REST API Contract Change)

POST /api/es/raw intentionally swaps its response body from the raw Elasticsearch wire JSON to the vendor-neutral ContentSearchResponse shape, with no legacy-wire adapter fallback (unlike the sibling /api/es/search endpoint, which does get one via toLegacyEsJson()). A rollback to N-1 flips external/curl clients back to the old shape with no warning. Full details, code pointers, and a suggested fix (extend the toLegacyEsJson() pattern to /api/es/raw behind an opt-in) are in the posted comment.

No CRITICAL/HIGH categories apply — no database migrations, ES mapping changes, or structural storage changes are present in this diff.

[github-actions]

🤖 dotBot Review (Bedrock)

Reviewed 8 file(s); 5 candidate(s) → 5 confirmed, 0 uncertain (unverified, kept for review).

Confirmed findings

• 🔴 Critical dotCMS/src/main/java/com/dotcms/rest/elasticsearch/ESContentResourcePortlet.java:138 — Clearing contentlets list after adding to response map causes empty array in response
  The code adds esresult.getContentlets() to the response map, then immediately clears the list. Since Java collections are reference-based, this clears the contentlets array in the response map, resulting in an empty contentlets array being returned to clients. The responseMap.put() stores a reference to the same list object that gets cleared, making API responses lose all contentlet data.
• 🟡 Medium dotCMS/src/main/java/com/dotcms/content/index/domain/ContentSearchResponse.java:239 — Unhandled OpenSearch suggest types may silently drop entries
  The osSuggestEntry method returns null for unknown suggest types without logging, which could lead to missing suggestions if new types are added. While current OpenSearch versions may only support Term, Phrase, and Completion, the lack of logging or error handling for unexpected types means developers wouldn't be alerted to new suggestion types introduced in future OpenSearch versions.
• 🟡 Medium dotCMS/src/main/java/com/dotcms/content/index/domain/ContentSearchResponse.java:272 — Potential NPE in SuggestOption.from due to unguarded option.getText()
  The method SuggestOption.from calls option.getText().string() without null checking getText(). While OpenSearch's API may enforce non-null text in valid suggestions, defensive coding is warranted as invalid/malformed responses could cause NPEs during deserialization. This matches the code seen in ContentSearchResponse.java line 272 where text assignment assumes non-null getText().
• 🟡 Medium dotCMS/src/main/java/com/dotcms/rest/elasticsearch/ESContentResourcePortlet.java:161 — Extra space in log message
  The log message at line 161 starts with a space before 'unable', resulting in ' unable JSON contentlet'. This creates a cosmetic inconsistency in log formatting. The message should begin without leading space for consistency with other log messages.
• 🟡 Medium dotcms-integration/src/test/java/com/dotcms/rest/elasticsearch/ESContentResourcePortletTest.java:283 — JSON query built via string concatenation with contentType.variable() may produce invalid syntax
  The test constructs a JSON query string via concatenation: "contentType:" + contentType.variable(). If contentType.variable() contains characters requiring JSON escaping (e.g. quotes, backslashes), this produces invalid JSON syntax. While test code, this creates brittle tests that would fail with certain content type variables. Should use a JSON serializer like ObjectMapper or parameterized builder to ensure proper escaping.

---

_{us.deepseek.r1-v1:0 · Run: #28637200530 · tokens: in: 62154 · out: 11617 · total: 73771 · calls: 16 · est. ~$0.147}

[claude]

Pull Request Unsafe to Rollback!!!

• Category: M-3 — REST / GraphQL / Headless API Contract Change
• Risk Level: 🟡 MEDIUM
• Why it's unsafe: The PR intentionally changes the JSON response shape of the public POST /api/es/raw endpoint from the raw Elasticsearch SearchResponse wire format to the vendor-neutral ContentSearchResponse shape (hits/scrollId/tookMillis/aggregationTree). The PR description explicitly calls this out ("⚠️ Response shape change (documented)"). Per M-3, any client (curl scripts, external integrations, or the "low-level query/aggregation API" callers the PR's own help-text update references) that adapts to N's new response shape will break if the app is rolled back to N-1, since N-1's /api/es/raw reverts to emitting the old ES SearchResponse.toString() structure. There is no migration task involved — the break is purely at the API-contract layer, matching M-3's definition exactly.
• Code that makes it unsafe: dotCMS/src/main/java/com/dotcms/rest/elasticsearch/ESContentResourcePortlet.java lines 293-297:

  final ContentSearchResponse searchResponse =
          esapi.searchRaw(esQuery, mode.showLive, user, mode.showLive);
  return responseResource.response(
          DotObjectMapperProvider.getInstance().getDefaultObjectMapper()
                  .writeValueAsString(searchResponse));

  This replaces the previous esapi.esSearchRaw(...).toString() call, which emitted the raw ES SearchResponse JSON. The new/old shapes are non-overlapping — hits alone changes from a bare ES hits object with _id/_source/_score fields to a neutral SearchHits object with id/sourceAsMap/score fields (see ContentSearchResponse.java record and the hitsToLegacyJson/aggregationsToLegacyJson adapter added for the sibling /api/es/search endpoint — no equivalent legacy-shape adapter exists for /api/es/raw).
• Alternative (if possible): Apply the same "legacy-wire adapter" pattern the PR already uses for /api/es/search's esresponse field (toLegacyEsJson() in the same file, lines ~435-513) to /api/es/raw as well — keep emitting the ES-wire JSON shape by default (rebuilt from the neutral ContentSearchResponse), and offer the new neutral shape behind an opt-in query parameter (e.g. ?shape=neutral) or a new endpoint/version. This lets /api/es/raw keep serving old clients unchanged after either a forward deploy or a rollback, consistent with M-3's guidance to avoid breaking deployed clients in the same release that changes the contract.

[github-actions]

🔴 [Critical] Jackson serialization conflict between value() and getValue() causes duplicate field

The TotalHits record has both a value() method (generated by the record) and a custom getValue() method. Jackson will serialize both as 'value' fields, causing a JsonMappingException due to duplicate property names. This breaks the API response serialization.

[fabrizzio-dotCMS]

Invalid — false positive.

value() (record component, annotated @JsonProperty("value")) and getValue() both map to the same logical property name value, so Jackson merges them into a single accessor — it does not emit two value fields nor throw a duplicate/conflicting-getter error.

Verified with the exact default ObjectMapper the endpoint uses: AggregationDomainTest#contentSearchResponse_jacksonSerializesNeutralShapeForEsRawEndpoint serializes ContentSearchResponse → SearchHits → TotalHits and asserts totalHits is present — 12/12 green (re-confirmed after getValue() was added). No JsonMappingException.

getValue() is required for Velocity backward compatibility ($dotcontent.raw().hits.totalHits.value, which was a public field on ES TotalHits). Removing it or adding @JsonIgnore would break that (or drop the value field entirely).

[github-actions]

🟠 [High] Uncaught JSON serialization exception leaks stack traces

The call to objectMapper.writeValueAsString(contentSearchResponse) at line 137 can throw JsonProcessingException, which is not caught in the try block. This would propagate as an unhandled exception, exposing internal stack traces via API responses. The endpoint needs explicit error handling around the serialization call to return controlled error responses.

[fabrizzio-dotCMS]

Invalid — false positive.

The writeValueAsString(...) call is inside a try { ... } catch (Exception e) block that returns responseResource.responseError(e.getMessage()). JsonProcessingException extends Exception, so it is caught and turned into a controlled error response carrying only the message — not an unhandled exception and not a stack trace. The same applies to /api/es/search (toLegacyEsJson() throws JSONException, caught by the method-level catch (Exception)).

(The referenced line 137 does not correspond to the writeValueAsString call in searchRaw() — likely a stale diff position.)

[claude]

test2

[github-actions]

🔴 [Critical] Clearing contentlets list after adding to response map causes empty array in response

The code adds esresult.getContentlets() to the response map, then immediately clears the list. Since Java collections are reference-based, this clears the contentlets array in the response map, resulting in an empty contentlets array being returned to clients. The responseMap.put() stores a reference to the same list object that gets cleared, making API responses lose all contentlet data.

[claude]

Pull Request Unsafe to Rollback!!!

• Category: M-3 — REST / GraphQL / Headless API Contract Change
• Risk Level: 🟡 MEDIUM
• Why it's unsafe: POST /api/es/raw changes its response body from the raw Elasticsearch SearchResponse wire JSON to the vendor-neutral ContentSearchResponse shape (hits/scrollId/tookMillis/aggregationTree). This is a documented, intentional shape change with no legacy-wire adapter for this endpoint (unlike its sibling /api/es/search, which does get a compat adapter — see below). Per M-3, this risk lives entirely outside the DB/ES layer: any client (curl scripts, external integrations, the "low-level query/aggregation API" consumers the PR description itself calls out) that adapts to N's new /api/es/raw shape will get the old ES wire shape back if the binary is rolled back to N-1 — a silent, breaking shape flip with no error thrown, no migration involved.
• Code that makes it unsafe: dotCMS/src/main/java/com/dotcms/rest/elasticsearch/ESContentResourcePortlet.java (searchRaw(), ~lines 285-297):

  final ContentSearchResponse searchResponse =
          esapi.searchRaw(esQuery, mode.showLive, user, mode.showLive);
  return responseResource.response(
          DotObjectMapperProvider.getInstance().getDefaultObjectMapper()
                  .writeValueAsString(searchResponse));

  This replaces the previous esapi.esSearchRaw(...).toString() call (raw ES SearchResponse JSON). There is no toLegacyEsJson()-style adapter applied here — that adapter exists only for the neighboring /api/es/search endpoint's esresponse field (same file, toLegacyEsJson() / hitsToLegacyJson() / aggregationsToLegacyJson(), ~lines 336-419), which intentionally preserves the ES-wire contract for its live Angular-portlet consumer. /api/es/raw gets no equivalent treatment, so its contract change is the one exposed to external/rollback risk.
• Alternative (if possible): Apply the same "legacy-wire adapter" pattern already built for /api/es/search to /api/es/raw — keep emitting the ES-wire JSON shape by default (rebuilt from the neutral ContentSearchResponse via the existing toLegacyEsJson() helper), and offer the new neutral shape behind an explicit opt-in (query parameter such as ?shape=neutral, an Accept header variant, or a new endpoint/version). That keeps /api/es/raw clients unaffected by a rollback to N-1, consistent with M-3's guidance to avoid changing a contract that a rolled-back binary can no longer serve.

Note: no database migrations, Elasticsearch mapping changes, or structural storage changes are present in this diff (confirmed via git diff — no runonce tasks, no ESMappingAPIImpl/putMapping touches), so no CRITICAL/HIGH categories apply. TotalHits.getValue() and the ContentSearchResponse Velocity back-compat accessors (getHits/getScrollId/getTookInMillis/getAggregations/getSuggest) are additive and @JsonIgnore'd where appropriate — they don't reopen H-8 (the ESContentTool#raw() return-type change to ContentSearchResponse predates this PR, from #35609, and is unchanged here).