test(node): Assert absence of PII attributes in OpenAI integration tests - #24031
Open
itz-puneet wants to merge 1 commit into
Open
test(node): Assert absence of PII attributes in OpenAI integration tests#24031itz-puneet wants to merge 1 commit into
itz-puneet wants to merge 1 commit into
Conversation
The OpenAI integration tests assert which attributes are present when `sendDefaultPii` is disabled, but never assert that the PII-gated attributes are absent. A regression which leaked prompt or response content onto spans would therefore pass the existing suite. Adds an explicit check over every span in the envelope, for both `instrument.mjs` (recording disabled) scenarios, covering the four content-bearing attributes which the paired `instrument-with-pii.mjs` tests assert are present: - `gen_ai.input_messages` - `gen_ai.system_instructions` - `gen_ai.response.text` - `gen_ai.embeddings.input` Iterating the whole envelope rather than the individually inspected spans also catches content leaking onto spans the tests do not assert against directly. The same gap exists in the anthropic, langchain and google-genai suites; happy to follow up in a separate PR. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
itz-puneet
requested review from
isaacs and
mydea
and
a lite review from Copilot
and removed request for
a team
September 3, 2026 21:30
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The change is a targeted, consistent test hardening that directly addresses the documented gap without introducing behavioral or structural risk.
Pull request overview
Strengthens the OpenAI Node integration test suite to prevent regressions where PII-bearing GenAI content attributes are accidentally attached to spans while sendDefaultPii is disabled, aligning coverage with the concerns raised in #19801.
Changes:
- Adds explicit assertions that content-bearing GenAI attributes are absent when
sendDefaultPiiis disabled. - Applies the absence checks across all spans in the captured envelope for both chat and embeddings scenarios to catch leaks on uninspected spans.
File summaries
| File | Description |
|---|---|
| dev-packages/node-integration-tests/suites/tracing/openai/test.ts | Adds envelope-wide assertions ensuring PII-gated GenAI attributes are undefined when sendDefaultPii is disabled (chat + embeddings). |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
RulaKhaled
self-requested a review
September 4, 2026 16:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #19801
Before submitting a pull request, please take a look at our
Contributing guidelines and verify:
yarn lint) & (yarn test). — see note belowWhat
The OpenAI integration tests assert which attributes are present when
sendDefaultPiiis disabled, but never assert that the PII-gated attributes are absent. As #19801 notes, a regression that leaked prompt or response content onto spans would pass the existing suite.This adds an explicit absence check to both
instrument.mjs(recording disabled) scenarios — chat and embeddings — covering the four content-bearing attributes that the pairedinstrument-with-pii.mjstests assert are present:gen_ai.input_messagesgen_ai.system_instructionsgen_ai.response.textgen_ai.embeddings.inputThe check iterates every span in the envelope rather than only the spans asserted individually above, so content leaking onto an uninspected span is also caught.
for (const span of container.items)already appears 4 times in this file, so the shape matches the surrounding style.Scope
Only the
openaisuite here, to keep the diff reviewable. The same gap exists inanthropic,langchainandgoogle-genai— happy to follow up in a separate PR if you'd like the same treatment there.I did not use the
flare-redactapproach suggested in the issue comments, sinceAGENTS.mdsays not to add dependencies unless explicitly asked. These assertions use vitest built-ins only.Testing note
I was unable to run
yarn install,yarn lint,yarn testoryarn formatlocally — my environment enforces an npm registry publish-date cutoff that blocks resolving this repo's dependency tree. The change is additive test assertions with no production code touched, and I verified the constants used are already imported in this file, but CI is the first real execution of these assertions. Please flag anything it turns up and I'll fix promptly.