Skip to content

test(node): Assert absence of PII attributes in OpenAI integration tests - #24031

Open
itz-puneet wants to merge 1 commit into
getsentry:developfrom
itz-puneet:test/ai-pii-absence-assertions
Open

test(node): Assert absence of PII attributes in OpenAI integration tests#24031
itz-puneet wants to merge 1 commit into
getsentry:developfrom
itz-puneet:test/ai-pii-absence-assertions

Conversation

@itz-puneet

Copy link
Copy Markdown

Closes #19801

Before submitting a pull request, please take a look at our
Contributing guidelines and verify:

  • If you've added code that should be tested, please add tests.
  • Ensure your code lints and the test suite passes (yarn lint) & (yarn test). — see note below
  • Link an issue if there is one related to your pull request.

What

The OpenAI integration tests assert which attributes are present when sendDefaultPii is disabled, but never assert that the PII-gated attributes are absent. As #19801 notes, a regression that leaked prompt or response content onto spans would pass the existing suite.

This adds an explicit absence check to both instrument.mjs (recording disabled) scenarios — chat and embeddings — covering the four content-bearing attributes that the paired instrument-with-pii.mjs tests assert are present:

  • gen_ai.input_messages
  • gen_ai.system_instructions
  • gen_ai.response.text
  • gen_ai.embeddings.input

The check iterates every span in the envelope rather than only the spans asserted individually above, so content leaking onto an uninspected span is also caught. for (const span of container.items) already appears 4 times in this file, so the shape matches the surrounding style.

Scope

Only the openai suite here, to keep the diff reviewable. The same gap exists in anthropic, langchain and google-genai — happy to follow up in a separate PR if you'd like the same treatment there.

I did not use the flare-redact approach suggested in the issue comments, since AGENTS.md says not to add dependencies unless explicitly asked. These assertions use vitest built-ins only.

Testing note

I was unable to run yarn install, yarn lint, yarn test or yarn format locally — my environment enforces an npm registry publish-date cutoff that blocks resolving this repo's dependency tree. The change is additive test assertions with no production code touched, and I verified the constants used are already imported in this file, but CI is the first real execution of these assertions. Please flag anything it turns up and I'll fix promptly.

The OpenAI integration tests assert which attributes are present when
`sendDefaultPii` is disabled, but never assert that the PII-gated
attributes are absent. A regression which leaked prompt or response
content onto spans would therefore pass the existing suite.

Adds an explicit check over every span in the envelope, for both
`instrument.mjs` (recording disabled) scenarios, covering the four
content-bearing attributes which the paired `instrument-with-pii.mjs`
tests assert are present:

-   `gen_ai.input_messages`
-   `gen_ai.system_instructions`
-   `gen_ai.response.text`
-   `gen_ai.embeddings.input`

Iterating the whole envelope rather than the individually inspected
spans also catches content leaking onto spans the tests do not assert
against directly.

The same gap exists in the anthropic, langchain and google-genai
suites; happy to follow up in a separate PR.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@itz-puneet
itz-puneet requested a review from a team as a code owner September 3, 2026 21:30
@itz-puneet
itz-puneet requested review from isaacs and mydea and a lite review from Copilot and removed request for a team September 3, 2026 21:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is a targeted, consistent test hardening that directly addresses the documented gap without introducing behavioral or structural risk.

Pull request overview

Strengthens the OpenAI Node integration test suite to prevent regressions where PII-bearing GenAI content attributes are accidentally attached to spans while sendDefaultPii is disabled, aligning coverage with the concerns raised in #19801.

Changes:

  • Adds explicit assertions that content-bearing GenAI attributes are absent when sendDefaultPii is disabled.
  • Applies the absence checks across all spans in the captured envelope for both chat and embeddings scenarios to catch leaks on uninspected spans.
File summaries
File Description
dev-packages/node-integration-tests/suites/tracing/openai/test.ts Adds envelope-wide assertions ensuring PII-gated GenAI attributes are undefined when sendDefaultPii is disabled (chat + embeddings).
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@RulaKhaled
RulaKhaled self-requested a review September 4, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AI integration tests don't assert absence of PII when sendDefaultPii is false

2 participants