API Contract: Automate Schema Updates by Prajna1999 · Pull Request #892 · ProjectTech4DevAI/kaapi-backend

Prajna1999 · 2026-05-26T10:33:01Z

Issue: #885

Summary

Currently breaking changes stay latent unless specified explicitly by the developer. This PR adds oasdiff a CI tool to automate the process of seeing diffs across base and main OpenAPI schemas. If the breaking changes are deliberate the developer can add breaking-change-approved tag to pass the test. It's defaulted to fail on making breaking changes. Also changelog of API is shown.

Checklist

Before submitting a pull request, please ensure that you mark these task.

Ran fastapi run --reload app/main.py or docker compose up in the repository root and test.
If you've fixed a bug or added code that is tested and has test cases.

Notes

JSON output — markdown-from-oasdiff was unstructured prose; JSON gives us per-row fields so we can format consistently.
jq → markdown table — readers scan rows (Method / Path / Change) faster than they parse a bulleted dump.
Severity dots replace badges — shields.io images add network latency and visual weight for what is fundamentally a one-bit signal.
Red reserved for ERR-level only — painting non-breaking changes red trained reviewers to ignore the color; now red means"actually blocks merge."
jq 'length' for counts — the old bullet-grep depended on oasdiff's markdown style and silently miscounted when the format shifted; JSON arrays are unambiguous.

coderabbitai · 2026-05-26T10:33:08Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds a CLI to export the FastAPI OpenAPI schema, a GitHub Actions workflow that diffs base vs head schemas with oasdiff and posts a Markdown PR comment, enforces a merge gate for breaking changes, and updates LLM response models by adding test and disabling status.

Changes

OpenAPI Diffing and API Schema Validation

Layer / File(s)	Summary
OpenAPI Export Script `backend/scripts/kaapi_export_openapi.py`	New CLI script that loads the FastAPI app from `app.main`, exports the generated OpenAPI schema as YAML (default `openapi.yaml`) and prints output path/size.
CI Workflow: Trigger and Environment Setup `.github/workflows/api-breaking-changes.yaml`	Workflow trigger (pull_request and label changes), permissions, concurrency; checks out base/head refs, installs Python, and prepares `.env.test` files.
CI Workflow: Schema Export and Diff Generation `.github/workflows/api-breaking-changes.yaml`	Exports base/head OpenAPI (bootstraps export script into base if missing), runs `tufin/oasdiff` in a container twice to produce breaking and full changelog JSON, and coerces invalid JSON to empty arrays.
CI Workflow: Report Comment and Merge Gate `.github/workflows/api-breaking-changes.yaml`	Renders JSON diffs into Markdown tables, appends to `$GITHUB_STEP_SUMMARY`, posts/updates a sticky PR comment, and fails when breaking changes exist without `breaking-change-approved`.
LLM Response Model Schema Updates `backend/app/models/llm/response.py`	Adds optional `test: str \| None = None` to `LLMJobImmediatePublic` and comments out the `status: str` field in `LLMJobPublic`.

Sequence Diagram

sequenceDiagram
  participant Actions as GitHub Actions
  participant ExportScript as kaapi_export_openapi.py
  participant OASDiff as tufin/oasdiff
  participant Formatter as WorkflowFormatter
  participant StickyComment as PR Sticky Comment
  Actions->>ExportScript: export base openapi.yaml (bootstrap script if missing)
  Actions->>ExportScript: export head openapi.yaml
  Actions->>OASDiff: run breaking diff (base vs head)
  Actions->>OASDiff: run full changelog diff (base vs head)
  OASDiff->>Actions: breaking.json, changelog.json
  Actions->>Formatter: render JSON -> Markdown tables (with counts)
  Actions->>StickyComment: post/update sticky PR comment and write step summary
  alt breaking changes detected AND label missing
    StickyComment->>Actions: workflow fails (merge gate)
  else approved or no breaking changes
    StickyComment->>Actions: workflow passes
  end

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

Suggested labels

enhancement

Suggested reviewers

kartpop
AkhileshNegi

"🐰 I hopped through YAML, code, and test,
Exported schemas, then ran the rest,
I flagged the breaks with a careful cheer,
Pushed a sticky note so reviewers hear,
Fields nudged and gates in place—hooray, good jest!"

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'API Contract: Automate Schema Updates' clearly and specifically summarizes the main change: automating OpenAPI schema diffing and validation in CI.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch chore/diffs-in-api-contract

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

sentry · 2026-05-26T10:43:25Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…contract' into chore/diffs-in-api-contract

Prajna1999 · 2026-05-26T10:50:41Z

restart PR

Prajna1999 · 2026-05-26T11:26:37Z

test the oasdiff workflow queued forever

github-actions · 2026-05-28T04:32:04Z

OpenAPI changes ⚪ No API surface changes

Note

This PR does not modify the API contract.

_{main ↔ 84396343 · generated by oasdiff}

Prajna1999 · 2026-06-01T11:45:39Z

CI will auto enable after the tests (removing mandatory params removed)

…contract' into chore/diffs-in-api-contract

Ayush8923

Added the few comments.

Ayush8923 · 2026-06-01T14:28:58Z

+
+on:
+  pull_request:
+    branches: ["main"]


Not need this branches filter here, run on PRs targeting any base branch.

let's run it on main, since that's where the action is i.e to be deployed in staging; other base branches (that eventually will me merged to main so redundant) or prod. We can add other branches in future if we have separate releases for staging and prod

Mostly to reduce github actions quota

Ayush8923 · 2026-06-01T14:40:46Z

+        # get_settings() loads `../.env.test` relative to the backend cwd when
+        # ENVIRONMENT=testing, so the file must sit at the repo root of each checkout.


Remove these unwanted comments.

Ayush8923 · 2026-06-01T14:43:07Z

+            cp "$GITHUB_WORKSPACE/head/backend/scripts/export_openapi.py" scripts/
+            uv run python scripts/export_openapi.py "$GITHUB_WORKSPACE/base-openapi.yaml"


Can we change the file name from export_openapi and use similar related to kaapi?

Co-authored-by: Ayush <80516839+Ayush8923@users.noreply.github.com>

coderabbitai

Actionable comments posted: 2

🧹 Nitpick comments (1)

.github/workflows/api-breaking-changes.yaml (1)
173-177: ⚡ Quick win

Sticky comment won't post if an earlier step fails.

This step uses the default if: success(). If the diff/build steps fail (e.g., the OASDIFF_IMAGE issue above), the comment is silently skipped even though "Build PR comment body" runs under if: always(). Add if: always() so the report still posts.
Proposed fix
       - name: Post sticky PR comment
+        if: always()
         uses: marocchino/sticky-pull-request-comment@v3
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/api-breaking-changes.yaml around lines 173 - 177, The
"Post sticky PR comment" job step uses the default conditional and is skipped if
earlier steps fail; update the step titled "Post sticky PR comment" that uses
marocchino/sticky-pull-request-comment@v3 (with header: oasdiff and path:
comment.md) to include "if: always()" so it runs regardless of prior step
failure and ensures the built comment is posted.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/api-breaking-changes.yaml:
- Around line 6-7: The branches filter contains an invalid pattern ("-- '**'")
that prevents the workflow from triggering; edit the YAML under the branches key
to either remove the branches filter entirely for the pull_request trigger
(since pull_request without branches matches all targets) or correct the list
entry to a single hyphen and proper glob ('- "**"') so it reads branches:
followed by - "**"; update the entry that currently reads "-- '**'" to one of
these two fixes in the workflow file.
- Around line 78-90: The workflow step fails because the shell function run_diff
uses the undefined variable OASDIFF_IMAGE; define OASDIFF_IMAGE at the job or
step level before run_diff is invoked (e.g., export or env entry) and set it to
a pinned oasdiff image tag or digest (not latest) for reproducibility; update
the workflow to ensure OASDIFF_IMAGE is available in the run environment so
run_diff (and its calls run_diff changelog changelog.json / run_diff breaking
breaking.json) can successfully execute and produce the JSON outputs.

---

Nitpick comments:
In @.github/workflows/api-breaking-changes.yaml:
- Around line 173-177: The "Post sticky PR comment" job step uses the default
conditional and is skipped if earlier steps fail; update the step titled "Post
sticky PR comment" that uses marocchino/sticky-pull-request-comment@v3 (with
header: oasdiff and path: comment.md) to include "if: always()" so it runs
regardless of prior step failure and ensures the built comment is posted.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 17f0621f-97de-43e8-9a8d-a2a382fe5f6a

📥 Commits

Reviewing files that changed from the base of the PR and between 19b3b28 and 6dea5a7.

📒 Files selected for processing (3)

.github/workflows/api-breaking-changes.yaml
backend/app/models/llm/response.py
backend/scripts/kaapi_export_openapi.py

vprashrex · 2026-05-28T11:30:53Z

 class CallbackResponse(SQLModel):
-    status: str
-    response_id: str
+    # status: str


make sure to uncomment this before merging

Ayush8923 · 2026-06-02T11:59:36Z


    job_id: UUID
-    status: str
+    # status: str


Revert these changes.

Ayush8923 · 2026-06-02T11:59:41Z

    job_id: UUID
    status: str
    message: str
+    test: str | None = None


…contract' into chore/diffs-in-api-contract

poc: test oasfiff API changelog

642dcc4

Prajna1999 self-assigned this May 26, 2026

Merge branch 'main' into chore/diffs-in-api-contract

bb5ebdb

Prajna1999 added 3 commits May 26, 2026 16:17

fix: in CI generate and compare diffs on the fly

192650f

fix: in CI generate and compare diffs on the fly

954c594

Merge remote-tracking branch 'refs/remotes/origin/chore/diffs-in-api-…

69e3590

…contract' into chore/diffs-in-api-contract

Prajna1999 closed this May 26, 2026

Prajna1999 reopened this May 26, 2026

Prajna1999 added 5 commits May 27, 2026 22:49

check CI

81e2cb4

fix: base cwd for .env.example

d61c828

fix: path for Diff breaking changes step

9eb4ffe

test: remove job_id from LLMJobImmediatePublic

e352cf7

chore: better siren and labeling to bypass

0b092d7

Prajna1999 added 2 commits May 28, 2026 10:06

fix: conditional failure of breaking change

b90bb97

chore: prettify md files

c40815b

Prajna1999 force-pushed the chore/diffs-in-api-contract branch from 17097d4 to c40815b Compare May 28, 2026 05:28

Prajna1999 added 2 commits May 28, 2026 11:00

test

8264a02

garnishing sticky message in the PR comment

e2fcd4d

Prajna1999 added the breaking-change-approved Reviewer-acknowledged API breaking change label May 28, 2026

chore: test with non_breaking changes

f385144

Prajna1999 removed the breaking-change-approved Reviewer-acknowledged API breaking change label May 28, 2026

chore: revert all schema changes

5fc4c82

Prajna1999 added the breaking-change-approved Reviewer-acknowledged API breaking change label May 28, 2026

test: breaking changes

f6a4a0f

Prajna1999 removed the breaking-change-approved Reviewer-acknowledged API breaking change label May 28, 2026

chore: non breaking gree

984c4cc

Prajna1999 moved this from In Review to To Do in Kaapi-dev Jun 1, 2026

Prajna1999 removed this from Kaapi-dev Jun 1, 2026

Ayush8923 and others added 3 commits June 1, 2026 19:45

Merge branch 'main' into chore/diffs-in-api-contract

19b3b28

chore: remove test breaking changes

985e738

Merge remote-tracking branch 'refs/remotes/origin/chore/diffs-in-api-…

257b1cc

…contract' into chore/diffs-in-api-contract

Ayush8923 requested changes Jun 1, 2026

View reviewed changes

Prajna1999 and others added 6 commits June 1, 2026 20:43

Update .github/workflows/oasdiff.yaml

9be2496

Co-authored-by: Ayush <80516839+Ayush8923@users.noreply.github.com>

Update .github/workflows/oasdiff.yaml

9020324

Co-authored-by: Ayush <80516839+Ayush8923@users.noreply.github.com>

Update .github/workflows/oasdiff.yaml

7653cc8

Co-authored-by: Ayush <80516839+Ayush8923@users.noreply.github.com>

Update .github/workflows/oasdiff.yaml

51350dd

Co-authored-by: Ayush <80516839+Ayush8923@users.noreply.github.com>

Update .github/workflows/oasdiff.yaml

bceb063

Co-authored-by: Ayush <80516839+Ayush8923@users.noreply.github.com>

fix: comments

6dea5a7

coderabbitai Bot reviewed Jun 2, 2026

View reviewed changes

Comment thread .github/workflows/api-breaking-changes.yaml Outdated

Comment thread .github/workflows/api-breaking-changes.yaml

fix: oasdiff undefined

d1bfa4b

Prajna1999 removed the breaking-change-approved Reviewer-acknowledged API breaking change label Jun 2, 2026

Prajna1999 added 2 commits June 2, 2026 09:33

fix: workflow not triggering

e321c02

fix: logic fix at generate comment

eb833cb

Prajna1999 added the breaking-change-approved Reviewer-acknowledged API breaking change label Jun 2, 2026

vprashrex approved these changes Jun 2, 2026

View reviewed changes

Merge branch 'main' into chore/diffs-in-api-contract

7c99e96

Ayush8923 requested changes Jun 2, 2026

View reviewed changes

Prajna1999 added 2 commits June 2, 2026 18:57

chore: remove test fields

2e87b88

Merge remote-tracking branch 'refs/remotes/origin/chore/diffs-in-api-…

c31cfe1

…contract' into chore/diffs-in-api-contract

Prajna1999 removed the breaking-change-approved Reviewer-acknowledged API breaking change label Jun 2, 2026

Ayush8923 approved these changes Jun 2, 2026

View reviewed changes

Comment thread backend/scripts/kaapi_export_openapi.py Outdated

Update backend/scripts/kaapi_export_openapi.py

6c49b2b

Prajna1999 merged commit 5aa1a25 into main Jun 2, 2026
4 checks passed

Prajna1999 deleted the chore/diffs-in-api-contract branch June 2, 2026 15:23

		# get_settings() loads `../.env.test` relative to the backend cwd when
		# ENVIRONMENT=testing, so the file must sit at the repo root of each checkout.

		cp "$GITHUB_WORKSPACE/head/backend/scripts/export_openapi.py" scripts/
		uv run python scripts/export_openapi.py "$GITHUB_WORKSPACE/base-openapi.yaml"

Conversation

Prajna1999 commented May 26, 2026 • edited by Ayush8923 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue: #885

Summary

Checklist

Notes

Uh oh!

coderabbitai Bot commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram

Estimated code review effort

Suggested labels

Suggested reviewers

❌ Failed checks (1 warning)

Uh oh!

sentry Bot commented May 26, 2026

Codecov Report

Uh oh!

Prajna1999 commented May 26, 2026

Uh oh!

Prajna1999 commented May 26, 2026

Uh oh!

github-actions Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

OpenAPI changes ⚪ No API surface changes

Uh oh!

Prajna1999 commented Jun 1, 2026

Uh oh!

Ayush8923 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Prajna1999 commented May 26, 2026 •

edited by Ayush8923

Loading

coderabbitai Bot commented May 26, 2026 •

edited

Loading

github-actions Bot commented May 28, 2026 •

edited

Loading