Skip to content

fix(gooddata-eval): set temperature=0 for all simulated users - #1735

Merged
myhoai merged 1 commit into
masterfrom
eval-simulated-user-temperature-0
Aug 13, 2026
Merged

fix(gooddata-eval): set temperature=0 for all simulated users#1735
myhoai merged 1 commit into
masterfrom
eval-simulated-user-temperature-0

Conversation

@myhoai

@myhoai myhoai commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Summary by CodeRabbit

  • Bug Fixes
    • Improved consistency and reproducibility of simulated AI-generated responses.
    • Clarification, alert, metric, KDA, and visualization responses now produce more deterministic results.

@myhoai
myhoai requested review from hkad98, lupko and pcerny as code owners August 13, 2026 07:04
@coderabbitai

coderabbitai Bot commented Aug 13, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 3ee34d80-b305-4d27-8637-0c807ca4bd0c

📥 Commits

Reviewing files that changed from the base of the PR and between d5e0fa1 and a97cba1.

📒 Files selected for processing (5)
  • packages/gooddata-eval/src/gooddata_eval/core/agentic/alert_skill.py
  • packages/gooddata-eval/src/gooddata_eval/core/agentic/conversation.py
  • packages/gooddata-eval/src/gooddata_eval/core/agentic/kda_skill.py
  • packages/gooddata-eval/src/gooddata_eval/core/agentic/metric_skill.py
  • packages/gooddata-eval/src/gooddata_eval/core/agentic/visualization.py

📝 Walkthrough

Walkthrough

All five simulated OpenAI request paths now set temperature to 0. This replaces 0.5 where previously specified and adds the parameter where it was absent.

Changes

Agentic response determinism

Layer / File(s) Summary
Zero-temperature simulated responses
packages/gooddata-eval/src/gooddata_eval/core/agentic/alert_skill.py, packages/gooddata-eval/src/gooddata_eval/core/agentic/conversation.py, packages/gooddata-eval/src/gooddata_eval/core/agentic/kda_skill.py, packages/gooddata-eval/src/gooddata_eval/core/agentic/metric_skill.py, packages/gooddata-eval/src/gooddata_eval/core/agentic/visualization.py
Simulated alert, clarification, KDA, metric, and visualization requests now use temperature=0.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Mergeability Score: ⚪ Minimal · up to a97cb

The change standardizes simulated-user temperature behavior in the evaluation code, with no actionable merge-blocking risk remaining beyond normal checks and review.

Suggested reviewers: lupko, pcerny, hkad98, jaceksan, frankhuynh

Poem

A rabbit tunes the prompts just right,
Zero heat for outputs bright.
Alert and metrics, clear and true,
KDA and visuals follow too.
No random hops—just steady flight!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: setting temperature=0 for all simulated users in gooddata-eval.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Aug 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.44%. Comparing base (8ad7eb3) to head (208f2e0).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1735      +/-   ##
==========================================
+ Coverage   79.42%   79.44%   +0.02%     
==========================================
  Files         272      272              
  Lines       18997    19012      +15     
==========================================
+ Hits        15088    15105      +17     
+ Misses       3909     3907       -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@myhoai
myhoai force-pushed the eval-simulated-user-temperature-0 branch from a97cba1 to 208f2e0 Compare August 13, 2026 07:44
@myhoai
myhoai merged commit 942b1e9 into master Aug 13, 2026
14 checks passed
@myhoai
myhoai deleted the eval-simulated-user-temperature-0 branch August 13, 2026 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants