Repository navigation
fix(rf-desktop): working fixture locators, locator check, less hook overhead; CI evals on GLM 5.3 Flash - #4
Merged
Conversation
Adds the fix-desktop-eval-turn-overhead change: the sut-platynui README gives locators that don't work on a real desktop (checked on Windows 11 with PlatynUI 0.12.0.dev330), keywords_resolve fails suites that wrap library keywords in user keywords, and rf-core hooks add turns the baseline arm never pays. Ticks openrouter-ci-eval-tier tasks 5.2, 5.3, 6.1 and 6.2 with the runs that verified them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
…verhead - sut-platynui README: @id locators (verified on Windows 11), language-independent window - keywords_resolve: min_calls counts library calls reached through user keywords - new custom.platynui:locators_ok check, gating on the narrow task and in the partial fallback (which now takes a list of checks) - rf-core: formatting drift opt-in (RF_AGENTSKILLS_FORMAT_CHECK); Stop reminder skipped once the suite ran after the last write - rf-platynui: robotcode doc keyword line in Library Import - scripts/desktop-live-check.py and docs/ci/desktop-live-check.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
Skill Evaluation: rf-mobile (narrow, treatment, N=3, z-ai/glm-5.3-flash)Model: OpenRouter spend
Gaterf-skill-eval reportRuns: 3 · pass: 3 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.02 Pass rate = runs whose gate result is Tasks
Skills
Skipped / errored checksNone. Incomplete runsNone. Process checks (treatment arm, not in deltas)
Adversarial tier (reported only, never gates)None. |
Skill Evaluation: rf-desktop (narrow, treatment, N=3, z-ai/glm-5.3-flash)Model: OpenRouter spend
Gaterf-skill-eval reportRuns: 3 · pass: 3 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.04 Pass rate = runs whose gate result is Tasks
Skills
Skipped / errored checksNone. Incomplete runsNone. Process checks (treatment arm, not in deltas)
Adversarial tier (reported only, never gates)None. |
Skill Evaluation: rf-web (narrow, treatment, N=3, z-ai/glm-5.3-flash)Model: OpenRouter spend
Gaterf-skill-eval reportRuns: 6 · pass: 6 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.04 Pass rate = runs whose gate result is Tasks
Skills
Skipped / errored checksNone. Incomplete runsNone. Process checks (treatment arm, not in deltas)
Adversarial tier (reported only, never gates)None. |
Skill Evaluation: rf-api (narrow, treatment, N=3, z-ai/glm-5.3-flash)Model: OpenRouter spend
Gaterf-skill-eval reportRuns: 6 · pass: 6 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.06 · turn-limit stops: 1 Pass rate = runs whose gate result is Tasks
Skills
Skipped / errored checksNone. Incomplete runsNone. Process checks (treatment arm, not in deltas)
Adversarial tier (reported only, never gates)None. |
Skill Evaluation: rf-core (narrow, treatment, N=3, z-ai/glm-5.3-flash)Model: OpenRouter spend
Gaterf-skill-eval reportRuns: 36 · pass: 36 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.26 · turn-limit stops: 4 Pass rate = runs whose gate result is Tasks
Skills
Skipped / errored checksNone. Incomplete runsNone. Process checks (treatment arm, not in deltas)
Adversarial tier (reported only, never gates)None. |
Re-baseline after the fixture, grader and hook changes of this PR (N=6, both arms, all tiers, DeepSeek V4.1 Flash, turn factor 1.5). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
z-ai/glm-5.3-flash replaces deepseek/deepseek-v4.1-flash as the only OpenRouter model (output $0.078/M instead of $1.20/M): allow-list, list prices, workflow OPENROUTER_MODEL, tests and docs. Baselines move to baselines/openrouter/glm-5.3-flash/ with the next re-baseline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
Replaces the DeepSeek V4.1 Flash baselines: N=6, both arms, all tiers, turn factor 1.5. The run cost $5.86 of OpenRouter spend. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
manykarim
changed the title
… ci] Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
manykarim
marked this pull request as ready for review
November 13, 2026 18:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes why rf-desktop's skill arm scored below the baseline arm on the OpenRouter tier, removes hook output that cost every skill arm turns, and switches the CI eval model to GLM 5.3 Flash. OpenSpec change:
fix-desktop-eval-turn-overhead, 28 of 29 tasks done.rf-desktop and the grader
eval/fixtures/sut-platynui/README.mdgave locators that don't work on a real desktop. It now uses@Id(num1Button,plusButton,equalButton,CalculatorResults), verified withplatynui-clion Windows 11, and a window locator that works in both the English and German display language. This was the main defect.custom.platynui:locators_ok(platynui_locators): it rejects a plain@AutomationIdand buttons found only by their English@Name. It gates the narrow task, and the partial task's fallback now takes a list of checks.keywords_resolve:min_callscounts library calls reached through user keywords. Seven correct suites no longer fail on call count.scripts/desktop-live-check.pyanddocs/ci/desktop-live-check.mdrun the eval suites against the real Calculator on Windows, after screening them for unsafe code.rf-core hooks (every plugin's skill arm)
RF_AGENTSKILLS_FORMAT_CHECK).Skill:
rf-platynuinamesrobotcode doc keyword PlatynUI.BareMetalin its Library Import section.CI eval model:
z-ai/glm-5.3-flashreplacesdeepseek/deepseek-v4.1-flashas the only OpenRouter model. Output tokens cost $0.078 per million instead of $1.20. Baselines for all plugins are underbaselines/openrouter/glm-5.3-flash/, and the DeepSeek ones are removed.Results: rf-desktop
robocop format: runs in the skill arms dropped from 1.17 and 0.67 per session to 0.robotcode doc: barely used. Sessions still parse the libdoc spec withpython -c: about 4 per session on GLM, 6–11 on DeepSeek.Results: the GLM baselines (all plugins)
Pass rate without / with the skill:
GLM passes most tasks without the skill, so the CI gate still catches regressions but shows little of a skill's benefit outside rf-core. The Claude baselines (task 11.4) are what show that benefit.
Spend: the GLM re-baseline cost $5.86 of key spend; the DeepSeek one cost $10.14.
Known follow-ups (not in this PR)
no_undefined_keyword_callscounts keyword names in comments and[Documentation]as calls, a false positive in rf-web's adversarial task, in both arms.robotcode docdoesn't read Libdoc spec files: [ENHANCEMENT] robotcode CLI: letrobotcode docread Libdoc spec files (.json, .libspec) robotcodedev/robotcode#689.Test plan
tests/eval,tests,plugins/rf-core/tests,plugins/rf-desktop/testsopenspec validate --strict🤖 Generated with Claude Code
https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP