Skip to content

Skill audit: fix inaccurate statements, fill gaps, add checks - #107

Merged
yannrichet merged 1 commit into
mainfrom
claude/skill-audit-fixes
Nov 6, 2026
Merged

yannrichet merged 1 commit into
mainfrom
claude/skill-audit-fixes

Conversation

@yannrichet

Copy link
Copy Markdown
Member

Pull Request

Description

This is an audit of the Agent Skill (skills/fz/: SKILL.md, reference.md, code-wrapper.md, algorithm-wrapper.md) and of the plugin commands (commands/*.md) on main (ec97161, version 2.0). Each statement was checked against the code (defaults, environment variables, CLI options, installer, python:// helpers, output_expression functions, MCP tools). Where possible it was also checked by running fz 2.0: 22 scenarios, plus a real install of the official fz-Modelica wrapper from a local zip.

The existing static tests only check that names are present. The inaccurate statements below passed them.

Related Issues

Relates to #106 (fzd leaves an empty iterNNN_<timestamp> directory per iteration — a code defect found during the audit, not addressed here)

Type of Change

  • Documentation update (Agent Skill)
  • Test addition/modification

Changes Made

Inaccurate statements fixed

File Before Measured / in the code
SKILL.md, fzd example algorithm_options={"max_iterations": 50} for bfgs.py bfgs.py reads only max_iter, so the option was silently ignored: 100 iterations ran instead of 50 (with max_iter=2: 2 iterations)
code-wrapper.md, runner a non-zero exit marks the case "error" status failed (Command failed with exit code N)
SKILL.md, reference.md, code-wrapper.md out.txt is "overwritten by the captured stdout" log.txt, info.txt, history.txt, .fz_hash are rewritten by fz; out.txt/err.txt are shared with the captured streams, and the last writer wins (both orders measured)
same files retried "on another calculator" on another calculator when there is one, otherwise on the same one, up to FZ_MAX_RETRIES
reference.md resume after Ctrl+C with cache://results_dir into the same results_dir that never hits (as SKILL.md already says); cache://_ is needed
algorithm-wrapper.md #title/#author "shown when listing algorithms" parsed but never used; list_installed_algorithms() returns name, file, type, global
code-wrapper.md, algorithm-wrapper.md repository named fz-<code> "(lowercase)" official repositories keep the code's case (fz-Modelica, fz-Moret, fz-PSO)
code-wrapper.md "fz > 1.2" "fz ≥ 2.0"

Gaps filled

  • Example algorithms nsga2.py (multi-objective, used with a list of output_expression) and montecarlo_uniform.R.
  • Template repositories fz-Model, fz-Algorithm, fz-AlgorithmR.
  • fzo's _output_error column and fzd's algorithm key.
  • python://grep(pattern, path, group=None, all=False, cast=True).
  • FZ_SSH_KEEPALIVE default (300 s).

Checks added to tests/test_skill_static.py

  • Every algorithm in examples/algorithms/ is mentioned in the skill.
  • The algorithm_options keys used in the skill's examples exist in the #options header of the algorithm they name.

Both fail on the previous skill (nsga2.py missing; max_iterations not declared by bfgs.py) and pass after the fixes.

NEWS

One entry in the 2.0 "Claude Code plugin and Agent Skill" section, since the 2.0 tag does not exist yet.

Testing Performed

  • tests/test_skill_static.py, test_docs_consistency.py, test_readme_structure.py, test_docs_python_blocks.py: 23 passed.
  • Behaviour verified by running fz 2.0, all consistent with the corrected text:
    • template syntax, fzc naming and renaming, fzo on a parent directory vs. a glob;
    • sh:// argument appending, statuses, reserved names;
    • cache://results vs. cache://_;
    • ValueError cases, callbacks, CLI exit code and the short form of --input_variables;
    • fzd return keys, layout, function model and calculator auto-discovery;
    • fz-Modelica install: Modelica.json, an executable Modelica.sh and localhost_Modelica.json, with --check passed.

Not verified

  • The two worked examples (skills/examples/*.md, Modelica + brent and OpenFOAM) were not run: omc and OpenFOAM are not available.
  • ssh://, slurm://, funz:// and fz-mcp were checked in the code only.
  • fz install model <name> downloads were blocked by the environment's proxy (HTTP 403 on GitHub archives); the install was tested from a local zip.

Generated by Claude Code

Each statement of skills/fz/ was checked against the code and, where possible,
by running fz 2.0 (22 scenarios, plus a real fz-Modelica install).

Inaccurate statements fixed:
- SKILL.md fzd example: algorithm_options {"max_iterations": 50} for bfgs.py,
  which reads only max_iter: the option was silently ignored (measured: 100
  iterations instead of the requested number).
- code-wrapper.md: a runner exiting non-zero gives status "failed", not
  "error".
- Reserved names (SKILL, reference, code-wrapper): log.txt, info.txt,
  history.txt and .fz_hash are rewritten by fz; out.txt/err.txt are shared
  with the captured stdout/stderr, so the last writer wins (measured both
  orders), rather than "overwritten by stdout".
- Retries: on another calculator when there is one, else on the same, up to
  FZ_MAX_RETRIES.
- reference.md: resuming after Ctrl+C into the same results_dir needs
  cache://_ (cache://<results_dir> never hits there, as SKILL.md says).
- algorithm-wrapper.md: #title/#author are parsed but not displayed.
- Official repository names keep the code's case (fz-Modelica, fz-PSO), not
  lowercase; "fz > 1.2" -> "fz >= 2.0".

Gaps filled: nsga2.py and montecarlo_uniform.R; fz-Model, fz-Algorithm and
fz-AlgorithmR templates; fzo's _output_error column; fzd's "algorithm" key;
python:// grep(group=, cast=); FZ_SSH_KEEPALIVE default (300).

tests/test_skill_static.py: every example algorithm is mentioned in the
skill; algorithm_options keys used in skill examples exist in the named
algorithm's #options header. Both fail on the previous skill.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RjXvVRjM6aMvKvvSFoxpZ6
@yannrichet
yannrichet merged commit 4ed796e into main Nov 6, 2026
40 checks passed
yannrichet pushed a commit to Funz/fz.github.io that referenced this pull request Nov 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants