Conversation
…port
Add a unified JSON degree format (the DegreeProgram model serialized to JSON)
plus tooling around it:
- `degree convert`: convert ai-landscape program JSON to the unified format —
maps category lists + picklists to requirements, flips their AND-of-OR
prerequisites into our PrereqExpr tree, defaults missing credits (3) with
warnings. Expands ai-landscape *cluster* pipeline files
(course_verifier/course_scraper.<program>.results) into one unified file per
program with collision-safe `<school>__<program>.unified.json` names.
- JSON as an input type alongside YAML (auto-detects + converts ai-landscape
shapes on load); extension dispatch across the degree subcommands.
- Prerequisites as a symmetric tagged structure ({"and"|"or":[...]}, bare
string = leaf) via serde on PrereqExpr.
- `tags` on Degree/Requirement/Course generalize ai-landscape's ai_program and
fixed category names.
- Metrics-rich report JSON (degree- and course-level stats + variations /
sample_type) and a school-level rollup (`--school`).
- `degree schema`: emit the unified-degree JSON Schema (src/assets/degree.schema.json).
- New DegreeParseError::JsonError so JSON inputs no longer report "YAML Parse Error".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ysis Run `degree analyze` as one OS process per file, JOBS at a time, each with a virtual-memory cap (ulimit -v) and a timeout. A pathological degree — e.g. a full-catalog scrape with thousands of courses — aborts itself under the cap instead of letting the OS OOM-killer take down the whole batch; the failure is logged and the run continues. Threads can't provide this isolation because they share one address space, so surviving an OOM requires separate processes. Usage: scripts/analyze-batch.sh <input-dir|glob> <metrics-dir> [jobs] [-- extra flags] Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…isolated per file A multi-file `degree analyze` now runs as a rolling pool of worker processes (default 8; `-j/--jobs N`), each analyzing one file in its own OS process. A pathological degree (e.g. a full-catalog scrape with thousands of courses) is contained to its own process: if it OOMs/crashes, the kernel kills only that child, the parent records it in <metrics-dir>/failures.log, and the rest carry on. Single-file, `--school`, `-j 1`, and worker-mode invocations run in-process with full per-degree output. The parent writes the index.csv header up front so concurrent workers only append rows (atomic under O_APPEND), avoiding a header race. Worker stdout/stderr is suppressed; the parent prints a progress line + summary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Carry each failed worker's exit status into the failures.log (one `path<TAB>status` line), so an OOM kill shows `signal: 9 (SIGKILL)` and is distinguishable from a non-zero exit. Extract the reap loop into `reap_finished` to cut nesting, add a `DEFAULT_ANALYZE_JOBS` constant, a `metrics_dir_or_default` helper, and a `WORKER_POLL` const. Document that `analyze_child_flags` must mirror every result-affecting flag in args.rs, and that the pool deliberately has no ulimit/timeout (unlike scripts/analyze-batch.sh). Add 10 tests: analyze_child_flags coverage incl. a clap round-trip, reap_finished, metrics_dir_or_default, write_index_csv_header, and a -j 1 in-process integration test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`degree trim` previously filtered to YAML only. It now accepts unified JSON (and raw ai-landscape JSON) via the existing degree-input filter, and writes the trimmed program back in the input's format: a `.json` input yields a trimmed `.json`, YAML stays YAML. Add `save_degree_auto` to mirror `load_degree_auto` (extension dispatch, path-carrying error) and route `trim_one` through it. Update the two non-YAML trim tests for the new degree-filter messages and add three integration tests: JSON round-trips to JSON, a mixed YAML+JSON batch keeps each format, and the overwrite guard covers JSON inputs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Findings from analyzing the converted cluster_outputs dump: two NuAnalytics engine root causes (eager C(N,k) combination materialization causing the 35 OOM/SIGKILL failures; `ENUMERABLE_CATEGORIES = ["major"]` collapsing every converted program to variations_run=1) plus ai-landscape data issues (ambiguous picklist [N] count-vs-credits yielding 316 impossible requirements, unparseable picklist tags, whole-catalog scrapes, missing course_hours, scraped/validated naming divergence). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two engine fixes that made the converted-catalog analysis usable: Bug A (OOM): RequirementResolver materialized every C(n,k) combination of a select pool. A pool like "choose 15 of 42" (~10^11) allocated tens of GB and the OS OOM-killed the worker — even for an otherwise tiny 78-course program. generate_combinations is now routed through bounded_combinations, which down-samples to MAX_MATERIALIZED_COMBINATIONS (2000) distinct combinations when C(n,k) exceeds the cap (combinations_exceed computes the bound without overflow; sample_combinations is deterministic). Peak memory on the worst catalog programs drops from >6 GB to ~25-120 MB. Bug B (variations_run=1): ENUMERABLE_CATEGORIES was ["major"], so every elective-category select was excluded from the plan space and converted programs produced a single plan with std_dev=0. It now includes "elective"; estimate_plan_count uses saturating multiplication since a capped pool can still push the nominal product past usize::MAX. American University CS goes from variations_run=1 to ~4760 with real metric spread. Add unit tests for combinations_exceed/sample_combinations/bounded_combinations (incl. k==0, k==n, determinism, cap) and for elective-category enumeration. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Output JSON now opens with the degree block. A new unified_value_to_string serializes converted files as degree, requirements, courses, conversion_warnings (nested objects keep serde_json's sorted, deterministic order); the unified report is laid out degree, analysis, requirements, selected_plans, courses. Each selected plan now carries its courses: credits, course_count, the critical path, and a term-by-term schedule mirroring the MCP analyze_degree shape — so a reader can see exactly which courses the shortest path contains. total_credits already exists on the Degree model and surfaces at the top (populated during the correction pass when the source omits it). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… tools
Add two MCP tools and make the existing degree tools accept the unified
(and raw ai-landscape) JSON format, not just YAML:
- parse_degree_auto: content-level format sniff (`{`/`[` -> JSON loader with
ai-landscape auto-convert; else YAML). validate/analyze/audit/trim/
course_detail now route through it, so unified/ai-landscape JSON content
(and the convert tool's cache:<hash> handle) work everywhere. validate
surfaces conversion_warnings.
- convert_degree: ai-landscape program JSON -> unified JSON + warnings; a
cluster file returns a bounded program inventory (or converts one via
the `program` selector). Caches the result for chaining by degree_id.
- get_degree_json_schema: returns the machine JSON Schema (the same
degree.schema.json the CLI emits).
Also fix the schema's `from` clause: it was an untyped object, so wildcard
pools weren't documented. Added a `fromClause` definition covering
courses/pattern/include/exclude/groups with wildcard examples ("CS:2500+",
"*:*") -- confirming the unified format supports the same wildcard
gen-ed/elective pools as YAML.
Verified via MCP stdio smoke (tools register; schema + convert respond;
convert->cache->validate chaining works). +12 unit tests; full suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…onvert/schema tools Version bump 0.4.0 -> 0.4.1 plus documentation for the additive features landed since 0.4.0: - CHANGELOG: 0.4.1 entry (unified JSON degree format, `degree convert` / `degree schema`, parallel/process-isolated `analyze -j`, `--school` rollup, JSON input for `trim`, `convert_degree` / `get_degree_json_schema` MCP tools, the bounded-combinations OOM fix and elective-enumeration fix). - Readme: version line + new feature bullets. - docs/degree.md: convert/schema command sections, `-j/--jobs` and `--school` options, a Parallel Analysis section, and a unified-JSON input-format note. - docs/mcp.md: `get_degree_json_schema` and `convert_degree` tool docs, and an input-formats note covering YAML/unified/ai-landscape JSON across the degree tools. No code changes; manifest verified via `cargo read-manifest`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CI's `cargo clippy --all-features -D warnings` (a newer clippy than last green) flagged `self.m2 += delta * delta2` as clippy::suboptimal_flops. Use the suggested fused `delta.mul_add(delta2, self.m2)` — one rounding instead of two, so slightly more accurate, and it clears the lint that was failing every PR. No behavioral change to the streaming statistics. Verified: `cargo clippy --all-targets --all-features -- -D warnings` is clean locally. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release v0.4.1
Additive release — no breaking changes from 0.4.0. Bumps the version, adds the 0.4.1 CHANGELOG entry, and documents the features that landed on
mcpsince 0.4.0.Highlights
degreesubcommand auto-detects the format and converts raw ai-landscape JSON on the fly.degree convert— ai-landscape program JSON → unified JSON (cluster pipeline files expand to one unified file per program).degree schema— emit the unified-degree JSON Schema.degree analyze(-j/--jobs, default 8) — process-isolated worker pool; a pathological degree can't take down the batch.--schoolrolls metrics up into one report. Plusscripts/analyze-batch.shfor an externally throttled (ulimit + timeout) variant.degree trim— round-trips the input format.convert_degreeandget_degree_json_schema; existing degree tools now accept unified/ai-landscape JSON and thecache:<hash>handle.Fixes
JsonError.Docs
CHANGELOG (0.4.1 entry), Readme (version + features),
docs/degree.md(convert/schema/-j/--school/JSON format),docs/mcp.md(the two new tools + JSON-input note).See CHANGELOG.md for the full entry.
🤖 Generated with Claude Code