feat(sleep): propagate harvested skill hints into mined tasks - #183
feat(sleep): propagate harvested skill hints into mined tasks#183bogdanbaciu21 wants to merge 3 commits into
Conversation
Add a backward-compatible SessionDigest.skills_used field and populate it from well-formed Claude Skill tool-use blocks only. tools_used behavior and legacy digest payload loading are unchanged. Refs microsoft#120
Add an optional TaskRecord.skill_hint and a session_skill_hint helper. A session with exactly one harvested skill names it; absent or ambiguous hints stay empty so those tasks remain in the existing catch-all path. Refs microsoft#120
There was a problem hiding this comment.
Pull request overview
Adds end-to-end propagation of harvested Claude Skill tool invocations into mined TaskRecords as a skill_hint, enabling later per-skill grouping without changing existing behavior for unknown/ambiguous sessions.
Changes:
- Introduces
SessionDigest.skills_usedharvesting from ClaudeSkilltool-use blocks and persists it via existingto_dict()serialization. - Adds
TaskRecord.skill_hintplusmine.session_skill_hint()and propagates the hint through both heuristic and LLM-backed miners. - Extends
dedup_tasks()to merge/resolveskill_hintacross duplicate tasks, and adds unit tests covering defaults, legacy payloads, propagation, and dedup behavior.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
skillopt_sleep/types.py |
Adds skills_used to SessionDigest and skill_hint to TaskRecord with backward-compatible loading. |
skillopt_sleep/harvest.py |
Extracts Skill tool targets from assistant tool-use blocks into SessionDigest.skills_used. |
skillopt_sleep/mine.py |
Adds session_skill_hint() and propagates skill_hint into heuristic mining and dedup logic. |
skillopt_sleep/llm_miner.py |
Propagates skill_hint into LLM-mined rule/rubric tasks via session_skill_hint(). |
tests/test_sleep_skill_hint.py |
Adds unit tests for skill_hint defaults/serialization, propagation, and dedup agreement/fill/conflict. |
tests/test_sleep_skill_harvest.py |
Adds unit tests for harvesting skills_used from Claude transcripts, including malformed cases. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # merged sessions disagreeing about the skill make the hint | ||
| # ambiguous, so drop back to the catch-all path | ||
| if not ex.skill_hint: | ||
| ex.skill_hint = t.skill_hint | ||
| elif t.skill_hint and t.skill_hint != ex.skill_hint: | ||
| ex.skill_hint = "" |
Yif-Yang
left a comment
There was a problem hiding this comment.
Thank you for building this as a follow-up to #182, which is now merged. Propagating a skill hint into mined tasks is useful, but the current head has one deterministic correctness bug and one propagation gap that should be fixed before merge.
dedup_tasks()uses an empty string for both “no hint seen yet” and “a conflict was already observed.” This makes the result order-dependent:A, B, Aends withA, whileA, A, Bends empty. Please aggregate the non-empty hints for each task ID and retain a hint only when the final set has exactly one value.dream_augment()andrecall_similar()reconstructTaskRecordobjects without carryingskill_hint, so the metadata is lost in those paths.
Please add permutation tests such as A/B/A, B/A/B, and unknown/empty interleavings, plus focused dream/recall propagation tests. It would also be safer to place the new defaulted TaskRecord.skill_hint field at the end of the dataclass to avoid changing positional construction for external callers.
Once these focused points are addressed, this should be straightforward to re-review. Thanks again for extending the original contribution cleanly.
|
Will do. Super helpful feedback. Will jump on this ASAP but need some time, want to get this right and deliver good quality implementation. Will follow up when the required changes are complete. Thanks so much. |
|
Hello. I have a proposed solution. Your insight was very helpful and informative. At this point focused tests pass, but it is late and a proper end to end full re-run would make me feel better we got this right, so will do tomorrow and hope to address and resolve this. Thx so much. Dan |
|
Hi Yif, Thank you again for the precise and very helpful review. I took some extra time with this because I wanted to address the underlying behavior properly and validate the result beyond the immediate regression cases. Proof from the updated PR head: 116 passed, 1 skipped, and 130 subtests passed across the focused sleep path. An exhaustive 5,460-case ordering check passed across empty, A, B, and C hints for sequence lengths one through six. Ruff, Python compilation, and the Git diff checks all passed. The full suite reports 585 passed and 6 skipped. The remaining two failures are macOS path-resolution assertions involving The core fix changes I also updated The committed regression coverage includes both alternating conflict patterns, permutations, empty-hint interleavings, an exhaustive final-set check, the dream and recall paths, and compatibility with the previous positional constructor layout. The update is pushed in commit Dan |
Summary
Second review-sized slice of #120. Mined tasks currently carry no notion of
which skill they exercised, so per-skill work later in the issue has nothing to
group on.
TaskRecord.skill_hintwith backward-compatibleserialization (
to_dict()/from_dict()still accept legacy payloads);mine.session_skill_hint(digest): a session that invoked exactly one skillnames it, while zero skills (unknown) and several skills (ambiguous) both
return
"";rubric tasks;
dedup_tasks, fill a missing hint from a merged duplicate but clear thehint when merged sessions disagree.
Nothing consumes
skill_hintyet: grouping, staging, and adoption behavior areunchanged, so absent or ambiguous hints stay in the existing catch-all path.
Tests
python -m pytest -q tests/test_sleep_skill_hint.py→ 13 passedpython -m pytest -q→ 580 passed, 7 skipped (basemainat8304e6c:556 passed, 7 skipped)
python -m ruff check skillopt_sleep/types.py skillopt_sleep/mine.py skillopt_sleep/llm_miner.py tests/test_sleep_skill_hint.pyreports only the three pre-existing
F401findings already present onmain.New coverage: field default, serialization round-trip, legacy payload without
skill_hint, single/absent/ambiguous/repeated skill sessions, heuristic-minepropagation, dedup agreement/fill/conflict, and LLM-miner rule and rubric tasks.
Refs #120