Skip to content

Review and fix remediation instructions across all assessors for alignment with scoring #505

@jwm4

Description

@jwm4

While reviewing PR #503, we noticed that the ThreatModelAssessor remediation includes the step "Point SECURITY.md at the threat model for scope guidance", but no assessor actually checks whether this is done. The remediation advises users to do something that won't affect their score.

This may be a broader pattern. We should audit all assessor remediation instructions and fix any misalignments:

  1. Each remediation step should map to something the assessor actually scores. Steps that don't affect the score are misleading, since users following remediation expect their score to improve.
  2. No scored criteria should lack corresponding remediation guidance. If an assessor checks for something, the remediation should tell users how to satisfy it.
  3. Remediation steps should be actionable and clear. Jargon or vague instructions (like "Point SECURITY.md at the threat model for scope guidance") should be rewritten as concrete actions.

For each misalignment found, use judgment on the right fix:

  • If the remediation step reflects something genuinely valuable to assess, add scoring logic to the assessor so the step is no longer unscored.
  • If the remediation step is aspirational or out of scope for the assessor, remove or scale back the instruction so users aren't misled.
  • If a scored criterion has no remediation guidance, add a remediation step explaining how to satisfy it.

Origin: Found during review of #503, where ThreatModelAssessor._create_remediation() includes an unscored step.

Suggested approach: Iterate through all BaseAssessor subclasses, compare each _create_remediation() (or equivalent) against the assess() scoring logic, and fix misalignments. Update docs/attributes.md for any scoring or remediation changes. Add tests for any new scoring logic.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions