Skip to content

Add outcome-aware evaluation loop for Cisco Firepower response - #14997

Draft
Ahmed Hassan (AAH20) wants to merge 7 commits into
Azure:masterfrom
AAH20:codex/firepower-response-evaluation
Draft

Add outcome-aware evaluation loop for Cisco Firepower response#14997
Ahmed Hassan (AAH20) wants to merge 7 commits into
Azure:masterfrom
AAH20:codex/firepower-response-evaluation

Conversation

@AAH20

Copy link
Copy Markdown

Dependency

Draft follow-up to #14925. Do not merge until #14925 is merged; this stacked PR currently includes its parent commits and will collapse afterward.

What changed

  • adds a versioned FirepowerOutcome:v1 incident-comment contract to both BlockIP playbooks
  • records ML-only policy denials, Teams HITL requests/rejections, successful containment, missing-IP, and FMC object failure outcomes
  • adds a Cisco Firepower Detection and Response Quality workbook for ML-only, signature, corroborated, sensor, and collector trends
  • adds an hourly SnortML signal-mix drift analytic with minimum-volume and seven-day baseline gates
  • adds deterministic evaluation cases and hard safety invariants for future agentic improvement proposals
  • packages the solution as version 3.1.0

Safety boundary

The agentic layer is deliberately outside the enforcement path. It may propose reviewed changes, but cannot edit production analytics, policies, or Firepower objects. ML-only evidence cannot enter automatic containment.

Validation

  • official local suite: 12 validator groups passed, 0 failed
  • ARM-TTK: 48 passed, 0 failed
  • solution-level checks: 112 passed, 0 failed
  • exact drift KQL compiled and executed in Log Analytics without semantic errors
  • Azure custom deployment completed successfully in a validation workspace
  • deployment operations confirmed the workbook template and all four analytic templates as OK
  • Sentinel listed and rendered the new workbook template and all three panels

Local .NET 3.1/8 validators were unavailable in the workstation runtime; KQL was instead validated against the live authenticated workspace. TruffleHog was not installed.

Ahmed Hassan (AAH20) and others added 7 commits August 16, 2026 18:57
Encode ML-only escalate vs signature/corroborated Gate/Prove dispositions so BlockIP playbooks do not treat SnortML GID 411 as classic signature true positives.
SnortML / GID 411 must not be treated as a signature true positive: NetworkGroup auto-contain cancels, Teams HITL warns and still requires analyst confirmation.

Co-authored-by: Cursor <cursoragent@cursor.com>
One solution version for Microsoft review: ASCII-only detections, stricter GID 411 matching, and colon-free Logic Apps action names.

Co-authored-by: Cursor <cursoragent@cursor.com>
aka.ms/deploytoazuregovbutton -> aka.ms/deploytoazuregovernbutton, per review comment.
Ran Create-Azure-Sentinel-Solution V3 against Solutions/Cisco Firepower
EStreamer/Data per review request, so mainTemplate.json and
createUiDefinition.json reflect the 3.0.3 analytic rules and Gate/Prove
BlockIP playbook changes. Version bumped 3.0.3 -> 3.0.4 (local mode,
patch), matching ReleaseNotes.md.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant