Skip to content

[Story] Telemetry for dynamic tool loading #582

@edelauna

Description

@edelauna

Context

We need data to know whether dynamic loading is actually helping: how often the threshold is hit, how large the attached set ends up after the rolling window, how often the model calls `mcp_load`, and which ranker tier fired. Without telemetry we can't tune the defaults.

Depends on #578, #579, #580.

Developer Notes

  • Events to emit (via the existing telemetry pipeline — check `src/services/telemetry/`):
    • `dynamic_tools.task_start`: `{ scopedCount, attachedCount, ranker: "embeddings" | "bm25" }`
    • `dynamic_tools.mcp_load_invoked`: `{ taskTurn, queryLength, returnedCount, alreadyAttachedCount }`
    • `dynamic_tools.rolling_window_step`: `{ turn, addedCount, totalAttached }` (only when something changes)
    • `dynamic_tools.ranker_fallback`: `{ from: "embeddings", to: "bm25", reason }`
  • Respect existing telemetry opt-out.
  • Emit from `src/services/tools/ToolRouter.ts` (most events) and `src/core/tools/mcpLoadTool.ts` (`mcp_load_invoked`).

Acceptance Criteria

  • All four event types fire in integration tests
  • Opt-out correctly suppresses all events
  • No telemetry emitted when dynamic loading is inactive (under threshold)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions