Skip to content

[TS Calls] Build a real-project call census and model roadmap #367

Description

@CaelmBleidd

Part of #360. Uses #364; contributes to #385.

Goal

Measure unknown calls reached on real TypeScript projects and select a small evidence-based model roadmap.

Scope

  • Freeze development-project revisions, licenses, selection criteria, analyzed methods, input domains, frontend/tool versions, seeds and budgets.
  • Reuse the existing native frontend, runner and observer. Aggregate decision events outside the engine; [TS Calls][Research] Capture stopped-path frontiers for article experiments #373 snapshots are not required for a basic census.
  • Define the observation profile explicitly. A stop-on-first-unknown run cannot reveal later calls on that path. Use a fixed opaque-continuation census and/or a separately reported stop profile to distinguish encountered calls and stopping points.
  • Opaque exploration can include infeasible concrete paths and omit callee effects. Report a census of observed events under the profile, not an exhaustive list of all calls or all runtime blockers in a project.
  • Normalize API families/methods without conflating same-named user methods with proven built-ins.
  • Count projects, distinct call sites, distinct containing functions and dynamic events with explicit denominators. Rank primarily by project prevalence and distinct sites; report loop event counts separately.
  • Classify observations as model candidates, frontend/engine defects, external/user behavior or already supported semantics.
  • Use [TS Calls] Audit and classify existing semantic approximations #368's inventory to avoid reimplementing existing models. Record candidate domains, residual cases and the rationale for source code versus a genuine engine intrinsic.
  • Produce a small P0/P1/P2 list. Select a finite experimental subset before final evaluation; held-out projects never select or tune it.

Definition of Done

  • Machine-readable observations preserve profile, limits, project/source examples and raw counts.
  • Source-site identity and aggregation are deterministic; repeated budget-limited runs report observed variability instead of promising identical explored events.
  • Stop-profile censoring and opaque-profile approximation are explicit.
  • A reviewed model subset is ready for [TS Calls] Evaluate fallback policies and models with concrete replay #385; missing implementation work is split into bounded family issues with evaluation dependencies.
  • No catalog expansion is implemented in this issue.

The integration examples in #365/#366 are allowed before this roadmap; they validate the mechanism and must not be retroactively presented as census-selected models. Corpus selection and aggregation work can proceed independently of #383.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions