.NET: [Feature]: FoundryEvals — uploaded (file-id) datasets and per-criterion judge configuration

[ANcpLua]

Description

Two independent widenings of the FoundryEvals client surface (Microsoft.Agents.AI.Foundry, main @ 7b6d25798). Both are supported by the underlying Evals API (evals.create / evals.runs.create) but not reachable through the client today.

First, credit where due: #6267 already removed the biggest client-side ceiling here — FoundryEvalConverter.ResolveEvaluator now passes any builtin.*-prefixed name through to the service instead of rejecting names missing from the client dictionary, so new server-side evaluators are usable without a package update. The two asks below are the remainder.

1. No uploaded (file-id) dataset source.

The wire layer materializes every eval row inline: WireJsonlDataSource only carries a WireFileContentSource (type: "file_content"), so each run re-uploads its dataset as inline JSONL content. The Evals API also accepts previously uploaded JSONL datasets referenced by file id. Large or shared eval sets shouldn't have to be re-materialized per run.

Proposal: an overload (or data-source parameter) that accepts a file id, mapping to the API's file-based data source alongside the existing inline path.

2. One judge model for all criteria, no per-criterion threshold.

Every FoundryEvals constructor takes a single string model that configures the judge for all criteria, and no threshold is expressible anywhere in the client surface. The API allows per-criterion configuration. FoundryEvaluatorSpec (introduced in #6267) is currently BuiltinName | GeneratedEvaluatorRef — it is the natural carrier for per-criterion settings.

Proposal: optional per-spec configuration, e.g.:

public readonly struct FoundryEvaluatorSpec
{
    // existing: BuiltinName, GeneratedRef, ...
    public string? Model { get; init; }      // overrides the FoundryEvals-level judge model
    public double? Threshold { get; init; }  // per-criterion pass threshold
}

• What problem does it solve? Removes the remaining client-side ceilings on server capability: big datasets without inline re-upload, and mixed judge models / thresholds per criterion in a single eval.
• Expected behavior: existing calls unchanged; both additions are opt-in and additive.
• Alternatives considered: hand-rolling raw evals.create payloads next to FoundryEvals (duplicates the wire layer); running one FoundryEvals instance per judge model (loses the single-eval grouping and duplicates runs).

Code Sample

FoundryEvals evals = new(
    projectClient,
    model: "gpt-4o",
    new FoundryEvaluatorSpec("relevance") { Threshold = 4.0 },
    new FoundryEvaluatorSpec("builtin.coherence") { Model = "gpt-4o-mini" });

// uploaded-dataset run instead of inline rows (shape illustrative):
await evals.EvaluateDatasetAsync(fileId: uploadedJsonlFileId);

Language/SDK

.NET