Skip to content

14-Python_API_Surfaces.md

Latest commit

 

History

History
412 lines (365 loc) · 24.9 KB

14-Python_API_Surfaces.md

File metadata and controls

412 lines (365 loc) · 24.9 KB
Raw

Python API Surfaces

Normative public Python surfaces for embedding Weft. Operation meaning is owned by the vertical specs; this document owns which names are public, the stability policy, result/error contracts, and surface layering.

Public surfaces [PY-1]

Surface Import form Role
weft.client package (__all__) Primary embedder interface for application logic.
weft.ext module (__all__) Extension and downstream contracts.
weft.commands package (__all__, lazy facade) CLI-equivalent adapter for process and CLI reuse.

Each __all__ is its authoritative public-name inventory. Names not exported there, including weft.core.*, helpers, constants, command leaves, and execute_run, are private.

weft.ext.__all__ exports RunnerHandle, RunnerCapabilities, RunnerRuntimeDescription, AgentResolverResult, AgentToolProfileResult, AgentMCPServerDescriptor, RunnerEnvironmentProfileResult, AgentResolver, AgentToolProfile, RunnerEnvironmentProfile, TaskRunnerBackend, RunnerPlugin, SpecRunInputRequest, ResourceMetrics, RunnerOutcome, SessionExecutionResult, NormalizedAgentMessage, NormalizedAgentWorkItem, CommandSessionProtocol, and AgentSessionProtocol.

weft.client.__all__ retains its existing inventory and adds exactly CommandError, CommandUsageError, CommandTimeoutError, CommandExecutionError, SubmissionError, SubmissionValidationError, SubmissionManagerError, and normalize_taskspec_payload.

The client additionally exports TaskSpec, SpecSection, IOSection, StateSection, LimitsSection, RunnerSection, ReservedPolicy, AgentSection, AgentTemplateSection, AgentToolSection, ParameterizationSection, ParameterizationArgumentSection, RunInputSection, RunInputArgumentSection, RunInputStdinSection, WeftContext, and build_context. These are the existing validated models and context owner, not parallel representations. Supported TaskSpec use is construction, validation, field inspection, and serialization. Lifecycle and transport helpers remain runtime internals. Template validation uses TaskSpec.model_validate(payload, context={"template": True, "auto_expand": False}); normal construction retains resolved expansion. Submission remains responsible for committing the task and its TID.

build_context retains its existing arguments and side effects: by default it resolves configuration and creates the metadata directories and broker; create_dirs=False and create_database=False disable those respective creation steps; resolution can still create the project root. A supplied context is preserved by WeftClient. The context does not own a permanently open broker; callers close queues they obtain and use the broker context manager to release its resources. The context's config and broker_config are immutable SimpleBroker Config snapshots with uppercase unprefixed keys, as specified in [SB-0.4]. Create a new context to apply changed configuration; copying its values does not reconfigure live handles. Broker types remain SimpleBroker's public contracts.

WeftContext.session() returns a new SimpleBroker BrokerSession bound to the context's resolved target and Config. It is intended for a with block entered and exited by the executing thread. WeftContext and WeftClient do not cache a live session. Existing queue() and broker() ownership and defaults are unchanged.

Implementation mapping: weft/context.py::WeftContext.session owns the factory. Exact target/config behavior is covered by tests/context/test_context.py; real-backend task/session cleanup is covered by tests/core/test_task_runtime_connections.py.

Related plan: SimpleBroker configuration migration. Related plan: Explicit broker session lifetimes.

Extension value types preserve their existing fields and behavior. CommandSessionProtocol exposes readonly pid, handle, last_metrics; send(data), close_stdin(), poll_stdout(), poll_stderr(), is_alive(), returncode(), terminate(*, deadline=None), close(), poll_limits(), and stop_monitor(). AgentSessionProtocol exposes readonly pid, handle, execute(work_item, *, cancel_requested=None) -> SessionExecutionResult, and close(*, deadline=None). Backend session methods return these structural protocols; concrete process session constructors remain private. These types preserve existing lifecycle semantics and introduce no new cleanup guarantee. Agent callbacks receive the AgentSection publicly available from weft.client and normalized work-item types from weft.ext. First-party runners may still use private implementation under coordinated versioning; these exports do not promise a complete standalone backend SDK.

Implementation: weft/client/__init__.py re-exports the schema from weft/core/taskspec/model.py and context from weft/context.py. weft/ext.py owns the public extension values and protocols; weft/core/tasks/runner.py, weft/core/tasks/consumer.py, and weft/core/tasks/interactive.py consume the structural session contracts.

WeftClient.from_context(spec_context=None, *, fallback_root=None, autostart=None) requests a resolved context from the core context owner. fallback_root follows [SB-0.4] and is not an explicit override of a parameterized TaskSpec reference's declared context. An explicit path argument, caller-supplied WeftContext, or resolved Config CONTEXT is explicit. Direct WeftClient() construction and the factory agree about this policy. Clients preserve supplied contexts and use their resolved Config snapshots; the Django adapter does not set core's context-precedence bookkeeping.

Implementation mapping: weft/client/_client.py::WeftClient.__init__, weft/client/_client.py::WeftClient.from_context, weft/context.py::build_context, weft/client/__init__.py, weft/commands/__init__.py, weft/ext.py, weft/core/taskspec/model.py, tests/core/test_client.py, tests/core/test_public_client_contracts.py, tests/core/test_public_extension_contracts.py, and tests/fixtures/public_extension_contract.py.

Commands surface contract [PY-2]

Task observations use the shared per-TID runtime-state readers ([OBS.6]); known-TID snapshots read that state queue directly. Snapshot and control waiters subscribe to the exact queue even before its first write; realtime snapshot requests remain fresh. CLI event watches remain incremental event streams. This storage change adds no public signature, result shape, iterator retention contract, or lifecycle precedence change. Python system status retains its unfiltered default; CLI status retains terminal filtering.

Every canonical CLI verb has one actual implementation exported from weft.commands: cmd_ plus its full CLI path, joined with underscores and hyphens normalized to underscores. The initial bijection is 41 verbs: 4 root, 14 queue, 6 spec, 6 task, 5 manager, and 6 system. The root callback and global --version are parser features, not verbs. The package resolves exports lazily and caches them.

Commands accept parsed semantic values, perform semantic validation and orchestration, return structured outcomes, and raise typed errors. They do not read process stdin or write process output. weft.cli owns shell decoding, human/JSON formatting, stream routing, and exit translation. Each Typer verb invokes its corresponding facade export exactly once.

Canonical CLI argument/long-option spellings become parameter names by replacing hyphens with underscores; callback-local names and short aliases do not govern them. Repeatable options retain the singular spelling and accept a Sequence; dual flags use the positive semantic spelling (apply, follow). Positional arguments are positional-or-keyword and semantic options are keyword-only. No command accepts **kwargs. Presentation-only --json, --quiet, --verbose, --error, and --timestamps are excluded. Task-list --stats is also presentation-only; queue-list --stats remains semantic.

The exact signature exceptions are: cmd_run has spec_args=(), describe=False, and stdin_text=None; cmd_queue_write(queue_name, message=None, *, endpoint=None) exposes its positional overload; cmd_system_task_monitor(..., follow=False) exposes the positive side of --once/--follow. cmd_run(describe=True, spec=REF) returns spec-aware help metadata and never submits.

Public command types

weft.commands.__all__ exports the 41 functions and these consumption types: InitResult, RunSpecDescription, RunSession, CommandStream, RunExecutionResult, SubmittedTaskReceipt, TaskSnapshot, TaskResult, TaskEvent, ServiceSnapshot, TaskPingResult, TaskControlResult, TaskControlFailure, QueueEntry, QueueInfo, QueueWriteReceipt, QueueMoveResult, QueueDeleteReceipt, QueueBroadcastReceipt, QueueAliasRecord, EndpointResolution, ManagerSnapshot, SpecRecord, SpecValidationResult, SpecMutationResult, SystemStatusSnapshot, SystemTidyResult, SystemLoadResult, SystemDumpResult, SystemPruneResult, BuiltinSpecRecord, TaskMonitorConfig, TaskMonitorResult, TaskMonitorRecord, and TaskMonitorSummary.

New/refined exact contracts:

  • InitResult(root: Path, config_path: Path, created: bool).
  • RunSpecDescription(reference: str, usage: str, arguments: tuple[Mapping[str, Any], ...], stdin: Mapping[str, Any] | None).
  • CommandStream[T] is an iterator with idempotent close(); exhaustion and close release resources and iteration failures use typed command errors. Broker-backed stream advancement and final closure share an executing-thread owner. Transport adapters preserve that ownership across async delivery and cancellation; they do not close an iterator concurrently with an active advance. A genuine stream-body failure stays primary when cleanup also fails; a cleanup failure during lifecycle-only generator closure remains observable to the close caller.
  • RunSession exposes tid, events() -> CommandStream[TaskEvent], send_input(text), close_input(), stop() -> TaskControlResult, wait(timeout=None) -> RunExecutionResult, and idempotent close(). Close releases owned resources but does not cancel the task.

Implementation mapping: weft/commands/_resources.py owns stream cleanup exception priority; weft/commands/_result_wait.py, _spawn_submission.py, events.py, queue.py, result.py, system.py, and tasks.py own the bounded and generator observation scopes; weft/commands/run.py::_LiveRunSession owns input and child-stream closure. Cross-thread Django adaptation is mapped under [DJ-12.1] and [DJ-12.2]. Firing cleanup coverage lives in tests/commands/test_resource_cleanup.py, test_queue.py, test_status.py, and integrations/weft_django/tests/test_weft_django.py.

  • QueueMoveResult(source: str, destination: str, entries: tuple[QueueEntry, ...], moved_count: int); entries are the exact ordered moved set.
  • TaskResult retains its result fields and carries optional reconciliation: dict[str, Any] | None = None so the CLI can render the existing claimed-result metadata contract [CLI-1.2.2]. The owner is weft/commands/result.py::await_task_result; the CLI renders that evidence without another lifecycle probe. Implementation plan: Dead generation retirement.
  • TaskPingResult(tid: str, acknowledged: bool, timed_out: bool, error: str | None, observed_at: int | None, pong: Mapping[str, Any] | None, snapshot: TaskSnapshot | None).
  • TaskControlFailure(tid: str, error: str, error_type: str) records a selected task whose control attempt could not be confirmed. error is the rendered failure message and error_type is the exception class name.
  • TaskControlResult(command: Literal["stop", "kill"], requested: tuple[str, ...], accepted: tuple[str, ...], failures: tuple[TaskControlFailure, ...], snapshots: tuple[TaskSnapshot, ...]). accepted and failures partition requested. An empty selection is a successful zero-count outcome. In a sweep, the command raises only when at least one task was requested and none was accepted; the raised ControlRejected carries the full failure tuple on its documented failures attribute. A single genuinely unknown TID instead propagates TaskNotFound, which the CLI renders with exit code 2. A known terminal task rejects stop with ControlRejected and an already <status> message before a control queue is written. It remains eligible for kill so kill escalation can reap runtime residue; when no live runtime can be proven, the rejection states the terminal status and absence of runtime residue. Client stop_many()/kill_many() with no selector are the empty-selection case. Explicit tids cannot be combined with all_tasks or pattern; mixed scope is a typed usage error.
  • QueueDeleteReceipt(queue: str | None, deleted_count: int, queues_deleted: int, all_queues: bool, exact_message: str | None).
  • SpecMutationResult(action: Literal["create", "delete"], record: SpecRecord).
  • SystemDumpResult(path: Path, queues: int, messages: int, aliases: int, omitted_claimed_queues: int, omitted_claimed_messages: int).
  • SystemPruneResult(families: tuple[str, ...], applied: bool, candidates: int, deleted: int, failed: int, details: Mapping[str, Any]).
  • BuiltinSpecRecord(name: str, description: str | None, category: str | None, function_target: str | None, supported_platforms: tuple[str, ...], path: Path, source: str = "builtin").
  • TaskMonitorRecord(record: Mapping[str, Any]) is the lossless pre-serialization form of each run-start, task-summary, and run-completed record in exact emission order.
  • TaskMonitorResult(log_path: Path | None, records_written: int, events_scanned: int, tids_seen: int, summaries_emitted: int, checkpoint_timestamp: int | None, records: tuple[TaskMonitorRecord, ...]). It has no exit-code, stdout, stderr, or JSON-rendering fields.
  • TaskMonitorConfig(context: str | Path | None = None, follow: bool = False, sink: Literal["stdout", "disk"] = "stdout", log_dir: Path | None = None, checkpoint: Path | None = None, no_checkpoint: bool = False, since: int | None = None, limit: int | None = None, monitor_name: str = "default") and TaskMonitorSummary(record: Mapping[str, Any]).
  • RunExecutionResult has no submission_error; terminal status, result, and failure detail remain outcome data.
  • SpecRecord has payload: Mapping[str, Any] | None = None; show always supplies it. TaskEvent.payload losslessly carries lifecycle, output chunk, result, and control envelopes in observed order.
  • TaskSnapshot adds host_pids, managed_pids, and live_managed_pids, each tuple[int, ...] | None = None; process mode populates them.
  • ManagerSnapshot adds liveness: Literal["live", "stale", "unknown", "non_live"] | None = None, proof_source: str | None = None, proof_detail: str | None = None, dispatch_eligible: bool | None = None, canonical_candidate: bool | None = None, and canonical: bool | None = None; diagnostic mode populates them. started_here: bool | None = None is populated only by cmd_manager_start so the CLI can preserve its started-versus-existing lifecycle message without a second semantic query.

Deterministic return matrix

Family Return
root init InitResult
root status SystemStatusSnapshot, or CommandStream[TaskEvent] iff watch
root result TaskResult, tuple in all mode, or CommandStream[TaskEvent] iff stream
root run RunSpecDescription iff describe; otherwise RunSession iff wait; otherwise RunExecutionResult
queue read/peek/watch tuple/stream of QueueEntry
queue write/move/delete/broadcast corresponding receipt/result
queue list/exists/stats/resolve/alias named QueueInfo, EndpointResolution, QueueAliasRecord, tuple, or bool
spec create/delete/list/show/validate/generate mutation, record tuple/record, validation, or mapping as named
task list/status/ping/stop/kill/tid snapshots/events, ping/control result, or full TID
manager start/status/list snapshot or tuple; serve blocks and returns None
manager stop terminal snapshot, or None when already absent
system tidy/load/task-monitor/prune/dump/builtins corresponding named outcome

Interactive run only changes session capabilities. Task monitor returns CommandStream[TaskMonitorSummary] iff follow=True, else TaskMonitorResult; sink does not alter the branch. cmd_manager_serve is the sole blocking non-stream exception and emits no process output.

Client system dump, load, and tidy preserve the supplied resolved context, including its broker target and artifact directory. Without an explicit output, the export is weft_export.jsonl in that context's Weft directory; without an explicit input, load reads that same file. A RunSession.stop likewise controls the task on the context the session was submitted with.

Implementation: weft/commands/dump.py::dump_system and cmd_system_dump share the resolved-context materializer; weft/commands/load.py and weft/commands/tidy.py share theirs the same way, and weft/commands/run.py::_LiveRunSession.stop passes its session context to the task-control owner. Correction plan: Complexity review corrections.

Typed errors and CLI exits

The exact hierarchy is CommandError(WeftError), CommandUsageError(CommandError, ValueError), CommandTimeoutError(CommandError, TimeoutError), and CommandExecutionError(CommandError, RuntimeError). Existing InvalidTID, TaskNotFound, SpecNotFound, ControlRejected, ManagerNotRunning, and ManagerStartFailed retain their inheritance. Submission errors are SubmissionError(CommandError), SubmissionValidationError(SubmissionError), and SubmissionManagerError(SubmissionError). Backend, validation, and OS errors are translated with chaining at the command seam. Error classes exported by client and commands are identical objects.

The CLI maps CommandUsageError, InvalidTID, TaskNotFound, and SpecNotFound to 2; CommandTimeoutError to 124; every other CommandError to 1 except the spec-pinned system load alias-conflict case, which returns 3 before writes begin; success maps to 0. When an internal remaining-budget or completion-grace wait expires, the public timeout diagnostic retains the caller's requested timeout rather than exposing the internal sub-budget. Ctrl-C remains a shell concern.

Client submission with declared arguments [PY-3]

WeftClient.submit_spec(reference, *, spec_args=(), payload=None, stdin_text=None, **overrides) and prepare_spec(...) share the run pipeline: parameterization first, remaining tokens to run-input, before TID commit. For a spec declaring run-input, adapter output is the payload and payload= is rejected. Without run-input, payload= is valid. cmd_run and client spec submission accept a single stdin_text: str | None. The submission seam routes it to declared run-input stdin when the spec's run_input declares stdin, as the initial work payload when the spec has no run_input contract, and rejects it with a typed usage error when a run_input contract exists but declares no stdin. Client and command surfaces never read process stdin; the CLI adapter reads piped stdin once and forwards it as stdin_text. Failures are typed; a returned Task is the committed receipt and uses the materialized spec's runtime context. Runtime roots expand ~ and resolve to an absolute path through the shared submission seam on every surface. Declared-argument parse errors are CommandUsageError; malformed/materialization and adapter failures are SubmissionValidationError with the original exception chained, except that an adapter-raised WeftError retains its exact public type and mapping. A non-persistent human-readable name is not an endpoint claim and therefore does not use endpoint-name syntax validation; persistent names still do. payload= and stdin_text= are mutually exclusive even when the spec has no run_input contract, so the initial work payload has one owner.

Submit-override semantics are identical on every surface that accepts **overrides: an override whose value is None is ignored and the template's value stands; a name outside the shared public override vocabulary raises TypeError from prepare(...), prepare_pipeline(...), and submit_command(...), and SubmissionValidationError from prepare_spec(...); a value the TaskSpec schema rejects raises the schema's validation error (a ValueError) from prepare(...) and SubmissionValidationError from prepare_spec(...); a name in the reserved _weft. namespace raises ValueError. Each submit(...), submit_spec(...), and submit_pipeline(...) raises exactly what its prepare* counterpart raises. weft.client.normalize_taskspec_payload(taskspec, **overrides) runs the shared definition-normalization contract without a client or context and returns the validated, normalized TaskSpec definition as a fresh JSON-compatible dict: the definition snapshot used by prepare(...) before runtime-context binding, submission-time transport encoding, or reserved-metadata handling. Overrides are applied, re-validated, and JSON round-tripped. Pure normalization preserves a relative or home-relative spec.weft_context declaration as text. It accepts a TaskSpec or a JSON-compatible mapping (a mapping without tid is validated as a template), raises the errors from prepare(...)'s shared definition-normalization stage, constructs no Weft context, reads no configuration, resolves no project root, opens no broker, and writes nothing. Every keyword argument is an override name: payload is not part of the override vocabulary and raises TypeError here. The mapping carries no top-level bundle-root marker and no bundle provenance; a TaskSpec input that already carries a bundle root is accepted, its root is re-resolved read-only by TaskSpec validation, and the result drops it. Embedders that need a TaskSpec definition for composition rather than submission call it; there is no second normalization path.

Context-aware prepare(), prepare_spec(), and prepare_pipeline() bind a nonempty declared spec.weft_context to an absolute path using the preparation-time CWD and home expansion. Binding occurs after parameterization and overrides and before any run-input adapter observes the runtime root. An absent declaration stays unset and uses the captured client's root at submission. Binding copies and validates the TaskSpec without mutating its input, TID, io, bundle provenance, or request flags. It opens no alternate broker; broker selection for an explicit alternate root remains at submission using captured Config. Pure normalize_taskspec_payload() does not perform runtime-context binding.

Implementation mapping: weft/client/_client.py::normalize_taskspec_payload over weft/commands/submission.py::prepare_definition. Runtime context binding is owned by weft/commands/submission.py::_bind_declared_context, weft/commands/submission.py::prepare, weft/commands/submission.py::prepare_spec, and weft/commands/submission.py::prepare_pipeline. Related preparation input and error contracts are covered by weft/ext.py, tests/core/test_client.py, and tests/commands/test_submission.py.

Layering [PY-4]

Runtime imports are one-way: cli -> commands -> core, client -> commands -> core, and core -> ext; commands may also import ext. Commands never import adapters; core never imports commands or adapters; CLI and client never import each other; runtime ext imports never point back to core, commands, or adapters. Type-checking-only extension annotations may refer to public client schema types. The client package initializer may re-export only the declared schema types from weft.core.taskspec.model; this is a value-type facade exception, not permission for client operations to call core. CLI and core initializers remain import-light markers; client and commands are public package facades and ext is a public module. Architecture tests enforce the graph, facade inventory/laziness, CLI bijection, no command stdin access, and exactly one matching facade invocation per Typer callback.

Related Plans