Conversation
nishantmonu51
added
Type:Feature
nishantmonu51
left a comment
nishantmonu51
left a comment
There was a problem hiding this comment.
Two notes alongside the inline comment, plus one clause on staleness.
- The
%#vfallback inwriteConfigHash(runtime/connection_cache.go:216) is effectively unreachable: instance connector config comes fromstructpb.Struct.AsMap()plus string env overrides (runtime/connections.go:339-355), and system connector config is likewise astructpbmap, so every value is JSON-encodable. Hashing the JSON alone and letting ajson.Marshalerror surface would be simpler than a fallback that would print pointer addresses and produce a fresh cache key on every acquire. - In
json_objectmode the schema instruction is prepended as a separatesystemmessage ahead of whateveropts.Messagesalready contains (runtime/drivers/openai/openai.go:428;runtime/ai/ai.go:1326passes the session's messages through unchanged, so Rill's own system prompt becomes the secondsystemmessage). Providers whose chat templates accept only a single leading system message would reject that; merging the instruction into the existing system message would be more portable. This is a portability concern, not a verified failure against a specific provider.
The branch is 62 commits behind origin/main, but git merge-tree reports a clean merge and none of the intervening commits touch runtime/drivers/openai/ or runtime/connection_cache.go.
| ### OpenAI-compatible APIs | ||
|
|
||
| The connector can also target APIs that implement the OpenAI chat completions protocol. Configure the provider's URL and model on the connector. If the provider supports JSON mode but not OpenAI's JSON Schema response format, set `structured_output_mode` to `json_object`: | ||
|
|
||
| ```yaml | ||
| type: connector | ||
| driver: openai | ||
| api_key: "{{ .env.PROVIDER_API_KEY }}" | ||
| base_url: https://llm.example.com/v1 | ||
| model: example-model | ||
| structured_output_mode: json_object | ||
| ``` | ||
|
|
||
| For provider-specific request extensions, use the advanced `extra_body` map. Its values must be JSON-serializable, and it cannot override core request fields such as `model`, `messages`, `tools`, `response_format`, or streaming controls. For example, an OpenAI-compatible endpoint can receive a chat-template option as follows: | ||
|
|
||
| ```yaml | ||
| extra_body: | ||
| chat_template_kwargs: | ||
| enable_thinking: false | ||
| ``` | ||
|
|
||
| These options belong to the connector, so different connectors in the same Rill process can use different provider behavior. |
There was a problem hiding this comment.
structured_output_mode and extra_body are not declared in the OpenAI block of runtime/parser/schema/project.schema.yaml, which this PR does not touch. That block generates docs/docs/reference/project-files/connectors.md through cli/cmd/docs/generate_project.go:29,488 (make docs.generate), so the OpenAI connector reference this page links to for "additional configuration options" will not list either property, and runtime/ai/instructions/instructions.go:198 feeds the same schema to the AI assistant, which will keep reporting that these keys do not exist on an openai connector. Nothing fails at runtime: runtime/parser/parse_connector.go:16 captures connector properties with mapstructure:",remain" and never validates them against the schema. main has since reworked that block, so the two properties (and ideally the extra_body example) need to go on top of its current version.
2 participants
Summary
structured_output_modewith the existingjson_schemabehavior as the default and ajson_objectfallback that includes the schema in the promptextra_bodymap for provider-specific chat completion fields while rejecting overrides of core request and response-shape fieldsMotivation
OpenAI-compatible providers differ in structured-output support and provider-specific request extensions. This was motivated by an interoperability report in kairosagentica/kairos-cloud#145.
The defaults remain unchanged for existing OpenAI connectors, and all new behavior is configured per connector.
Test plan
go test -race ./runtime/drivers/openai -run "TestComplete(AppliesConnectorRequestBehavior|PassesNestedExtraBody|DefaultsToJSONSchema)|TestOpenValidatesProviderRequestBehavior" -count=1go test ./runtime -run TestGenerateConnectionKey -count=1go test ./runtime/... -run "^$"go vet ./runtime/drivers/openai ./runtimegit diff --check upstream/main...HEAD