Analysis: HTTP Streaming Support in WordPress Core for AI/LLM Use Cases

[saraichinwag]

Summary

WordPress core's HTTP API (wp_remote_get() / wp_remote_post()) cannot stream responses. Every layer — from the Requests library transports through Requests::parse_response() to WP_HTTP_Requests_Response::to_array() — buffers the full response body as a string before returning.

This is a blocker for AI/LLM integration in WordPress, where Server-Sent Events (SSE) streaming from APIs like OpenAI, Anthropic, etc. is the standard pattern.

Full Analysis

See streaming-analysis.md for the complete call chain trace and layer-by-layer breakdown.

Key Findings

8 Buffering Bottlenecks Identified

#	Location	Issue	Severity
1	Curl::stream_body()	\$this->response_data .= \$data accumulates all chunks	Low
2	Fsockopen::request()	\$body .= \$block accumulates all chunks	Low
3	Curl::process_response()	Concatenates headers+body	Medium
4	Requests::parse_response()	Expects complete body string	High
5	Requests::parse_response()	decode_chunked() on complete body	Medium
6	Requests::parse_response()	decompress() on complete body	High (fsockopen) / Low (cURL)
7	WP_HTTP_Requests_Response::to_array()	'body' => string	High
8	WP_Http::request() return value	Flat array with string body	High

The request.progress Hook Already Exists

Both cURL and fsockopen transports fire a request.progress hook per-chunk during data reception. cURL handles decompression in-flight, so chunks arrive decompressed. This is the closest thing to streaming that exists today — but the body is still accumulated regardless.

Most AI Plugins Bypass wp_remote_* Entirely

Because the API fundamentally cannot stream, plugins use raw curl_init() / curl_exec() with CURLOPT_WRITEFUNCTION. This works but loses WordPress proxy, SSL cert, cookie, and hook handling.

Proposed Approaches

Approach	Core Changes?	Effort	Usability
request.progress hook workaround	No	Low	Poor (still accumulates)
Bypass WP HTTP, raw cURL	No	Low	Good (but loses WP features)
Transport-level stream_callback option	Requests lib only	Medium	Good
Full streaming API	WP Core + Requests	Medium-High	Excellent

Roadmap Context

This analysis supports enabling HTTP streaming in WordPress core for AI tool integration. The Requests library documentation in this repo provides the foundation for understanding the transport layer.

[saraichinwag]

Implementation Spec Added

The streaming analysis doc now includes a full implementation spec for native HTTP streaming support.

Design Approach

Extends the existing 'stream' => true pattern. 'on_data' callback sits alongside 'filename' — same flag, new capability. Feels native, not tacked-on.

Changes Required (6 files, ~50 lines core + ~150 SSE parser)

File	Change
class-wp-http.php	New on_data default + pass-through as stream_callback
Requests/src/Requests.php	Skip body parsing/decompression when stream_callback is active
Requests/src/Transport/Curl.php	stream_body() calls callback instead of $this->response_data .= $data
Requests/src/Transport/Fsockopen.php	Read loop calls callback instead of $body .= $block
Requests/src/Response.php	New $body_streamed flag
New: class-wp-sse-parser.php	Server-Sent Events parser for AI/LLM API streaming

Usage (Final API)

wp_remote_post( $url, [
    'stream'  => true,
    'on_data' => function( $chunk ) {
        echo $chunk;
        flush();
    },
    'timeout' => 120,
]);

Key Points

• Fully backward-compatible (new optional args, existing behavior unchanged)
• 19-point test plan included
• All code references cite actual source line numbers

See the full spec: streaming-analysis.md

[saraichinwag]

Update: REST API Streaming + Missing Docs

New docs added

• WP_Http_Requests_Hooks — The bridge between WordPress hooks and the Requests library. Critical for streaming: request.progress fires through Requests hooks directly, NOT through WordPress actions. Direct registration on the hooks object is required for chunk-by-chunk access.
• WP_Http_IXR_Client — XML-RPC client (no streaming relevance, documented for completeness).

REST API Outbound Streaming Analysis

The streaming analysis now covers both halves:

Inbound (calling LLM APIs): wp_remote_post() with 'on_data' callback — specced in previous update.

Outbound (serving SSE to clients): WP_REST_Server::serve_request() currently:

1. Hardcodes Content-Type: application/json (line 318)
2. Assembles complete response via response_to_data()
3. JSON-encodes the entire thing
4. Single echo at the end
5. No flush() calls

Proposed: WP_REST_Streaming_Response class that extends WP_REST_Response with a generator/callback for chunked output. The rest_pre_serve_request filter already provides an escape hatch that works today for plugins.

Full proxy pattern: An AI plugin could combine both halves — inbound on_data from the LLM API piped directly to outbound SSE via the REST API. The streaming analysis doc now includes the integration diagram.

Remaining buffering layers to consider

• PHP output buffering (ob_end_flush())
• nginx proxy_buffering
• Apache mod_deflate
• PHP zlib output compression

See: streaming-analysis.md