feat(runtime): Implement streaming execution output (fixes #1813)

[tinodj]

Closes #1813.

Description

This pull request addresses the lack of real-time feedback during command execution in the DatabricksRuntime.

Problem

Previously, the output of executed commands was buffered and only displayed after the entire process had completed.
For long-running tasks, this left the user without any visibility into the command's progress, making it difficult t
determine if the process was still running or had stalled.

Solution

This PR refactors the execution logic to process output as a stream. This is achieved by:

1. Introducing an onStatusUpdate callback: This function is now passed to the executionContext.execute
   method.
2. Handling Streaming Data: The callback processes output in chunks as they are generated by the remote process
3. Tracking Output Position: A new outputPosition variable tracks the portion of the stream that has already
   been sent to the UI, ensuring that only new output is displayed.

This change provides users with immediate, real-time feedback from their running commands, significantly improving t
user experience.

Testing Notes

This feature has not yet been covered by automated tests. Manual verification is required.

To Test:

1. Run a long-running command (e.g., a Python script with multiple print statements separated by time.sleep
   calls).
2. Confirm that the output appears in the VS Code output panel in real-time as it is generated, rather than all at
   once at the end.

[github-actions]

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/vscode

Inputs:

• PR number: 1814
• Commit SHA: 55336697342c3e7988e9121fccf6b2709246c98d

Checks will be approved automatically on success.

[rugpanov]

Hi @tinodj — first, a genuine apology for the long silence here. This PR (and #1813) sat since early December without a proper review; the team was stretched thin and PRs didn't get the attention they deserved. That's on us. We now have more bandwidth to work through the backlog and invest in improvements, and I really appreciate your patience — and the fact that you filed a clear issue and took a run at fixing it.

On the change itself: the intent is spot-on, but I don't think this approach can deliver streaming output, and I want to explain why so we can steer it toward something that does help.

The limitation is in the execution API, not the client. This run path goes through the legacy 1.2/commands execution API via ExecutionContext.execute → Command. The onStatusUpdate callback you're wiring up does fire on every status poll (see Command.response() in src/sdk-extensions/Command.ts), so that part works — but the callback only ever receives result.results populated once the command reaches a terminal state (Finished/Error/Cancelled). While status === "Running", results is empty, so onStatusUpdate has nothing to emit and the output still arrives all at once at the end.

You can see this in the SDK model too: the only output-shaping fields on Results are pos and truncated, which handle large results being paginated/size-capped — there's no field carrying incremental stdout produced over time. So no client-side callback wiring can surface partial output the API doesn't return. (Real streaming needs API/server-side support, which is a bigger effort on our side.)

A couple of smaller notes:

• The status poll uses exponential backoff capped at ~10s, so even if partial results were available, the cadence wouldn't be truly "real-time."
• (result: any) / (result.results as any).data drop the typing the surrounding code relies on.

Good news — there's a way to get live output today: if you run the file via the Databricks Connect run/debug mode instead of "Upload and Run File", execution happens locally through Spark Connect and streams stdout to the integrated terminal in real time. That's the path we'd recommend for interactive, long-running scripts right now, and it's the one we're investing in making frictionless.

Where I think we can still get value on this path — #1813 actually describes two problems, and the second one is fixable client-side:

1. Output doesn't appear live — blocked on API-level streaming support for the Upload-and-Run path; can't be solved from the extension today (use the Databricks Connect mode above in the meantime).
2. "If the script fails with an exception, none of the printed output appears" — this is a separate, real bug. On failure the API returns only the traceback and the earlier print output is dropped; we can capture stdout in resources/python/bootstrap.py and surface it alongside the error.

Also worth noting: this is a duplicate of the older #1325, which reports the same thing — we'll consolidate those.