Skip to content

feat(app-preview): serve stored dataframes for dataframe() - #622

Draft
tkislan wants to merge 4 commits into
tomaskislan/app-preview-app-datafrom
tomaskislan/app-preview-dataframes
Draft

tkislan wants to merge 4 commits into
tomaskislan/app-preview-app-datafrom
tomaskislan/app-preview-dataframes

Conversation

@tkislan

@tkislan tkislan commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

@coderabbitai ignore

Summary

In the app preview, a page's Deepnote.connect().dataframe(name, { pageIndex, pageSize }) now pages through a stored dataframe instead of answering "not supported". The preview reads the frame from deepnote_dataframes/<name>/ in the app's project folder (the folder of the nearest .deepnote file at or above the app folder), in the layout deepnote-toolkit's store_dataframe(df, name) writes, and answers one page the way Deepnote's GET /v2/projects/:projectId/dataframes/:name does. When the project folder holds no frame under that name, it reads <home>/work/deepnote_dataframes/<name>/, where the deepnote-toolkit version the extension installs stores frames, so a frame stored from a notebook shows up in the preview.

The bundled copy of Deepnote's client has no dataframe() yet, so the preview adds one to it.

Changes

  • GET /v2/projects/:projectId/dataframes/:name replaces the 501 stub:
    • validates name ([A-Za-z0-9_-]{1,128}) and the query (pageIndex ≥ 0, pageSize 1–1000, no other keys) with Deepnote's schemas and 400 messages;
    • answers 404 Dataframe not found when projectId is not the preview's current project, when the app has no project, and for a missing or invalid manifest or frame file;
    • reads the project on every request, so a project added later is served after Preview App runs again.
  • Reader (storedDataframePage.node.ts, new dependency apache-arrow ^21.2.0):
    • looks in the project folder first. When that folder holds no manifest.json for the name, it reads $HOME/work/deepnote_dataframes/<name>/, where deepnote-toolkit 2.9.1 stores frames when no notebook root is set, as on the extension's servers. A broken manifest in the project folder still answers 404;
    • reads manifest.json (file, rows, batch_rows), opens the Arrow IPC file, and decodes only the record batches the page overlaps. If the manifest's file is gone, it re-reads the manifest once, since a newer store may have replaced it;
    • answers Deepnote's body, keys in order: referenceId, rowCount, columnCount, pageIndex, pageSize, pageCount, columns (name, type, timezone), rows;
    • encodes values the way Deepnote does for every type it serves: int64 and decimal as strings, NaN and infinities as null, dates, timestamps (floored to ms, Z only when zoned), times, durations, binary as base64, arrow.uuid as a UUID string, lists, structs and maps;
    • 400 for a page past the end, and 422 for an unsupported column type, a date outside the years 0–9999, lz4 compression, and zstd on an editor whose Node.js has no zlib.zstdDecompressSync (localized message asking to update the editor).
  • dataframe() for the bundled client (src/webviews/webview-side/appPreview/, built as a classic script to dist/webviews/webview-side/appPreview/dataframeShim.js):
    • The page's tag for Deepnote's client now points at the bundled copy, keeping its other attributes, and a tag for this script follows it, from /__deepnote/dataframe.js. The script needs the client loaded, and adds dataframe() only to connections made after it runs. So a client tag with defer, async or type=module, or an onload on it that connects, leaves the app without dataframe() in the preview.
    • It gives the client's connections a dataframe(name, { pageIndex, pageSize }) when they have none, so it steps aside once the bundled client has its own.
    • Each call asks POST /__deepnote/api-token for the token, the API origin and the project, as the version of Deepnote's client that adds dataframe() does. It then calls the route above and rejects with the API's message.
    • Without a project, it rejects with a message saying the app folder has no .deepnote file at or above it.
  • Removed: the warn-once "not supported" machinery and its localized string, since dataframe() was the last stub.
  • README.

Testing

  • Unit tests for the dataframe() script with a fake fetch: the requests it makes, the page it returns, a client's own dataframe() left in place, connect()'s options passed through, and each rejection. Unit tests for the session serving both scripts.

  • Unit tests for the reader against in-test frames written with apache-arrow:

    • one test per encoding rule;
    • paging across and exactly on record batch boundaries, zero-row batches and the empty frame;
    • every 404 cause, the manifest re-read, and each 422;
    • the $HOME/work fallback: read when the project folder has no frame (also with HOME unset, from the home folder Node reports), the project folder's frame preferred, and a broken project manifest answering 404.
  • The reader, route and session tests point HOME at an empty folder, so frames in the developer's own ~/work cannot change their results.

  • Unit tests read a small zstd frame written by deepnote-toolkit 2.9.1's store_dataframe (int64, float with NaN, timestamp with zone, non-ASCII string), and test the route against a real server: the body and key order, the 400 messages, the 404s, the project being read again, a non-UUID project id, and the 422 passthrough.

  • E2E, in the appPreview group: it mirrors Deepnote's own static-site E2E for dataframe(). It stores the same 200,000-row zstd frame with the E2E venv's deepnote-toolkit store_dataframe, which writes it under $HOME/work in record batches of 65,536 rows, as for a notebook. The site, in a project folder that holds no frames, pages through it 10 rows at a time:

    • first page;
    • next page;
    • page 6,554, whose rows cross the first record batch boundary at row 65,536;
    • last page, with Next disabled.

    It takes a screenshot of each step. A local run writes into the developer's own ~/work, so the frame is stored as app_preview_e2e_customers, and the test removes it afterwards, with the folders the store created. The other tests' page now calls dataframe() from an app folder that has no project, and expects the rejection for that.

  • Locally, on VS Code 1.140.0, the appPreview group passes 8 of 8.

Notes

  • Where frames come from. The deepnote-toolkit version the extension installs stores frames in <home>/work/deepnote_dataframes/, one folder that every project shares, so the preview falls back to it. Until that changes, a project's preview can serve a frame another project stored under the same name, unless its own folder has one. The fallback is meant to go once deepnote-toolkit reads DEEPNOTE_ARTIFACTS_ROOT and this extension sets it to the project folder for the servers it starts and bumps the toolkit pin. Frames stored then land in the project folder, which the preview already reads.

  • Not applied: Deepnote's protections against forged files:

    • read and decompression byte budgets;
    • the per-page limits on values, record batches and JSON size, and the column limit;
    • symbolic link containment;
    • the 429 concurrency cap.

    A corrupt Arrow file answers 500, where Deepnote answers 404.

  • projectId is compared with the preview's project, not validated as a UUID like Deepnote does, because a .deepnote file's project id can be any string.

  • A published app cannot call dataframe() yet, because Deepnote's own client does not have it. The README says so. Once a copy of Deepnote's client with dataframe() is bundled, the script can be deleted.

  • Reading zstd frames needs Node.js 22.15 or later in the editor. VS Code 1.95, the minimum, runs Node 20.

Stacked on #620.

@codecov

codecov Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.70312% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 39%. Comparing base (bdaf545) to head (9b54733).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...ks/deepnote/appPreview/storedDataframePage.node.ts 95% 4 Missing and 4 partials ⚠️
...otebooks/deepnote/appPreview/appPreviewApi.node.ts 90% 1 Missing and 1 partial ⚠️
.../webviews/webview-side/appPreview/dataframeShim.ts 0% 1 Missing ⚠️
Additional details and impacted files
@@                        Coverage Diff                        @@
##           tomaskislan/app-preview-app-data    #622    +/-   ##
=================================================================
  Coverage                                39%     39%            
=================================================================
  Files                                   834     837     +3     
  Lines                                 42147   42384   +237     
  Branches                               9255    9325    +70     
=================================================================
+ Hits                                  16618   16848   +230     
- Misses                                23431   23435     +4     
- Partials                               2098    2101     +3     
Files with missing lines Coverage Δ
...epnote/appPreview/appPreviewCommandHandler.node.ts 98% <ø> (-1%) ⬇️
...ooks/deepnote/appPreview/appPreviewSession.node.ts 88% <100%> (+2%) ⬆️
src/platform/common/utils/localize.ts 93% <100%> (-1%) ⬇️
...views/webview-side/appPreview/deepnoteDataframe.ts 100% <100%> (ø)
.../webviews/webview-side/appPreview/dataframeShim.ts 0% <0%> (ø)
...otebooks/deepnote/appPreview/appPreviewApi.node.ts 96% <90%> (-2%) ⬇️
...ks/deepnote/appPreview/storedDataframePage.node.ts 95% <95%> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@tkislan
tkislan force-pushed the tomaskislan/app-preview-dataframes branch from 2a988f0 to 8805032 Compare November 16, 2026 07:21
@tkislan
tkislan force-pushed the tomaskislan/app-preview-app-data branch from 69bba98 to 8752d8a Compare November 16, 2026 07:21
The preview's `dataframe(name, { pageIndex, pageSize })` route answered 501
and VS Code warned that it was not supported. It now reads the frame that
deepnote-toolkit's `store_dataframe(df, name)` layout keeps in
`deepnote_dataframes/<name>/` of the app's project folder (the folder of the
nearest `.deepnote` file), and answers one page in the shape Deepnote's
`GET /v2/projects/:projectId/dataframes/:name` uses: the same paging, value
encoding per Arrow type, 400/404/422 messages and parameter validation.

- The reader uses apache-arrow (new dependency), decodes only the record
  batches the page overlaps, and decompresses zstd with Node's zlib. An
  editor whose Node.js lacks zstd gets a 422 that asks to update it.
- Deepnote's forged-file protections (byte budgets, per-page value, batch
  and JSON limits, symlink containment, the 429 cap) are not applied.
- `projectId` is compared with the preview's project, not checked as a UUID,
  because a .deepnote file's project id can be any string.
- With the last 501 stub gone, the warn-once machinery for unsupported
  operations and its localized message are removed.

The toolkit version the extension installs still stores frames in
`<home>/work/deepnote_dataframes/`; storing them in the project folder needs
the deepnote-toolkit change that reads DEEPNOTE_ARTIFACTS_ROOT.
…client

The copy of Deepnote's client that the preview bundles has no dataframe()
yet, so an app could not page through its stored dataframes in the
preview. The preview now loads a small script right after the client. It
gives the client's connections a dataframe(name, { pageIndex, pageSize })
when they have none, and takes the token, the API origin and the project
from the preview's token endpoint, as the version of the client that adds
dataframe() does. Once the bundled client has its own dataframe(), the
script leaves it alone.
Mirrors Deepnote's own static-site E2E for `dataframe()`: a project folder
holds a 200,000-row zstd frame written by pyarrow (with its manifest written
by the test, as the toolkit lays it out) and a site that pages through it 10
rows at a time. The test checks the first page, the next page, page 6,554
(whose rows cross the first record batch boundary at row 65,536) and the
last page, with a screenshot of each.

The existing app preview tests now expect `dataframe()` to reject with
`Dataframe not found` instead of the 501 message, and no longer wait for the
removed "not supported" warning.
deepnote-toolkit 2.9.1 stores frames under $HOME/work when nothing sets
its notebook root, and the extension's servers set none, so a frame a
notebook stored never reached the preview. The reader now reads
$HOME/work/deepnote_dataframes/<name>/ when the project folder has no
manifest for the name. A broken manifest in the project folder still
answers 404.

The E2E stores its frame with the venv's deepnote-toolkit instead of
laying the files out in the project folder.

Remove the fallback once the extension sets a storage root the toolkit
reads.
@tkislan
tkislan force-pushed the tomaskislan/app-preview-dataframes branch from 1e841f9 to 9b54733 Compare November 18, 2026 15:52
@tkislan
tkislan force-pushed the tomaskislan/app-preview-app-data branch from 8752d8a to bdaf545 Compare November 18, 2026 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant