Repository navigation
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## tomaskislan/app-preview-app-data #622 +/- ##
=================================================================
Coverage 39% 39%
=================================================================
Files 834 837 +3
Lines 42147 42384 +237
Branches 9255 9325 +70
=================================================================
+ Hits 16618 16848 +230
- Misses 23431 23435 +4
- Partials 2098 2101 +3
🚀 New features to boost your workflow:
|
tkislan
force-pushed
the
tomaskislan/app-preview-dataframes
branch
from
November 16, 2026 07:21
2a988f0 to
8805032
Compare
tkislan
force-pushed
the
tomaskislan/app-preview-app-data
branch
from
November 16, 2026 07:21
69bba98 to
8752d8a
Compare
The preview's `dataframe(name, { pageIndex, pageSize })` route answered 501
and VS Code warned that it was not supported. It now reads the frame that
deepnote-toolkit's `store_dataframe(df, name)` layout keeps in
`deepnote_dataframes/<name>/` of the app's project folder (the folder of the
nearest `.deepnote` file), and answers one page in the shape Deepnote's
`GET /v2/projects/:projectId/dataframes/:name` uses: the same paging, value
encoding per Arrow type, 400/404/422 messages and parameter validation.
- The reader uses apache-arrow (new dependency), decodes only the record
batches the page overlaps, and decompresses zstd with Node's zlib. An
editor whose Node.js lacks zstd gets a 422 that asks to update it.
- Deepnote's forged-file protections (byte budgets, per-page value, batch
and JSON limits, symlink containment, the 429 cap) are not applied.
- `projectId` is compared with the preview's project, not checked as a UUID,
because a .deepnote file's project id can be any string.
- With the last 501 stub gone, the warn-once machinery for unsupported
operations and its localized message are removed.
The toolkit version the extension installs still stores frames in
`<home>/work/deepnote_dataframes/`; storing them in the project folder needs
the deepnote-toolkit change that reads DEEPNOTE_ARTIFACTS_ROOT.
…client
The copy of Deepnote's client that the preview bundles has no dataframe()
yet, so an app could not page through its stored dataframes in the
preview. The preview now loads a small script right after the client. It
gives the client's connections a dataframe(name, { pageIndex, pageSize })
when they have none, and takes the token, the API origin and the project
from the preview's token endpoint, as the version of the client that adds
dataframe() does. Once the bundled client has its own dataframe(), the
script leaves it alone.
Mirrors Deepnote's own static-site E2E for `dataframe()`: a project folder holds a 200,000-row zstd frame written by pyarrow (with its manifest written by the test, as the toolkit lays it out) and a site that pages through it 10 rows at a time. The test checks the first page, the next page, page 6,554 (whose rows cross the first record batch boundary at row 65,536) and the last page, with a screenshot of each. The existing app preview tests now expect `dataframe()` to reject with `Dataframe not found` instead of the 501 message, and no longer wait for the removed "not supported" warning.
deepnote-toolkit 2.9.1 stores frames under $HOME/work when nothing sets its notebook root, and the extension's servers set none, so a frame a notebook stored never reached the preview. The reader now reads $HOME/work/deepnote_dataframes/<name>/ when the project folder has no manifest for the name. A broken manifest in the project folder still answers 404. The E2E stores its frame with the venv's deepnote-toolkit instead of laying the files out in the project folder. Remove the fallback once the extension sets a storage root the toolkit reads.
tkislan
force-pushed
the
tomaskislan/app-preview-dataframes
branch
from
November 18, 2026 15:52
1e841f9 to
9b54733
Compare
tkislan
force-pushed
the
tomaskislan/app-preview-app-data
branch
from
November 18, 2026 15:52
8752d8a to
bdaf545
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@coderabbitai ignore
Summary
In the app preview, a page's
Deepnote.connect().dataframe(name, { pageIndex, pageSize })now pages through a stored dataframe instead of answering "not supported". The preview reads the frame fromdeepnote_dataframes/<name>/in the app's project folder (the folder of the nearest.deepnotefile at or above the app folder), in the layout deepnote-toolkit'sstore_dataframe(df, name)writes, and answers one page the way Deepnote'sGET /v2/projects/:projectId/dataframes/:namedoes. When the project folder holds no frame under that name, it reads<home>/work/deepnote_dataframes/<name>/, where the deepnote-toolkit version the extension installs stores frames, so a frame stored from a notebook shows up in the preview.The bundled copy of Deepnote's client has no
dataframe()yet, so the preview adds one to it.Changes
GET /v2/projects/:projectId/dataframes/:namereplaces the 501 stub:name([A-Za-z0-9_-]{1,128}) and the query (pageIndex≥ 0,pageSize1–1000, no other keys) with Deepnote's schemas and 400 messages;Dataframe not foundwhenprojectIdis not the preview's current project, when the app has no project, and for a missing or invalid manifest or frame file;storedDataframePage.node.ts, new dependencyapache-arrow^21.2.0):manifest.jsonfor the name, it reads$HOME/work/deepnote_dataframes/<name>/, where deepnote-toolkit 2.9.1 stores frames when no notebook root is set, as on the extension's servers. A broken manifest in the project folder still answers 404;manifest.json(file,rows,batch_rows), opens the Arrow IPC file, and decodes only the record batches the page overlaps. If the manifest's file is gone, it re-reads the manifest once, since a newer store may have replaced it;referenceId,rowCount,columnCount,pageIndex,pageSize,pageCount,columns(name,type,timezone),rows;int64anddecimalas strings, NaN and infinities asnull, dates, timestamps (floored to ms,Zonly when zoned), times, durations, binary as base64,arrow.uuidas a UUID string, lists, structs and maps;zlib.zstdDecompressSync(localized message asking to update the editor).dataframe()for the bundled client (src/webviews/webview-side/appPreview/, built as a classic script todist/webviews/webview-side/appPreview/dataframeShim.js):/__deepnote/dataframe.js. The script needs the client loaded, and addsdataframe()only to connections made after it runs. So a client tag withdefer,asyncortype=module, or anonloadon it that connects, leaves the app withoutdataframe()in the preview.dataframe(name, { pageIndex, pageSize })when they have none, so it steps aside once the bundled client has its own.POST /__deepnote/api-tokenfor the token, the API origin and the project, as the version of Deepnote's client that addsdataframe()does. It then calls the route above and rejects with the API's message..deepnotefile at or above it.dataframe()was the last stub.Testing
Unit tests for the
dataframe()script with a fakefetch: the requests it makes, the page it returns, a client's owndataframe()left in place,connect()'s options passed through, and each rejection. Unit tests for the session serving both scripts.Unit tests for the reader against in-test frames written with apache-arrow:
$HOME/workfallback: read when the project folder has no frame (also withHOMEunset, from the home folder Node reports), the project folder's frame preferred, and a broken project manifest answering 404.The reader, route and session tests point
HOMEat an empty folder, so frames in the developer's own~/workcannot change their results.Unit tests read a small zstd frame written by deepnote-toolkit 2.9.1's
store_dataframe(int64, float with NaN, timestamp with zone, non-ASCII string), and test the route against a real server: the body and key order, the 400 messages, the 404s, the project being read again, a non-UUID project id, and the 422 passthrough.E2E, in the
appPreviewgroup: it mirrors Deepnote's own static-site E2E fordataframe(). It stores the same 200,000-row zstd frame with the E2E venv's deepnote-toolkitstore_dataframe, which writes it under$HOME/workin record batches of 65,536 rows, as for a notebook. The site, in a project folder that holds no frames, pages through it 10 rows at a time:It takes a screenshot of each step. A local run writes into the developer's own
~/work, so the frame is stored asapp_preview_e2e_customers, and the test removes it afterwards, with the folders the store created. The other tests' page now callsdataframe()from an app folder that has no project, and expects the rejection for that.Locally, on VS Code 1.140.0, the
appPreviewgroup passes 8 of 8.Notes
Where frames come from. The deepnote-toolkit version the extension installs stores frames in
<home>/work/deepnote_dataframes/, one folder that every project shares, so the preview falls back to it. Until that changes, a project's preview can serve a frame another project stored under the same name, unless its own folder has one. The fallback is meant to go once deepnote-toolkit readsDEEPNOTE_ARTIFACTS_ROOTand this extension sets it to the project folder for the servers it starts and bumps the toolkit pin. Frames stored then land in the project folder, which the preview already reads.Not applied: Deepnote's protections against forged files:
A corrupt Arrow file answers 500, where Deepnote answers 404.
projectIdis compared with the preview's project, not validated as a UUID like Deepnote does, because a.deepnotefile's project id can be any string.A published app cannot call
dataframe()yet, because Deepnote's own client does not have it. The README says so. Once a copy of Deepnote's client withdataframe()is bundled, the script can be deleted.Reading zstd frames needs Node.js 22.15 or later in the editor. VS Code 1.95, the minimum, runs Node 20.
Stacked on #620.