Skip to content

Commit 0048fa5

Browse files
committed
Merge remote-tracking branch 'origin/staging' into fix/elasticsearch-cloud-id-host
# Conflicts: # apps/sim/tools/generated/tool-metadata.ts # apps/sim/tools/generated/tool-outputs.ts
2 parents 823dc08 + f8f2d5f commit 0048fa5

171 files changed

Lines changed: 32646 additions & 961 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

apps/docs/content/docs/integrations/file.mdx

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -87,15 +87,16 @@ Fetch and parse a file from a URL with optional custom headers.
8787

8888
### File Write
8989

90-
Create a new workspace file. If a file with the same name already exists, a numeric suffix is added (e.g., "data (1).csv").
90+
Create a new workspace file, either from text content or from an existing file. If a file with the same name already exists, a numeric suffix is added (e.g., "data (1).csv").
9191

9292
#### Input
9393

9494
| Parameter | Type | Required | Description |
9595
| --------- | ---- | -------- | ----------- |
96-
| `fileName` | string | Yes | File name \(e.g., "data.csv"\). If a file with this name exists, a numeric suffix is added automatically. |
97-
| `content` | string | Yes | The text content to write to the file. |
98-
| `contentType` | string | No | MIME type for new files \(e.g., "text/plain"\). Auto-detected from file extension if omitted. |
96+
| `fileName` | string | No | File name \(e.g., "data.csv"\). Required when writing text; optional when storing a file, which keeps its own name unless this overrides it. If the name already exists, a numeric suffix is added automatically. |
97+
| `content` | string | No | The text content to write to the file. Provide exactly one of content or fileInput. |
98+
| `fileInput` | file | No | An existing file to store in the workspace, such as one produced by an earlier tool. Use this for anything that is not text — PDFs, images, audio, archives. Provide exactly one of content or fileInput. |
99+
| `contentType` | string | No | MIME type for new files \(e.g., "text/plain"\). Auto-detected from the file extension, or taken from the stored file, if omitted. |
99100

100101
#### Output
101102

apps/docs/content/docs/tables/index.mdx

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,13 +24,16 @@ Every column has a type, which decides how its values are stored and validated.
2424
| **Currency** | An amount in a currency you pick per column | `$1,234.56` |
2525
| **Boolean** | `true` or `false` | `true` |
2626
| **Date** | A date | `2026-03-16` |
27+
| **Expiration** | An absolute row expiration time, stored as Unix epoch seconds (seconds since January 1, 1970 UTC) | `1773671400` |
2728
| **JSON** | An object or array | `{ "tier": "pro" }` |
2829
| **Select** | One of a fixed set of options, or several | `Pro` |
2930

3031
Types are enforced as you enter values, so a Number column only takes numbers.
3132

3233
A Currency column stores a plain number and renders it in the currency you choose for that column, so filters, sorts, and exports all see the amount itself. Changing a column's currency relabels it — it does not convert the amounts.
3334

35+
A table can have one Expiration column. Adding it enables row expiration; rows with a non-empty expiration value become eligible for deletion after that time passes. Cleanup runs periodically, so actual row removal may happen after the expiration timestamp rather than exactly at it. Deleting the Expiration column disables expiration for the table. Expiration cells use the date editor, while APIs and workflows read and write integer Unix epoch seconds.
36+
3437
## Editing a table
3538

3639
Open the **Tables** section in the sidebar and click **New table** to create one. Add columns from the column header, type into a cell to edit it, and paste rows from a spreadsheet to bulk-load. Filter and sort from the toolbar without changing the underlying data. The editor has full keyboard support; see [keyboard shortcuts](/keyboard-shortcuts).

apps/docs/content/docs/workflows/blocks/function.mdx

Lines changed: 45 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,49 @@ Sim supplies the rendered heredoc privately while preserving the quoted delimite
102102
| --- | --- |
103103
| `<function.result>` | The value your code returns (object, array, string, number, …) |
104104
| `<function.stdout>` | Anything printed with `console.log()` or `print()` |
105+
| `<function.files>` | Files your code wrote to `/tmp/sim/outputs`, ready to attach or upload |
106+
107+
## Files
108+
109+
**Reading.** Reference a file's `path` and it is mounted for you:
110+
111+
```python
112+
import pandas as pd
113+
114+
frame = pd.read_csv(<gmail.attachments[0].path>)
115+
frame.describe().to_csv('/tmp/sim/outputs/summary.csv')
116+
```
117+
118+
`.path` resolves to the file's location on the sandbox filesystem, so any language
119+
can open it — pandas, ffmpeg, a CLI. It is the counterpart to `.base64`, which
120+
inlines the contents instead and works only in JavaScript. Both appear in the
121+
reference dropdown next to `.name` and `.size`.
122+
123+
**Writing.** Anything your code writes to `/tmp/sim/outputs` comes back as
124+
`<function.files>`, a list of file objects any file-accepting block takes directly —
125+
attach them to an email, upload them to storage, or save them to the workspace with
126+
the File block. There is nothing to turn on.
127+
128+
The one exception is a call that names an explicit `outputSandboxPath`. That asks
129+
for particular paths to be exported and answers with that export's own result, so
130+
the output directory is not harvested alongside it — choose one or the other rather
131+
than expecting both in the same run.
132+
133+
<Callout type="info">
134+
Referencing `.path` runs the block in the remote sandbox, since the local
135+
JavaScript VM has no filesystem — expect the slower start of a remote run even for
136+
plain JavaScript. Referencing the file itself (`<gmail.attachments[0]>`, `.name`,
137+
`.url`) does not, and stays local. Up to 20 files come back per run, 50MB total,
138+
nested no more than 11 directories deep; a run that exceeds any of these fails
139+
rather than returning part of what your code wrote.
140+
</Callout>
141+
142+
<Callout type="warn">
143+
Returned files live with the execution rather than in your workspace, and a text
144+
file containing a resolved secret value is refused rather than returned — there is
145+
nowhere on an execution file to record that it carries one. Write such a file to a
146+
workspace path instead, or keep the secret out of the output.
147+
</Callout>
105148

106149
## Language
107150

@@ -401,8 +444,8 @@ The lazy `sim.files` and `sim.values` helpers are available only in JavaScript f
401444
{ question: "What languages does the Function block support?", answer: "JavaScript, Python, and Shell. JavaScript is the default. Python remains a stable saved language choice; Shell and custom Sandbox controls appear when a remote sandbox provider is enabled. Python and Shell execution require that provider." },
402445
{ question: "When does code run locally vs. in a sandbox?", answer: "JavaScript without external imports runs in a local isolated sandbox for speed. JavaScript that uses import or require, Python, and Shell run in the configured remote sandbox." },
403446
{ question: "Does JavaScript still work without E2B or Daytona?", answer: "Yes. JavaScript without import or require runs in Sim's local isolated VM and does not require a remote provider. JavaScript with external imports, Python, Shell, and custom Sandboxes require E2B or Daytona and fail explicitly when it is unavailable." },
404-
{ question: "How do I reference outputs from other blocks inside my code?", answer: "Use angle-bracket syntax directly, like <agent.content> or <api.data>, with no quotes around the tag — Sim replaces it with the real value before execution. For environment variables, use double curly braces: {{API_KEY}}." },
405-
{ question: "What does the Function block return?", answer: "Two outputs: result and stdout. Use return in JavaScript, assign __sim_result__ in Python, or print an __SIM_RESULT__= marker in Shell to set result. Ordinary console, print, and command output goes to stdout." },
447+
{ question: "How do I reference outputs from other blocks inside my code?", answer: "Use angle-bracket syntax directly, like <agent.content> or <api.data>, with no quotes around the tag — Sim replaces it with the real value before execution. For environment variables, use double curly braces: {{API_KEY}}. To read a file, reference its path — <gmail.attachments[0].path> mounts it and resolves to a location any language can open." },
448+
{ question: "What does the Function block return?", answer: "Three outputs: result, stdout, and files. Use return in JavaScript, assign __sim_result__ in Python, or print an __SIM_RESULT__= marker in Shell to set result. Ordinary console, print, and command output goes to stdout. Anything your code writes to /tmp/sim/outputs comes back in files as a file object later blocks can accept directly." },
406449
{ question: "Can I make HTTP requests from a Function block?", answer: "Yes. fetch() is available in JavaScript with async/await. In Python, use requests or httpx. In Shell, use curl or a CLI available on the selected sandbox." },
407450
{ question: "Is there a timeout for Function block execution?", answer: "Yes, a configurable execution timeout. If your code exceeds it, the run is terminated and the block reports an error. Keep this in mind for external calls or heavy processing." },
408451
]} />

apps/docs/openapi-v2-tables.json

Lines changed: 81 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -4979,7 +4979,16 @@
49794979
},
49804980
"type": {
49814981
"type": "string",
4982-
"enum": ["string", "number", "currency", "boolean", "date", "json", "select"],
4982+
"enum": [
4983+
"string",
4984+
"number",
4985+
"currency",
4986+
"boolean",
4987+
"date",
4988+
"ttl",
4989+
"json",
4990+
"select"
4991+
],
49834992
"description": "Data type of values stored in the column."
49844993
},
49854994
"required": {
@@ -5257,7 +5266,16 @@
52575266
},
52585267
"type": {
52595268
"type": "string",
5260-
"enum": ["string", "number", "currency", "boolean", "date", "json", "select"],
5269+
"enum": [
5270+
"string",
5271+
"number",
5272+
"currency",
5273+
"boolean",
5274+
"date",
5275+
"ttl",
5276+
"json",
5277+
"select"
5278+
],
52615279
"description": "Column data type."
52625280
},
52635281
"required": {
@@ -5436,7 +5454,16 @@
54365454
},
54375455
"type": {
54385456
"type": "string",
5439-
"enum": ["string", "number", "currency", "boolean", "date", "json", "select"],
5457+
"enum": [
5458+
"string",
5459+
"number",
5460+
"currency",
5461+
"boolean",
5462+
"date",
5463+
"ttl",
5464+
"json",
5465+
"select"
5466+
],
54405467
"description": "Data type of values stored in the column."
54415468
},
54425469
"required": {
@@ -5536,7 +5563,16 @@
55365563
},
55375564
"type": {
55385565
"type": "string",
5539-
"enum": ["string", "number", "currency", "boolean", "date", "json", "select"],
5566+
"enum": [
5567+
"string",
5568+
"number",
5569+
"currency",
5570+
"boolean",
5571+
"date",
5572+
"ttl",
5573+
"json",
5574+
"select"
5575+
],
55405576
"description": "Column data type."
55415577
},
55425578
"required": {
@@ -5633,7 +5669,7 @@
56335669
"type": {
56345670
"description": "Replacement column data type.",
56355671
"type": "string",
5636-
"enum": ["string", "number", "currency", "boolean", "date", "json", "select"]
5672+
"enum": ["string", "number", "currency", "boolean", "date", "ttl", "json", "select"]
56375673
},
56385674
"required": {
56395675
"description": "Whether inserts must supply a value for this column.",
@@ -7397,7 +7433,16 @@
73977433
},
73987434
"type": {
73997435
"type": "string",
7400-
"enum": ["string", "number", "currency", "boolean", "date", "json", "select"],
7436+
"enum": [
7437+
"string",
7438+
"number",
7439+
"currency",
7440+
"boolean",
7441+
"date",
7442+
"ttl",
7443+
"json",
7444+
"select"
7445+
],
74017446
"description": "Data type of values stored in the column."
74027447
},
74037448
"required": {
@@ -7597,7 +7642,16 @@
75977642
},
75987643
"type": {
75997644
"type": "string",
7600-
"enum": ["string", "number", "currency", "boolean", "date", "json", "select"],
7645+
"enum": [
7646+
"string",
7647+
"number",
7648+
"currency",
7649+
"boolean",
7650+
"date",
7651+
"ttl",
7652+
"json",
7653+
"select"
7654+
],
76017655
"description": "Output column data type."
76027656
},
76037657
"required": {
@@ -7738,7 +7792,16 @@
77387792
},
77397793
"type": {
77407794
"type": "string",
7741-
"enum": ["string", "number", "currency", "boolean", "date", "json", "select"],
7795+
"enum": [
7796+
"string",
7797+
"number",
7798+
"currency",
7799+
"boolean",
7800+
"date",
7801+
"ttl",
7802+
"json",
7803+
"select"
7804+
],
77427805
"description": "Output column data type."
77437806
},
77447807
"required": {
@@ -7856,7 +7919,16 @@
78567919
},
78577920
"type": {
78587921
"type": "string",
7859-
"enum": ["string", "number", "currency", "boolean", "date", "json", "select"],
7922+
"enum": [
7923+
"string",
7924+
"number",
7925+
"currency",
7926+
"boolean",
7927+
"date",
7928+
"ttl",
7929+
"json",
7930+
"select"
7931+
],
78607932
"description": "Data type of values stored in the column."
78617933
},
78627934
"required": {

apps/sim/.env.example

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,6 +201,7 @@ CRON_SECRET=your_cron_secret # Use `openssl rand -hex 32` to generate. Authentic
201201
# DATA_DRAINS_ENABLED= / NEXT_PUBLIC_DATA_DRAINS_ENABLED= # Export streams
202202
# FORKING_ENABLED= # Workspace forks
203203
# CREDENTIAL_GROUPS= # Enterprise managed OAuth collections
204+
# TABLE_ROW_TTL= # Table TTL columns and expired-row cleanup
204205
# ORGANIZATIONS_ENABLED= / NEXT_PUBLIC_ORGANIZATIONS_ENABLED= # Organizations only
205206

206207
# Instance organization (Optional). Most enterprise features read their settings from the
Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
/**
2+
* @vitest-environment node
3+
*/
4+
import { createMockRequest } from '@sim/testing'
5+
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
6+
7+
const { mockEnqueue, mockGetJobQueue, mockIsTableRowTtlEnabled, mockVerifyCronAuth } = vi.hoisted(
8+
() => ({
9+
mockEnqueue: vi.fn(),
10+
mockGetJobQueue: vi.fn(),
11+
mockIsTableRowTtlEnabled: vi.fn(),
12+
mockVerifyCronAuth: vi.fn(),
13+
})
14+
)
15+
16+
vi.mock('@/lib/auth/internal', () => ({ verifyCronAuth: mockVerifyCronAuth }))
17+
vi.mock('@/lib/core/async-jobs', () => ({ getJobQueue: mockGetJobQueue }))
18+
vi.mock('@/lib/table/ttl-availability', () => ({
19+
isTableRowTtlEnabled: mockIsTableRowTtlEnabled,
20+
}))
21+
22+
import { GET } from '@/app/api/cron/cleanup-table-row-ttl/route'
23+
24+
describe('table row TTL cleanup route', () => {
25+
beforeEach(() => {
26+
vi.clearAllMocks()
27+
vi.useFakeTimers()
28+
vi.setSystemTime(new Date('2026-08-22T17:01:00Z'))
29+
mockVerifyCronAuth.mockReturnValue(null)
30+
mockIsTableRowTtlEnabled.mockResolvedValue(true)
31+
mockEnqueue.mockResolvedValue('job-ttl-1')
32+
mockGetJobQueue.mockResolvedValue({ enqueue: mockEnqueue })
33+
})
34+
35+
afterEach(() => {
36+
vi.useRealTimers()
37+
})
38+
39+
it('enqueues one serialized cleanup job', async () => {
40+
const response = await GET(
41+
createMockRequest(
42+
'GET',
43+
undefined,
44+
{},
45+
'http://localhost:3000/api/cron/cleanup-table-row-ttl'
46+
)
47+
)
48+
49+
expect(response.status).toBe(200)
50+
await expect(response.json()).resolves.toEqual({ triggered: true, jobId: 'job-ttl-1' })
51+
expect(mockEnqueue).toHaveBeenCalledWith(
52+
'cleanup-table-row-ttl',
53+
{},
54+
expect.objectContaining({
55+
maxAttempts: 1,
56+
jobId: 'cleanup-table-row-ttl:1986020',
57+
concurrencyKey: 'cleanup:table-row-ttl',
58+
concurrencyLimit: 1,
59+
runner: expect.any(Function),
60+
})
61+
)
62+
})
63+
64+
it('deduplicates retries within the same fifteen-minute schedule window', async () => {
65+
const request = () =>
66+
createMockRequest(
67+
'GET',
68+
undefined,
69+
{},
70+
'http://localhost:3000/api/cron/cleanup-table-row-ttl'
71+
)
72+
73+
await GET(request())
74+
vi.advanceTimersByTime(13 * 60 * 1000)
75+
await GET(request())
76+
77+
expect(mockEnqueue.mock.calls[0]?.[2]?.jobId).toBe(mockEnqueue.mock.calls[1]?.[2]?.jobId)
78+
})
79+
80+
it('uses a new id immediately after the next fifteen-minute window begins', async () => {
81+
const request = () =>
82+
createMockRequest(
83+
'GET',
84+
undefined,
85+
{},
86+
'http://localhost:3000/api/cron/cleanup-table-row-ttl'
87+
)
88+
89+
vi.setSystemTime(new Date('2026-08-22T17:14:59.999Z'))
90+
await GET(request())
91+
vi.setSystemTime(new Date('2026-08-22T17:15:00.000Z'))
92+
await GET(request())
93+
94+
expect(mockEnqueue.mock.calls[0]?.[2]?.jobId).not.toBe(mockEnqueue.mock.calls[1]?.[2]?.jobId)
95+
})
96+
97+
it('returns the cron auth refusal without touching the queue', async () => {
98+
mockVerifyCronAuth.mockReturnValue(new Response(null, { status: 401 }))
99+
100+
const response = await GET(
101+
createMockRequest(
102+
'GET',
103+
undefined,
104+
{},
105+
'http://localhost:3000/api/cron/cleanup-table-row-ttl'
106+
)
107+
)
108+
109+
expect(response.status).toBe(401)
110+
expect(mockGetJobQueue).not.toHaveBeenCalled()
111+
})
112+
113+
it('does not enqueue cleanup while the feature is disabled', async () => {
114+
mockIsTableRowTtlEnabled.mockResolvedValue(false)
115+
116+
const response = await GET(
117+
createMockRequest(
118+
'GET',
119+
undefined,
120+
{},
121+
'http://localhost:3000/api/cron/cleanup-table-row-ttl'
122+
)
123+
)
124+
125+
expect(response.status).toBe(200)
126+
await expect(response.json()).resolves.toEqual({
127+
triggered: false,
128+
reason: 'feature-disabled',
129+
})
130+
expect(mockGetJobQueue).not.toHaveBeenCalled()
131+
})
132+
})

0 commit comments

Comments
 (0)