Build tooling to guarantee correct output from interactive redis-cli commands by paoloredis · Pull Request #3643 · redis/docs

paoloredis · 2026-07-15T13:25:47Z

Note

Low Risk
Changes are mostly documentation transcripts and a new optional CI script that calls the public redis.io CLI API; no runtime product or auth paths are modified.

Overview
Adds build/check_tryit_output.py, a CI-oriented verifier that parses clients-example transcripts, runs each page’s commands in one shared redis.io/cli session (batched to 20 commands), formats API replies like static/js/cli.js, and compares them to documented output. ERROR (command fails when docs expect real output) fails the build; MISMATCH is reported but only gates with --strict. It handles unordered commands, skips non-deterministic ones, and can mask volatile values (stream IDs, timestamps).

Across command and develop docs, example transcripts are updated so they match live redis-cli text: unquoted OK for simple strings, quoted bulk strings, nested array layout (e.g. ARSCAN, ARGREP, CMS.INCRBY), fixed typos/corruption in XADD examples, DEL setup in ZRANGE / ARGREP / SDIFF for isolated runs, fuller JSON wildcard/filter output, expanded TS.INFO blocks, runnable="false" on a heavy TS.MRANGE example, and small numeric/format fixes (t-digest, FT.SEARCH, lists DEL return values).

^{Reviewed by Cursor Bugbot for commit 9b38047. Bugbot is set up for automated code reviews on this repo. Configure here.}

…commands

CLAassistant · 2026-07-15T13:25:56Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ andy-stark-redis
❌ paoloredis
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

github-actions · 2026-07-15T13:26:02Z

DOC-6844

github-actions · 2026-07-15T13:26:08Z

paoloredis · 2026-07-15T13:29:41Z

The command outputs embedded in the Markdown docs are the source of truth, each example records exactly what Redis should return. This script replays those commands against redis.io/cli and checks that the live output still matches the docs, so you can catch stale or broken examples.

Some examples won't match right now and that's expected for commands for which the output is random (e.g. SRANDMEMBER)

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7fb1cec. Configure here.}

andy-stark-redis · 2026-07-15T15:03:20Z

I posted this a while ago but unfortunately, I added it to the wrong PR :-( </age>

Anyway, among other mismatches, I'm getting:

MISMATCH  content/develop/data-types/vector-sets/_index.md  [vecset_tutorial/vsim_options]
    cmd:      VSIM points ELE pt:A WITHSCORES COUNT 4
    expected: 1) "pt:A"
              2) "1"
              3) "pt:E"
              4) "0.8535534143447876"
              5) "pt:C"
              6) "0.5"
              7) "pt:D"
              8) "0.5"
    actual:   1) "pt:A"
              2) "1"
              3) "pt:E"
              4) "0.8535534143447876"
              5) "pt:D"
              6) "0.5"
              7) "pt:C"
              8) "0.5"

Looks like this is down to non-deterministic ordering rather than an actual error.

paoloredis · 2026-07-15T15:33:40Z

I posted this a while ago but unfortunately, I added it to the wrong PR :-( </age>

Anyway, among other mismatches, I'm getting:

MISMATCH  content/develop/data-types/vector-sets/_index.md  [vecset_tutorial/vsim_options]
    cmd:      VSIM points ELE pt:A WITHSCORES COUNT 4
    expected: 1) "pt:A"
              2) "1"
              3) "pt:E"
              4) "0.8535534143447876"
              5) "pt:C"
              6) "0.5"
              7) "pt:D"
              8) "0.5"
    actual:   1) "pt:A"
              2) "1"
              3) "pt:E"
              4) "0.8535534143447876"
              5) "pt:D"
              6) "0.5"
              7) "pt:C"
              8) "0.5"

Looks like this is down to non-deterministic ordering rather than an actual error.

I've added VSIM to the commands for which the order is not relevant

andy-stark-redis · 2026-07-16T09:57:28Z

@paoloredis A few more issues:

I think the mismatch with the FT.SEARCH example is probably also just down to the result ordering (so this will presumably affect FT.AGGREGATE too, even though the error isn't currently showing up). I guess that after deployment we'll have to just add offending commands to the list as we find them?
The mismatch for content/develop/data-types/sets.md [sets_tutorial/srem] happens because it uses non-deterministic commands. This example isn't really checkable (and we don't check it in the client code either), so we should skip this if we'll be using this script as a Hugo build check or a Github action on each PR.
Some of the commands in UNORDERED_CMDS aren't correct because they have fully non-deterministic output rather than non-deterministic ordering. These are: SRANDMEMBER, SPOP, ZRANDMEMBER, HRANDFIELD. An alternative check might be to auto-skip whole examples that include these commands or warn if they've been used in examples that aren't manually skipped.

Aside from these things, the script catches some genuine, fixable mistake already - really useful!

paoloredis · 2026-07-16T12:16:02Z

@paoloredis A few more issues:

* I think the mismatch with the `FT.SEARCH` example is probably also just down to the result ordering (so this will presumably affect `FT.AGGREGATE` too, even though the error isn't currently showing up). I guess that after deployment we'll have to just add offending commands to the list as we find them?

* The mismatch for `content/develop/data-types/sets.md  [sets_tutorial/srem]` happens because it uses non-deterministic commands. This example isn't really checkable (and we don't check it in the client code either), so we should skip this if we'll be using this script as a Hugo build check or a Github action on each PR.

* Some of the commands in `UNORDERED_CMDS` aren't correct because they have fully non-deterministic output rather than non-deterministic ordering. These are: `SRANDMEMBER`, `SPOP`, `ZRANDMEMBER`, `HRANDFIELD`. An alternative check might be to auto-skip whole examples that include these commands or warn if they've been used in examples that aren't manually skipped.

Aside from these things, the script catches some genuine, fixable mistake already - really useful!

Thanks, I've now differentiated between unordered and non deterministic cmds

andy-stark-redis

The checker gives a clear run now, so LGTM.

paoloredis · 2026-07-20T08:46:52Z

@dwdougherty what do you think?

dwdougherty

I haven't had a chance to look at this yet, but if @andy-stark-redis says it's good, then that's good enough for me.

Build tooling to guarantee correct output from interactive redis-cli …

63244e7

…commands

cursor Bot reviewed Jul 15, 2026

View reviewed changes

Comment thread build/check_tryit_output.py Outdated

Comment thread build/check_tryit_output.py

Comment thread build/check_tryit_output.py

Fix typo

7fb1cec

cursor Bot reviewed Jul 15, 2026

View reviewed changes

Comment thread build/check_tryit_output.py

paoloredis requested review from andy-stark-redis and dwdougherty July 15, 2026 13:38

Add VSIM to commands for which order does not matter

2b6c591

Address bugbot comment

b069fdf

Differentiate between unordered cmds and non deterministic cmds

95fb3ec

Fix remaining dodgy CLI examples in data type pages

9b38047

andy-stark-redis approved these changes Jul 17, 2026

View reviewed changes

dwdougherty approved these changes Jul 20, 2026

View reviewed changes

paoloredis merged commit a53b029 into main Jul 20, 2026
77 of 78 checks passed

paoloredis deleted the DOC-6844 branch July 20, 2026 14:12

Uh oh!

Conversation

paoloredis commented Jul 15, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 15, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

paoloredis commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andy-stark-redis commented Jul 15, 2026

Uh oh!

paoloredis commented Jul 15, 2026

Uh oh!

andy-stark-redis commented Jul 16, 2026

Uh oh!

paoloredis commented Jul 16, 2026

Uh oh!

andy-stark-redis left a comment

Choose a reason for hiding this comment

Uh oh!

paoloredis commented Jul 20, 2026

Uh oh!

dwdougherty left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

paoloredis commented Jul 15, 2026 •

edited by cursor Bot

Loading

CLAassistant commented Jul 15, 2026 •

edited

Loading

github-actions Bot commented Jul 15, 2026 •

edited by atlassian Bot

Loading

github-actions Bot commented Jul 15, 2026 •

edited

Loading

paoloredis commented Jul 15, 2026 •

edited

Loading