Feature Request: Advanced Custom Test Case Generation and Individual Test Execution

[tanmaythegreat]

There are some advanced use cases that could improve the experience significantly.

1. Run Individual Test Cases

Currently, the testing window provides a "Run All" option. It would be useful to also have an option to run a single selected test case.
Use cases:
* Quickly debugging a specific failing case.
* Avoiding unnecessary execution of all test cases when only one case needs verification.
Also i might need to debug a particular test case, via the IDE built-in debugging.

2. Advanced Custom Test Case Generator

Currently, custom test cases are useful for manually entering edge cases, which works well for small inputs.
However, for competitive programming, there are many situations where manually writing large test cases is impractical.
An advanced mode could allow users to define generated test cases, for example:

Large random arrays

Example:

• Generate an array of size 200000.
• Fill it with random values.
• Shuffle the order randomly.

Ordered arrays

Generate:

• Increasing order:

1 2 3 4 5 ...

• Decreasing order:

200000 199999 199998 ...

Constant arrays

Generate:

1 1 1 1 1 1 ...

Adversarial inputs

Allow generating special inputs such as:

• Anti-hash test cases.
• Worst-case structures for common algorithms.
• Inputs targeting specific edge cases.

The goal is not necessarily to verify correctness, but to test whether the solution respects the time limit and memory constraints.

3. Optional Expected Output

For generated test cases, it may not always be possible or practical to provide the expected output.
The expected output field could be optional.
Behavior:

• If expected output is provided:
  • Compare the program output normally.
  • Show success/failure status.
• If expected output is empty:
  • Run the program.
  • Only check execution status, runtime, and possible errors.
  • Display a neutral state instead of success/failure.

---

4. Handling Problems With Multiple Valid Answers

Some competitive programming problems allow multiple correct outputs.
It would be useful to know how the plugin currently handles such cases.
Possible future support:

• Allow custom output validators/checkers.
• Or provide a neutral verification mode when exact output matching is not possible.
  For such problems, showing a neutral result would be preferable over marking the test as failed due to output mismatch.

---

Motivation

The main goal is to make the testing environment closer to what competitive programmers use locally:

• Stress testing solutions.
• Finding TLE cases before submission.
• Testing edge cases automatically.
• Avoiding dependence on external scripts/tools.

These features would make the plugin much more powerful for advanced competitive programming workflows.

[dj258255]

Thanks for the detailed write-up — accepting all of it. Here's the plan:

First PR — testing features:

1. Run individual test cases — a per-case ▶ button next to Run All.
2. Optional expected output — cases without one run in a neutral mode: no pass/fail, just output + runtime + memory (exactly what you want for TLE/stress checks).
3. Test case generator — element count, value range, patterns (random / increasing / decreasing / constant, optional shuffle), optional first-line N, and a seed for reproducibility. Generated cases start with an empty expected output and use the neutral mode. Adversarial/anti-hash generators may come in a later iteration.
4. Multiple valid answers — leaving the expected output empty gives you the neutral mode immediately, and I'll also add a custom checker: a small program (in any supported language) that receives the input + your output and decides the verdict, reusing the plugin's existing compile/run infrastructure.

Second PR — debugging:
Debugging a specific test case with the IDE debugger is planned too. The plugin will launch your solution in debug mode with that case's input on stdin and attach the IDE's debugger, so breakpoints in your file just work. Rollout by IDE capability:

• Java/Kotlin first (IntelliJ IDEA, incl. Community) via JDWP attach.
• Node/Python next where the corresponding debugger plugin is available.
• Go/Rust/C++/Ruby debuggers don't exist inside IntelliJ IDEA itself, but the plugin runs in any JetBrains IDE — so the plan is GoLand/RustRover/CLion/RubyMine support for those.

Wrapper-style runs (LeetCode/Programmers) will be debuggable as well — the wrappers are being restructured so your code keeps its original line numbers, keeping breakpoints accurate.

I'll keep this issue open and update here as the PRs land.

[tanmaythegreat]

Here are a few additional suggestions.

Instead of adding a Run button alongside Run All, I think it would be better to place a Run button next to each individual test case.

Also, one small UI improvement: you could hide the Actual output box until the user has run the test case for the first time. It would make the interface a bit cleaner.

I would also like to better understand what you mean by:

"Multiple valid answers — leaving the expected output empty gives you the neutral mode immediately, and I'll also add a custom checker: a small program (in any supported language) that receives the input + your output and decides the verdict, reusing the plugin's existing compile/run infrastructure."

How would the checker decide the verdict?

I don't think it's possible to provide a custom checker for every competitive programming problem. There are simply too many problems across different platforms.

What I had in mind was something simpler. For problems with multiple valid outputs, the problem statement usually explicitly mentions something like:

"If there are multiple valid answers, you may print any of them."

The plugin could detect this information from the problem statement and simply treat such problems differently—for example, by displaying a neutral result instead of marking the output as correct or incorrect based on an exact match.

Did I misunderstand what you meant by the custom checker?

May be you should implement debugging in the second PR after #32 because i think it would be easier after this, I might be wrong.

[dj258255]

The first PR (#37) is merged — and all three of your suggestions made it in.

The Run button works exactly the way you sketched it: every test case card has its own ▶ in its header (the debug button will go right next to it later). The Actual output box is also hidden now until you run a case for the first time.

About the checker — you didn't misunderstand anything, I just explained it badly. Your idea was the better default, so that's what I built: the plugin now looks for phrases like "print any of them" or "in any order" in the problem statement, and when it finds them, a mismatched output shows a neutral "CHECK" state instead of FAIL. Exact matches still PASS. No setup needed.

The checker is just an optional extra on top of that, and you'd never write one per problem. It's for one specific situation: you generate a few hundred random cases to stress-test a multiple-answer problem — there's no expected output, and far too many results to check by eye. A tiny script (any of the 8 languages, template provided) reads the input and your output from stdin and prints OK or WRONG, so the plugin can judge the generated cases for you. If that situation never comes up, you can ignore the feature entirely.

One thing that grew out of your generator idea: it now fills in all parameters automatically — on LeetCode from the API's type info, on Programmers by inferring types from the example case. So for Two Sum it generates both nums and target. Stress-testing this also uncovered two real runner bugs (broken pipe on large inputs, pipe deadlocks on large outputs) — both fixed.

And agreed on the order — I'm on #32 right now, debugging comes right after. Everything ships together in one release; I'll post here when it's out.

[dj258255]

Both parts are included in the v1.7.0 update:

• Per-test-case debugging — each test case has a debug button that launches the debugger with that case's input and stops at your breakpoints. Works across Java/Kotlin (IntelliJ), Go (GoLand), Python (PyCharm), C++ (CLion), Rust (RustRover), JavaScript (WebStorm) and Ruby (RubyMine).
• Advanced test generation — pattern-based input generator (numbers, arrays, strings, graphs…) plus adversarial cases: quicksort-killer sequences, anti-hash strings (Thue–Morse — collides for any odd base mod 2^64), 2D arrays and binary trees. You can run a single case, use neutral verdicts when the expected answer is unknown, and plug in a custom checker.

Would love to hear if this matches what you had in mind — I'll close this once you confirm it works for you.