diff --git a/clang_flag_explorer.md b/clang_flag_explorer.md index 689004f1..d059b875 100644 --- a/clang_flag_explorer.md +++ b/clang_flag_explorer.md @@ -25,8 +25,8 @@ Requirements: - Do not use identical or illustrative before/after assembly as a placeholder. If the flag acts in preprocessing, diagnostics, language selection, or debug metadata, show the appropriate observable artifact (`-E`, diagnostics, `-###`, IR, or object sections) and explain why machine instructions are unchanged. - Keep `SourceCode` examples genuinely multiline and readable in the rendered page; do not pass a long one-line JSX string when formatted source is required. - **Why use it / when not to use it**: include ABI, portability, correctness, and build-consistency warnings where relevant. - - **Performance impact**: explain qualitative tradeoffs. Use `PerfReport` only with real measurements obtained by actually running the [CompilerSutra Perf tool](https://github.com/CompilerSutra/CompilerSutraPerfTool) or its documented CLI. Include the without/with arrays, workload, machine, compiler, trial setup, limitations, and exact reproduction commands. Run the commands in the workspace before reporting any numbers; never invent, estimate, or copy placeholder measurements. If the tool or workload cannot be run, say so and omit `PerfReport` data. - - Link the `CompilerSutra Perf` name to https://pypi.org/project/compilersutra-perf/ whenever measurements are reported. + - **Performance impact**: explain qualitative tradeoffs. Use `PerfReport` only with real measurements obtained by actually running [CompilerSutraPerf](https://www.compilersutra.com/docs/project/compilersutra-perf/) (`csperf`, [PyPI](https://pypi.org/project/compilersutra-perf/)) or its documented CLI. Include the without/with arrays, workload, machine, compiler, trial setup, limitations, and exact reproduction commands. Run the commands in the workspace before reporting any numbers; never invent, estimate, or copy placeholder measurements. If the tool or workload cannot be run, say so and omit `PerfReport` data. + - Link the `CompilerSutraPerf` name to the in-site docs and https://pypi.org/project/compilersutra-perf/ whenever measurements are reported. - Format reproduction commands for mobile and desktop: wrap commands line-by-line, avoid repeating long absolute paths, and use tabs or separate compact blocks for comparison variants. - **Compatibility**: table covering Clang, GCC, MSVC where relevant, language modes, and target restrictions. - **Usage example**: exact, copy-pasteable Clang command lines; include `-###` or IR inspection when useful. @@ -61,7 +61,7 @@ Requirements: 12. After writing, sanity-check: - the exact generated page path and slug; - responsive rendering of long commands and comparison blocks on mobile and desktop; - - performance claims link to the CompilerSutra Perf package when measurements are used; + - performance claims link to CompilerSutraPerf docs and PyPI when measurements are used; - `{/* HAND_AUTHORED */}` is still present; - all links and source paths are correct; - Mermaid/MDX syntax is valid; diff --git a/docs/articles/machineblockplacement-329-benchmark-prevalence-and-fix.md b/docs/articles/machineblockplacement-329-benchmark-prevalence-and-fix.md index 15887e46..593ab189 100644 --- a/docs/articles/machineblockplacement-329-benchmark-prevalence-and-fix.md +++ b/docs/articles/machineblockplacement-329-benchmark-prevalence-and-fix.md @@ -314,7 +314,7 @@ If **`-O2` is slower than `-O1`** on branchy text/parse code: 4. **Do not ship** with placement globally disabled — other loops benefit from fall-through layout. 5. **PGO** may supply missing `branch_weights` on char switches ([part 2](/docs/articles/machineblockplacement-wrong-bet-static-probabilities-pgo)). -Structured harness alternative: [CompilerSutra Perf](https://pypi.org/project/compilersutra-perf/) (`pip install compilersutra-perf`) with manifests like `configs/testcase_O{1,2,2np}.json` in the part 1 bundle. +Structured harness alternative: [CompilerSutraPerf](/docs/project/compilersutra-perf/) ([PyPI](https://pypi.org/project/compilersutra-perf/)) (`pip install compilersutra-perf`) with manifests like `configs/testcase_O{1,2,2np}.json` in the part 1 bundle. ## 9. What we do not claim @@ -337,7 +337,7 @@ Structured harness alternative: [CompilerSutra Perf](https://pypi.org/project/co - [Part 2: Static Probabilities, Fall-Through, and PGO](/docs/articles/machineblockplacement-wrong-bet-static-probabilities-pgo) - [LLVM PR #219126](https://github.com/llvm/llvm-project/pull/219126) · [Issue #218248](https://github.com/llvm/llvm-project/issues/218248) - [MachineBlockPlacement](https://llvm.org/doxygen/MachineBlockPlacement_8cpp.html) · [`opt-bisect-limit`](https://llvm.org/docs/CommandGuide/opt.html) -- [CompilerSutra Perf](https://pypi.org/project/compilersutra-perf/) +- [CompilerSutraPerf](/docs/project/compilersutra-perf/) · [PyPI](https://pypi.org/project/compilersutra-perf/) :::note Lab note Prevalence CSVs: research kit `results/testsuite_singlesource_mbp_sweep.csv`, `testsuite_multisource_mbp_sweep.csv`, `hpc_external_mbp_sweep.csv`, `text_scan_mbp_sweep.csv`, `parser_mbp_sweep.csv`, `sci_calc_mbp_sweep.csv`, `parallel_mbp_sweep.csv`, `oss_apps_mbp_sweep.csv` (Aug–Sep 2026 sweeps). Unfixed Clang: `/home/aitr/osc/llvm_project_without_change/llvm-project/build/bin/clang` (24.0.0git). Lab repro numbers: part 1/2 artifact bundles. CPU: AMD Ryzen 7 9700X. diff --git a/docs/articles/when-o2-layout-hurts-machineblockplacement.md b/docs/articles/when-o2-layout-hurts-machineblockplacement.md index 7063d32a..f394eed1 100644 --- a/docs/articles/when-o2-layout-hurts-machineblockplacement.md +++ b/docs/articles/when-o2-layout-hurts-machineblockplacement.md @@ -189,7 +189,7 @@ We run **three separate experiments** with distinct roles. Absolute counter valu | Role | Tool | Why it is here | |------|------|----------------| | **Primary evidence** | Linux `perf` | Direct `perf stat` on the binary; headline table for this article | -| **Independent benchmark harness** | [CompilerSutra Perf](https://pypi.org/project/compilersutra-perf/) (`csperf`) | Same builds, different runner — confirms the regression is not a `perf` artifact | +| **Independent benchmark harness** | [CompilerSutraPerf](/docs/project/compilersutra-perf/) ([PyPI](https://pypi.org/project/compilersutra-perf/)) (`csperf`) | Same builds, different runner — confirms the regression is not a `perf` artifact | | **Microarchitectural corroboration** | AMD uProf 5.2 | Branch-hotspot view on Zen 5; checks the story at frame level | ### Experiment A — Primary evidence: Linux `perf` @@ -206,7 +206,7 @@ Paired: O1 faster in **97/100** runs. Every O2 run has more branch misses than e `-O0` / `-O3` are not the investigation. For completeness: `-O3` ≈ `-O2` here; `-O0` is slower than `-O1` and is not needed to name the pass. -### Experiment B — Independent benchmark harness: CompilerSutra Perf (`csperf`) +### Experiment B — Independent benchmark harness: CompilerSutraPerf (`csperf`) Same source, same `N`, same stage1 Clang. Manifests live under `configs/csperf/testcase_*.json` (warmup 3, repeat 100). Install: `pip install compilersutra-perf`. @@ -540,7 +540,7 @@ Expected sum for `1048576` / seed `29`: `15723844160` (asserted in the full kit - LLVM [MachineBlockPlacement](https://llvm.org/doxygen/MachineBlockPlacement_8cpp.html) (“Branch Probability Basic Block Placement”) - Upstream report: [llvm/llvm-project#218248](https://github.com/llvm/llvm-project/issues/218248) - [`opt-bisect-limit`](https://llvm.org/docs/CommandGuide/opt.html) (same knob via `clang -mllvm`) -- [CompilerSutra Perf](https://pypi.org/project/compilersutra-perf/) (`pip install compilersutra-perf`) +- [CompilerSutraPerf](/docs/project/compilersutra-perf/) · [PyPI](https://pypi.org/project/compilersutra-perf/) (`pip install compilersutra-perf`) - Related: [Part 2 — static probabilities, PGO, and layout](/docs/articles/machineblockplacement-wrong-bet-static-probabilities-pgo), [Part 3 — 329-benchmark prevalence and LLVM fix](/docs/articles/machineblockplacement-329-benchmark-prevalence-and-fix), [GCC vs Clang real benchmarks](/docs/articles/gcc_vs_clang_real_benchmarks_2026_reporter), [stencil pass trace](/docs/articles/where_gcc_and_clang_diverge_stencil_pass_trace) :::note Lab note diff --git a/docs/project/compilersutra-perf/architecture.md b/docs/project/compilersutra-perf/architecture.md new file mode 100644 index 00000000..a6031d18 --- /dev/null +++ b/docs/project/compilersutra-perf/architecture.md @@ -0,0 +1,260 @@ +--- +title: "Architecture" +sidebar_label: "Architecture" +description: "Project architecture for compilersutra-perf — orchestration, adapters, measurement, and result artifacts." +displayed_sidebar: csperfSidebar +keywords: + - csperf architecture + - native runner + - pipeline phases +--- + +# Architecture + +**CompilerSutraPerf** (`csperf`, [PyPI: `compilersutra-perf`](https://pypi.org/project/compilersutra-perf/)) is a **host-side orchestration layer** around existing compilers and profilers. It does not replace Clang, GCC, HIP, OpenCL, Vulkan, or Linux `perf` — it sequences them and normalizes what they produce. + +Research framing: [Observatory](/docs/project/compilersutra-perf/observatory/). +Day-one usage: [Getting started](/docs/project/compilersutra-perf/getting-started/) · [Tutorial](/docs/project/compilersutra-perf/tutorial/). + +:::tip How to read this page +1. Skim the **project architecture** diagram (layers). +2. Follow the **pipeline phases** left → right (one `csperf run`). +3. Use the tables when you need field names or “what works today.” +::: + +## Project architecture + +Like IREE’s overview, think in **layers**: a stable core, pluggable adapters, optional measurement, and a common artifact. + +```mermaid +graph TB + accTitle: csperf project architecture + accDescr: Host Python core orchestrates toolchain adapters and native runners, optional profilers and energy plugins, then writes a common result artifact. + + subgraph Core["Core · Python"] + CLI["CLI / config
csperf run · doctor · diff"] + Detect["Source detect"] + Plan["Pipeline plan"] + Schema["Result schema
JSON / CSV / XLSX"] + Report["Reports
HTML / PDF / dashboard"] + end + + subgraph Adapters["Toolchain & backend adapters"] + CPU["CPU
clang / g++"] + HIP["HIP
hipcc / ROCm"] + OCL["OpenCL"] + VK["Vulkan
glslang / SPIR-V"] + end + + subgraph Native["Native execution · C++"] + NR["csperf_native_runner"] + OR["OpenCL runner"] + VR["Vulkan runner"] + end + + subgraph Measure["Measurement plugins"] + Perf["CPU profiler
perf / PAPI"] + Energy["Energy
RAPL / rocm-smi / …"] + end + + Out["Common result artifact
results/*.json"] + + CLI --> Detect --> Plan + Plan --> CPU & HIP & OCL & VK + CPU --> NR + HIP --> NR + OCL --> OR + VK --> VR + NR & OR & VR --> Schema + Perf -.-> Schema + Energy -.-> Schema + Schema --> Report + Schema --> Out + NR -.-> Perf + NR -.-> Energy +``` + +| Layer | Responsibility | +| --- | --- | +| **Core** | Flags, manifests, detection, planning, schema, compare, reports | +| **Adapters** | Map a file type to a vendor/open toolchain | +| **Native** | Low-overhead launch, timers, device interaction | +| **Measurement** | Optional counters and energy — never invent values | +| **Artifact** | One reusable JSON (plus CSV / reports) | + +**Design rule:** integrate existing tools; do not invent another general-purpose IR just to unify backends. Preserve native outputs; normalize only what must be compared. + +## Pipeline phases + +A single `csperf run` advances through discrete phases (same idea as IREE’s Input → … → VM overview): + +```mermaid +graph LR + accTitle: csperf run phases + accDescr: Input to Detect to Plan to Compile to Execute to Measure to Artifact + + A([Input]) --> B([Detect]) + B --> C([Plan]) + C --> D([Compile]) + D --> E([Execute]) + E --> F([Measure]) + F --> G([Artifact]) +``` + +| Phase | What happens | Stop early with | +| --- | --- | --- | +| **Input** | Read CLI / manifest / policy | — | +| **Detect** | Map extension → backend (`cpu`, `hip`, `opencl`, `vulkan`) | — | +| **Plan** | Build compile + run + profiler command list | `--plan-only` | +| **Compile** | Invoke toolchain (clang, hipcc, glslang, …) | — | +| **Execute** | Native runner: warmup + measured trials | `--execution-timeout` | +| **Measure** | Optional `perf` / PAPI / energy | `--no-perf`, `--no-energy` | +| **Artifact** | Write JSON (+ CSV; optional XLSX / HTML) | — | + +Concrete CPU example (perfwiki-style: command first, then what it means): + +```python +csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu \ + --warmup-runs 1 --repeat-runs 3 --output results/cpu.json +csperf profile results/cpu.json +``` + +That path is: **Input → Detect(cpu) → Plan → Compile → Execute → Measure → Artifact**. + +## Workflow overview + +Using `csperf` in practice: + +1. **Prepare a workload** — C/C++, HIP, OpenCL `.cl`, or Vulkan shader +2. **Pick a backend** — explicit `--backend`, or `gpu` to auto-route by extension +3. **Run** — `csperf run` (or `quickstart` for the smoke path) +4. **Inspect** — `profile`, `diff`, `visualize` + +```mermaid +graph LR + accTitle: csperf user workflow + accDescr: Workload to backend to run to inspect + + W([Workload]) --> B([Backend]) + B --> R([csperf run]) + R --> I([Inspect]) +``` + +| Step | Typical command | +| --- | --- | +| Smoke test | `csperf quickstart --output-dir results/quickstart` | +| Time only | `csperf run … --no-perf --no-energy` | +| Opt sweep | `csperf run … --diff-optimize --report-format both` | +| Compare two JSON files | `csperf diff a.json b.json --csv out.csv` | + +## CPU path (stable on Linux) + +```mermaid +graph LR + accTitle: CPU backend path + accDescr: Source to compiler to binary to native runner to optional perf to JSON + + S([Source .c/.cpp]) --> C([Compiler]) + C --> Bin([Binary]) + C -.-> IR(["IR *.ll · best-effort"]) + Bin --> N([Native runner]) + N --> J([JSON]) + N -.-> P([perf / PAPI]) + P --> J +``` + +- Compiler resolution: `CXX_COMPILER` → `CXX` → default `clang++` +- `CXXFLAGS` is **not** read — pass `--compiler-flag` +- Prefer same-execution `perf` for branch/cache studies ([Methodology](/docs/project/compilersutra-perf/methodology/)) + +## GPU paths (experimental) + +```mermaid +graph TB + accTitle: GPU backend routing + accDescr: gpu alias routes by extension to HIP OpenCL or Vulkan + + IN([Input]) --> EXT{Extension} + EXT -->|.hip| HIP([HIP · hipcc + ROCm]) + EXT -->|.cl| OCL([OpenCL · build + launch]) + EXT -->|.comp / .glsl| VK([Vulkan · validate only]) + HIP & OCL & VK --> OUT([JSON artifact]) +``` + +| Backend | Execute today? | Notes | +| --- | --- | --- | +| HIP | Yes | `kernel_time_ms` (events) ≠ `hip_profiler_*` (rocprof) | +| OpenCL | Yes | State `--opencl-measure` when publishing times | +| Vulkan | Partial | Shader-module validation; **no** full dispatch yet | + +## Measurement model + +Counters and energy are **plugins beside** the timed trials — not the timer itself. + +```mermaid +graph TB + accTitle: Measurement plugins + accDescr: Timed trials always write JSON; optional profiler and energy feed the same artifact + + T([Timed trials]) --> J([JSON artifact]) + T -.-> Perf([CPU profiler]) + T -.-> E([Energy backend]) + Perf --> J + E --> J +``` + +| Flag | Effect | +| --- | --- | +| `--no-perf` | Skip hardware counters | +| `--no-energy` | Skip `power` block | +| `--cpu-profiler auto\|perf\|papi\|papi-native` | Choose counter path | + +On Linux, the CPU profiler path is built on the same ideas as the [perfwiki tutorial](https://perfwiki.github.io/main/tutorial/): **events** (software vs PMU), **`perf stat`-style counting**, and **multiplexing/scaling** when you ask for more events than hardware counters. `csperf` records whether counts came from the **same execution** as wall time — that provenance is what makes a claim honest. + +Before publishing numbers, read [Methodology](/docs/project/compilersutra-perf/methodology/) (events, same-execution, scaling). + +## Result artifact + +Every successful or partial run aims to be **self-describing** (commands, host, tool version, provenance). + +| If you want… | Look at | +| --- | --- | +| Runtime | `metrics.execution_time_ms` | +| Per-trial times | `execution.trial_results` | +| Exact commands | `commands` | +| Same-exec counters? | `execution.measurement.same_execution` | +| Why counters missing | `execution.perf_error` / `metrics_availability` | + +Top-level sections: `pipeline`, `commands`, `hardware`, `metrics`, `execution`, `artifacts`, optional `power`. + +## Module map (0.2.0) + +| Tree area | Role | +| --- | --- | +| `cli` | Unified entry (`run`, `doctor`, `diff`, …) | +| `execution/` | Compile/run adapters | +| `profiler/` | `perf`, PAPI, macOS paths | +| `energy/` | RAPL / AMD GPU / macOS backends | +| `report/` | HTML/PDF exporters | +| `experiments/` | row / column / tiled macros | +| `native/runtime/` | C++ runners | +| `examples/` | Seed workloads | + +## Status matrix + +| Path | Ready? | +| --- | --- | +| Linux CPU | Stable | +| OpenCL / HIP | Experimental | +| Vulkan | Partial (validate) | +| CUDA / Metal | Not implemented | + +Roadmap milestones: [Observatory](/docs/project/compilersutra-perf/observatory/). + +## See also + +- [Observatory](/docs/project/compilersutra-perf/observatory/) — research goal and evidence labels +- [Usage](/docs/project/compilersutra-perf/usage/) — flags and recipes +- [Methodology](/docs/project/compilersutra-perf/methodology/) — what you can trust +- [Energy & reports](/docs/project/compilersutra-perf/energy-and-reports/) — RAPL and HTML/PDF +- [Troubleshooting](/docs/project/compilersutra-perf/troubleshooting/) — when a phase fails diff --git a/docs/project/compilersutra-perf/energy-and-reports.md b/docs/project/compilersutra-perf/energy-and-reports.md new file mode 100644 index 00000000..a484f312 --- /dev/null +++ b/docs/project/compilersutra-perf/energy-and-reports.md @@ -0,0 +1,116 @@ +--- +title: "Energy & reports" +description: "Energy backends and HTML/PDF reports for compilersutra-perf 0.2.0." +displayed_sidebar: csperfSidebar +keywords: + - csperf energy + - RAPL + - HTML report +--- + +# Energy & reports + +Energy backends and HTML/PDF reports for **CompilerSutraPerf** 0.2.0 ([PyPI: `compilersutra-perf`](https://pypi.org/project/compilersutra-perf/)). + +## Energy + +Results land in the top-level `power` block. Energy is **on by default** when a backend is available. + +```python +csperf list-energy-backends + +csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu --no-perf +csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu --no-energy +csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu \ + --energy-backend amd-cpu --output results/cpu-energy.json +``` + +| Flag | Meaning | +| --- | --- | +| `--energy` | Force enable (default when available) | +| `--no-energy` | Skip all energy backends | +| `--energy-backend` | `auto`, `amd-cpu`, `amd-gpu`, … | + +Manifest: `"use_energy": true`, `"energy_backends": "auto"`. + +### Backends (0.2.0) + +| Backend | Domain | Platform | Mechanism | +| --- | --- | --- | --- | +| `amd-cpu` | CPU package | Linux | RAPL via `perf` `power/energy-pkg/` | +| `amd-gpu` | GPU | Linux HIP | `rocm-smi --showpower` | +| `macos-cpu` | CPU package | macOS | powermetrics | + +Stubs (`intel-cpu`, `nvidia-gpu`, `external-meter`) show as unavailable until implemented. + +### Linux RAPL + +```python +sudo sysctl kernel.perf_event_paranoid=-1 +``` + +If `perf` hits LLVM symbol errors: + +```python +export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu +``` + +### How to read `power.domains[]` + +- `energy_j`, `avg_power_w` +- `domain`, `backend`, `same_execution` + +Package RAPL is **not** per-process energy. + +Example: + +```text +power: collected + cpu_package: energy_j=0.842, avg_power_w=12.8 W, source=perf, same_execution=true +``` + +With opt sweeps: + +```python +csperf run --input examples/cpp/matrix_traversal.cpp --diff-optimize --no-perf +``` + +Compare runtime (and energy when collected) across the generated JSON/CSV files. + +## Reports + +```mermaid +flowchart LR + A[JSON artifacts] --> B[ReportDocument] --> C[HTML] + B --> D[PDF] +``` + +```python +csperf visualize results/cpu.json --output results/report.html +pip install 'compilersutra-perf[pdf]' +csperf visualize results/cpu.json --format pdf --output results/report.pdf +csperf visualize results/cpu.json --format both --output results/report +csperf list-report-formats +``` + +| Section | Content | +| --- | --- | +| `core` | timing, metrics, trials (single + comparison) | +| `energy` | `power` domains when present | + +| Extra | Enables | +| --- | --- | +| `[visualize]` | HTML + Streamlit `dashboard` | +| `[pdf]` | WeasyPrint | +| `[excel]` | XLSX | + +```python +csperf visualize results/cpu.json results/hip.json --output results/cmp.html +csperf dashboard results/cpu.json results/hip.json +``` + +## See also + +- [Methodology](/docs/project/compilersutra-perf/methodology/) +- [Troubleshooting](/docs/project/compilersutra-perf/troubleshooting/) +- [Usage](/docs/project/compilersutra-perf/usage/) diff --git a/docs/project/compilersutra-perf/getting-started.md b/docs/project/compilersutra-perf/getting-started.md new file mode 100644 index 00000000..496982a8 --- /dev/null +++ b/docs/project/compilersutra-perf/getting-started.md @@ -0,0 +1,219 @@ +--- +title: "Getting started" +description: "Install compilersutra-perf and run your first CPU or GPU workload with csperf." +displayed_sidebar: csperfSidebar +keywords: + - csperf tutorial + - compilersutra-perf install +--- + +# Getting started + +Short path to a first JSON with **CompilerSutraPerf**. Full walkthrough: [Tutorial](/docs/project/compilersutra-perf/tutorial/). Aligned with **[0.2.0](/docs/project/compilersutra-perf/releases/0.2.0/)**. Package: [PyPI: `compilersutra-perf`](https://pypi.org/project/compilersutra-perf/). + +## 1. Install + +```python +pip install compilersutra-perf +pip install 'compilersutra-perf[excel,pdf,visualize]' +csperf doctor +csperf quickstart --output-dir results/quickstart +``` + +`quickstart` = doctor + bundled CPU example + profile summary + HTML report (when available). + +```python +csperf doctor --install +csperf doctor --install --all +``` + +Need **Python 3.11+**. Prefer a venv. YAML policy files need `pip install pyyaml`. + +### Editable install (if you have a local tree) + +```python +# local checkout of the package sources +cd /path/to/compilersutra-perf +python3 -m venv .venv +source .venv/bin/activate +pip install -e '.[dev,pdf,visualize]' +csperf doctor +csperf quickstart +``` + +Prefer the published package when possible: [PyPI: `compilersutra-perf`](https://pypi.org/project/compilersutra-perf/). + +Native runners build on first use, or: + +```python +cmake -S native/runtime -B build/native && cmake --build build/native +``` + +## 2. Energy and reports (optional) + +```python +csperf list-energy-backends +csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu \ + --energy-backend amd-cpu --output results/cpu-energy.json +csperf list-report-formats +csperf visualize results/cpu-energy.json --format both --output results/report +``` + +Opt-level sweep with comparison report: + +```python +csperf run --input examples/cpp/matrix_traversal.cpp \ + --diff-optimize --no-perf --report-format both +# → results/optimize/*.json + optimize_report.html +``` + +Details: [Energy & reports](/docs/project/compilersutra-perf/energy-and-reports/). + +## 3. List devices + +```python +csperf list-devices --backend gpu +csperf gpuinfo +csperf deviceinfo + +csperf list-devices --backend cpu +csperf list-devices --backend hip +csperf list-devices --backend opencl +csperf list-devices --backend vulkan +csperf list-devices --backend opencl --vendor amd +``` + +Use the reported `device_index` for GPU runs. +Note: `csperf list-devices --backend` **without a value** is invalid. + +### Policy config (optional) + +```json +{ + "vendor": "amd", + "opencl": { "device_policy": "vendor" }, + "vulkan": { "queue_policy": "compute", "library_policy": "default" } +} +``` + +```python +csperf run --input path/to/kernel.cl --backend opencl \ + --policy-config configs/policy_config.sample.json \ + --device-index 0 --output results/my-opencl.json +``` + +CLI flags override the policy file. + +## 4. Run your C / C++ file + +```python +csperf run --input path/to/my_benchmark.cpp --backend cpu \ + --warmup-runs 1 --repeat-runs 3 --output results/my-cpp.json + +csperf profile results/my-cpp.json +``` + +Pin a core (Linux): + +```python +csperf run --input path/to/my_benchmark.cpp --backend cpu \ + --cpu-affinity 0 --warmup-runs 1 --repeat-runs 5 \ + --output results/my-cpp.json +``` + +JSON records `execution.host_platform` (`linux` / `macos`) and `execution.metrics_availability`. + +Defaults without overrides: `--warmup-runs 3`, `--repeat-runs 15`. + +## 5. macOS CPU (experimental) + +```python +csperf cpuinfo +csperf run --input path/to/program.cpp --backend cpu --no-perf \ + --output results/macos-cpu.json +``` + +`powermetrics` needs passwordless `sudo` and is **not** the same process as timed trials. `--cpu-affinity` is rejected on macOS. Metal is not implemented. See [Methodology](/docs/project/compilersutra-perf/methodology/). + +## 6. GPU workloads + +**HIP** + +```python +csperf run --input path/to/my_kernel.hip --backend hip \ + --device-index 0 --warmup-runs 1 --repeat-runs 3 \ + --output results/my-hip.json +``` + +**OpenCL** (simple) + +```python +csperf run --input path/to/my_kernel.cl --backend opencl \ + --device-index 0 --warmup-runs 1 --repeat-runs 3 \ + --output results/my-opencl.json +``` + +**OpenCL** (explicit args) + +```python +csperf run \ + --input path/to/my_kernel.cl \ + --backend opencl \ + --device-index 0 \ + --kernel-name my_kernel \ + --global-size 4096 \ + --local-size 64 \ + --kernel-arg buffer:float:read:4096:1.0 \ + --kernel-arg buffer:float:write:4096:0.0 \ + --readback-arg 1 \ + --output results/my-opencl.json +``` + +**Vulkan** — compile + validate only (`status: partial`), not full dispatch: + +```python +csperf run --input path/to/my_shader.comp --backend vulkan \ + --device-index 0 --output results/my-vulkan.json +``` + +## 7. Compare CPU vs GPU results + +```python +csperf visualize results/my-cpu.json results/my-hip.json \ + --output results/my-cpu-vs-gpu.html +csperf dashboard results/my-cpu.json results/my-hip.json +``` + +Only compare matched workloads. + +## Recommended workflow + +1. Small input that compiles quickly +2. One backend first → save JSON +3. Change compiler, flags, or device +4. Compare with `profile` / `diff` / `visualize` +5. Read [Methodology](/docs/project/compilersutra-perf/methodology/) before publishing + +## Common mistakes + +- `list-devices --backend` with no value +- Inactive venv on a source checkout +- Treating unmatched CPU/GPU runs as equivalent +- Expecting Vulkan validation = full kernel benchmark +- Assuming `device-index 0` is the same GPU on every backend +- Treating package RAPL joules as per-process energy + +## Success checklist + +- [ ] `doctor` / `quickstart` OK +- [ ] Your own `results/*.json` exists +- [ ] `csperf profile` shows timing + +## Next + +| Goal | Page | +| --- | --- | +| Flags, manifests, profilers | [Usage](/docs/project/compilersutra-perf/usage/) | +| Trust numbers | [Methodology](/docs/project/compilersutra-perf/methodology/) | +| Research vision | [Observatory](/docs/project/compilersutra-perf/observatory/) | +| Failures | [Troubleshooting](/docs/project/compilersutra-perf/troubleshooting/) | diff --git a/docs/project/compilersutra-perf/index.md b/docs/project/compilersutra-perf/index.md new file mode 100644 index 00000000..ad9bbf09 --- /dev/null +++ b/docs/project/compilersutra-perf/index.md @@ -0,0 +1,49 @@ +--- +title: "CompilerSutraPerf" +sidebar_label: "Overview" +description: "Release catalogue for CompilerSutraPerf (csperf / compilersutra-perf) — pick a version, then install and read the matching docs." +displayed_sidebar: csperfSidebar +keywords: + - compilersutra-perf + - csperf releases + - changelog +--- + +# CompilerSutraPerf + +**CompilerSutraPerf** (`csperf`) — compile, run, profile, and compare C/C++ and GPU workloads. Install from [PyPI: `compilersutra-perf`](https://pypi.org/project/compilersutra-perf/). + +This hub is **release-first**. Open a version page for what shipped, how to install that line, and which docs apply. + +## Releases + +| Release | Date | Status | Page | +| --- | --- | --- | --- | +| **0.2.0** | 2026-09-02 | Current · Alpha | [0.2.0 release notes](/docs/project/compilersutra-perf/releases/0.2.0/) | +| 0.1.2 | — | Maintenance | [0.1.2 release notes](/docs/project/compilersutra-perf/releases/0.1.2/) | +| 0.1.1 | — | Infrastructure | [0.1.1 release notes](/docs/project/compilersutra-perf/releases/0.1.1/) | +| 0.1.0 | — | First MVP | [0.1.0 release notes](/docs/project/compilersutra-perf/releases/0.1.0/) | + +**Start here for the current line:** [Release 0.2.0](/docs/project/compilersutra-perf/releases/0.2.0/) + +```python +pip install 'compilersutra-perf==0.2.0' +csperf quickstart --output-dir results/quickstart +``` + +## Docs (current line · 0.2.x) + +| Page | Role | +| --- | --- | +| [Tutorial](/docs/project/compilersutra-perf/tutorial/) | Hands-on path (perfwiki-style) | +| [Getting started](/docs/project/compilersutra-perf/getting-started/) | Short install → first JSON | +| [Usage](/docs/project/compilersutra-perf/usage/) | How-tos + CLI | +| [Methodology](/docs/project/compilersutra-perf/methodology/) | How to read counters | +| [Observatory](/docs/project/compilersutra-perf/observatory/) | Research goal + roadmap | +| [Architecture](/docs/project/compilersutra-perf/architecture/) | Layers and pipeline phases | +| [Energy & reports](/docs/project/compilersutra-perf/energy-and-reports/) | RAPL + HTML/PDF | +| [Troubleshooting](/docs/project/compilersutra-perf/troubleshooting/) | Common failures | + +Package: [PyPI — CompilerSutraPerf (`compilersutra-perf`)](https://pypi.org/project/compilersutra-perf/). + +Part of the [Projects catalogue](/docs/project/). diff --git a/docs/project/compilersutra-perf/methodology.md b/docs/project/compilersutra-perf/methodology.md new file mode 100644 index 00000000..f7b9a1c5 --- /dev/null +++ b/docs/project/compilersutra-perf/methodology.md @@ -0,0 +1,175 @@ +--- +title: "Methodology" +sidebar_label: "Methodology" +description: "How to read csperf measurements — events, same-execution, multiplexing, and what not to publish. Styled after the Linux perf tutorial." +displayed_sidebar: csperfSidebar +keywords: + - perf methodology + - same execution + - multiplexing + - branch-misses +--- + +# Methodology + +This page teaches how to **read** numbers from **CompilerSutraPerf** (`csperf`), the same way the [Linux perf tutorial](https://perfwiki.github.io/main/tutorial/) teaches `perf stat`: what an event is, what a count means, and when the number is only an estimate. + +CompilerSutraPerf ([PyPI: `compilersutra-perf`](https://pypi.org/project/compilersutra-perf/)) is **Alpha**. Prefer Linux CPU + `perf` for serious claims. + +## What CompilerSutraPerf is doing + +On the CPU path, CompilerSutraPerf is not a new profiler. It: + +1. Compiles and runs your program through a native runner +2. Optionally wraps trials with Linux **`perf stat`** (or falls back to PAPI) +3. Writes wall time + counters into a JSON artifact with **provenance** + +If you already know `perf`, think of `csperf run` as: *compile → timed trials → structured `perf stat` → JSON*. + +## Events (the important vocabulary) + +`perf` (and therefore `csperf`) can count events from different sources. Mixing them without labels is how bad papers happen. + +| Kind | What it is | Examples | +| --- | --- | --- | +| **Software events** | Kernel counters | context-switches, page-faults | +| **Hardware / PMU events** | CPU Performance Monitoring Unit | cycles, instructions, branch-misses, cache-misses | +| **Generic hardware names** | Portable aliases mapped to a real PMU event *if the CPU has one* | `cycles`, `instructions`, `branches` | +| **Tracepoints** | Kernel `ftrace` points | `sched:sched_switch`, … | + +On each CPU, a generic name is mapped to a real event when possible; otherwise it cannot be used. Vendor PMU tables differ (Intel vs AMD vs Arm). `csperf` probes what works on the host and records availability in the artifact. + +Common hardware names you will see in `csperf` / `perf` output: + +| Event | Meaning (tutorial sense) | +| --- | --- | +| `cycles` / `cpu-cycles` | Total cycles (beware frequency scaling) | +| `ref-cycles` | Cycles not affected by frequency scaling | +| `instructions` | Retired instructions (can be noisy) | +| `branches` / `branch-instructions` | Retired branches | +| `branch-misses` | Mispredicted branches | +| `cache-references` / `cache-misses` | Usually LLC-oriented; pair them for a miss rate | + +Derived metrics (after the `#` in `perf stat`, or in `csperf` summaries) include things like **IPC** (instructions / cycles) and **branch-miss rate**. + +## Counting: wall time vs counters + +### Wall time (always the primary timer) + +`metrics.execution_time_ms` and `execution.trial_results` come from the **native runner** for the measured trials. That is the stopwatch. + +### Counters (`perf` / PAPI) + +When profiling is enabled, `csperf` tries to collect counters on the **same execution** as those trials (`execution.measurement.same_execution: true`). That is the trustworthy path for “did `-O2` change branch behavior?” + +If same-execution collection fails, timing can still succeed. Counters may be omitted or collected on a **separate** run (`separate_execution`). Do not treat that as the same evidence. + +| Field | Same process as wall time? | +| --- | --- | +| `execution_time_ms` | Yes (timed trials) | +| Linux `perf` (successful wrap) | Yes when `same_execution` | +| PAPI grouped fallback | Usually **no** | +| macOS `powermetrics` | **No** for primary trials | +| HIP `kernel_time_ms` | Kernel only | +| HIP `hip_profiler_*` | rocprof trace — not the same as HIP events | + +Never mix `perf_source: perf` with `papi` / powermetrics as one counter set. `csperf diff` warns when sources differ. + +## Multiplexing and scaling (read this twice) + +Hardware PMUs have a **small** number of counters. If you ask for more events than counters, the kernel **time-multiplexes** them and later **scales** the counts: + +```text +final_count ≈ raw_count × (time_enabled / time_running) +``` + +That is an **estimate**, not a continuous count. Blind spots can introduce error. The perfwiki tutorial stresses this; `csperf` inherits the same kernel behavior when it asks `perf` for many events. + +Practical rules: + +1. Prefer fewer, important events for a claim (for example branches + branch-misses). +2. Prefer `same_execution: true` in the JSON. +3. Use enough `--repeat-runs` to see variance (like `perf stat -r N`). +4. If scaling percentages look ugly in raw `perf` output, shrink the event set. + +`csperf` groups `perf stat` invocations partly to reduce `` / multiplexing pain — still treat crowded event sets carefully. + +## A concrete mental loop (perfwiki style) + +**Command:** + +```python +csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu \ + --cpu-affinity 0 --warmup-runs 1 --repeat-runs 5 \ + --output results/cpu.json +csperf profile results/cpu.json +``` + +**What to check in the JSON (in order):** + +1. Did it run? → `status`, `execution.trial_results` +2. How long? → `metrics.execution_time_ms` +3. Were counters on the same run? → `execution.measurement.same_execution` +4. What was counted? → counter fields / `perf_source` +5. Can I reproduce? → `commands`, compiler flags, `tool_version`, `hardware` + +That is the same discipline as reading a `perf stat` block: elapsed time first, then event counts, then derived ratios, then caveats. + +## Environment matters + +Like `perf`: + +| Mode | `csperf` analogue | +| --- | --- | +| Pin to one CPU | `--cpu-affinity 0` (Linux only) | +| Repeat for variance | `--repeat-runs N` | +| Skip counters | `--no-perf` | +| User-only style focus | Prefer same binary, quiet machine, pinned core | + +macOS: no Linux `perf`; use `--no-perf`. Optional `powermetrics` is a **different** measurement class — see platform matrix below. + +## Power / energy + +Package energy (RAPL via `perf` `power/energy-pkg/`, or `rocm-smi`, or macOS powermetrics) is **not** “this process used exactly X joules.” + +| Domain | Typical source | Caveat | +| --- | --- | --- | +| `cpu_package` | RAPL | Package-level estimate | +| `gpuN` | `rocm-smi` | Instantaneous / average watts | + +Details: [Energy & reports](/docs/project/compilersutra-perf/energy-and-reports/). + +## Platform matrix (0.2.0) + +| Feature | Linux | macOS | +| --- | --- | --- | +| Compile + run CPU | Supported | Experimental | +| `--cpu-affinity` | Supported | Rejected | +| Hardware counters | `perf` / PAPI | powermetrics (passwordless `sudo`) | +| HIP / OpenCL / Vulkan | As in [Usage](/docs/project/compilersutra-perf/usage/) | Toolchains often absent | + +## Evidence labels ([Observatory](/docs/project/compilersutra-perf/observatory/)) + +| Label | Examples | +| --- | --- | +| Hardware measurement | Wall time, `perf` / PAPI, rocprof, RAPL | +| Model-based estimation | `llvm-mca` (roadmap) | +| Static analysis only | Asm / instruction count suites (roadmap) | + +Never present theoretical analysis as measured hardware performance. + +## What not to publish yet + +- Cross-backend “speedup” without matched workload, size, and measure mode +- OpenCL times without `--opencl-measure` +- macOS cycles as Linux `perf` `cpu-cycles` +- Scaled / separate-execution counters as if they were continuous same-exec counts +- CUDA / Metal (not implemented) +- Vulkan validation as a kernel benchmark + +## See also + +- [Usage](/docs/project/compilersutra-perf/usage/) — flags +- [Architecture](/docs/project/compilersutra-perf/architecture/) — where measurement sits in the pipeline +- [Troubleshooting](/docs/project/compilersutra-perf/troubleshooting/) — `perf` access and symbol errors +- External: [perfwiki tutorial](https://perfwiki.github.io/main/tutorial/) — `perf stat`, events, multiplexing diff --git a/docs/project/compilersutra-perf/observatory.md b/docs/project/compilersutra-perf/observatory.md new file mode 100644 index 00000000..f728da71 --- /dev/null +++ b/docs/project/compilersutra-perf/observatory.md @@ -0,0 +1,117 @@ +--- +title: "Observatory" +sidebar_label: "Observatory" +description: "CPU/GPU hardware and compiler observatory vision for compilersutra-perf — problem, approach, evidence labels, and roadmap." +displayed_sidebar: csperfSidebar +keywords: + - compiler observatory + - csperf research + - hardware measurement +--- + +# Observatory + +Research framing for **CompilerSutraPerf** (`csperf`, [PyPI: `compilersutra-perf`](https://pypi.org/project/compilersutra-perf/)). This is the north star. Alpha **0.2.0** already ships the Linux CPU path; GPU and static-analysis pieces grow toward this vision. + +## Problem + +Hardware ecosystems are fragmented. Each CPU/GPU stack brings its own compiler, IR, profiler, SDK, and scripts. A simple loop — compile → inspect → benchmark → compare — becomes a different workflow per platform, hard to reproduce, and easy to mislabel (static estimates presented as measured performance). + +## Approach + +**Do not reinvent the compiler ecosystem. Integrate and orchestrate it.** + +```mermaid +graph TB + accTitle: Observatory orchestration layer + accDescr: csperf common interface sits above CPU GPU and accelerator toolchains + + CS([csperf · common interface]) + CS --> CPU([CPU tools]) + CS --> GPU([GPU tools]) + CS --> ACC([Accelerators · roadmap]) + CPU --> V([Vendor / open toolchains]) + GPU --> V + ACC --> V +``` + +`csperf` provides: unified CLI, experiment config, result collection, a common schema, `diff` / `--diff-optimize`, and reports. Vendor tools still do the architecture-specific work (Clang/GCC, HIP/ROCm, OpenCL, Vulkan, `perf`, …). + +## Evidence labels + +Different numbers are different kinds of evidence. Keep them labeled: + +| Label | Examples | +| --- | --- | +| **Hardware measurement** | Wall time, `perf` / PAPI, rocprof, RAPL | +| **Model-based estimation** | `llvm-mca`, cost models (roadmap) | +| **Static analysis only** | IR/asm size, instruction count (roadmap) | + +Never present theoretical analysis as measured hardware performance. Never claim GPU speed without running on that GPU. Details: [Methodology](/docs/project/compilersutra-perf/methodology/). + +## What you can run today (0.2.0) + +```python +pip install compilersutra-perf +csperf doctor +csperf quickstart --output-dir results/quickstart + +csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu \ + --warmup-runs 1 --repeat-runs 3 --output results/cpu.json + +csperf profile results/cpu.json +csperf diff results/run-a.json results/run-b.json --csv results/compare.csv + +csperf run --input examples/cpp/matrix_traversal.cpp \ + --diff-optimize --no-perf --report-format both +``` + +| Concern | Status now | +| --- | --- | +| Common CLI | Shipped | +| CPU compile / run / profile | Solid on Linux | +| HIP / OpenCL / Vulkan | Experimental (Vulkan = validate) | +| Result schema + diff + HTML/PDF | Shipped | +| Energy backends | Partial (RAPL / AMD GPU / macOS) | +| LLVM discovery, asm / llvm-mca suites | Planned | +| Android / embedded telemetry | Planned | + +## Target workflow + +```mermaid +graph LR + accTitle: Target experimentation pipeline + accDescr: Source to Compile to Binary to Benchmark and Profile to Compare to Report + + S([Source]) --> C([Compile]) + C --> B([Binary / IR]) + B --> Ben([Benchmark]) + B --> Pro([Profile]) + Ben --> Cmp([Compare]) + Pro --> Cmp + Cmp --> R([Report]) +``` + +IR emission (clang) is best-effort today. Full assembly extraction, static instruction metrics, and `llc`-driven AMDGPU/RISC-V suites are roadmap work. + +## Roadmap (milestones) + +| Milestone | Theme | +| --- | --- | +| **0.2.0** (shipped) | run / profile / diff / report, energy, quickstart, honesty docs | +| **0.3.0** | Plugins, schema, registries | +| **0.4.0** | Observatory MVP: LLVM discovery, IR/asm JSON, first static experiments | +| **0.5.0** | Asm analyzer, llvm-mca, research Markdown reports | +| **0.6.0** | Regression taxonomy, A-vs-B research diffs, vendor reports | +| **0.7.0** | Android / embedded field telemetry | + +## Vision (one line) + +> A unified experimentation and analysis layer across heterogeneous CPU, GPU, and accelerator toolchains — integrating existing tools rather than replacing them. + +## See also + +- [Getting started](/docs/project/compilersutra-perf/getting-started/) +- [Architecture](/docs/project/compilersutra-perf/architecture/) — how the current MVP is built +- [Methodology](/docs/project/compilersutra-perf/methodology/) — measurement honesty +- [Usage](/docs/project/compilersutra-perf/usage/) — CLI surface today diff --git a/docs/project/compilersutra-perf/releases/0.1.0.md b/docs/project/compilersutra-perf/releases/0.1.0.md new file mode 100644 index 00000000..793398be --- /dev/null +++ b/docs/project/compilersutra-perf/releases/0.1.0.md @@ -0,0 +1,44 @@ +--- +title: "0.1.0" +sidebar_label: "0.1.0" +description: "compilersutra-perf 0.1.0 — first public MVP snapshot." +displayed_sidebar: csperfSidebar +keywords: + - compilersutra-perf 0.1.0 + - MVP +--- + +# Release 0.1.0 + +**CompilerSutraPerf** 0.1.0 · **Status:** First public MVP (superseded by [0.2.0](/docs/project/compilersutra-perf/releases/0.2.0/)) + +Package: [PyPI: `compilersutra-perf`](https://pypi.org/project/compilersutra-perf/). + +## Install this release + +```python +pip install 'compilersutra-perf==0.1.0' +``` + +## What shipped + +- CLI: `run`, `profile`, `visualize`, `dashboard`, `list-backends`, `list-experiments`, `list-devices` +- CPU C/C++ compile-and-run, LLVM IR emission, Linux `perf` counters +- Warmup / repeat runs, CPU affinity, native C++ runner +- OpenCL execution + argument binding +- Vulkan shader compile + validation +- HIP compile-and-run + rocprof-style profiling +- GPU device discovery; JSON/CSV; HTML + Streamlit dashboard + +## Known gaps (at 0.1.0) + +- No workload matching / baseline regression store +- No Vulkan compute dispatch +- No CUDA / Metal + +## Next / previous + +- Next: [0.1.1](/docs/project/compilersutra-perf/releases/0.1.1/) +- [All releases](/docs/project/compilersutra-perf/releases/) +- Hub: [CompilerSutraPerf](/docs/project/compilersutra-perf/) +- Current: [0.2.0](/docs/project/compilersutra-perf/releases/0.2.0/) diff --git a/docs/project/compilersutra-perf/releases/0.1.1.md b/docs/project/compilersutra-perf/releases/0.1.1.md new file mode 100644 index 00000000..065b88a6 --- /dev/null +++ b/docs/project/compilersutra-perf/releases/0.1.1.md @@ -0,0 +1,36 @@ +--- +title: "0.1.1" +sidebar_label: "0.1.1" +description: "compilersutra-perf 0.1.1 — portable CPU execution and macOS smoke improvements." +displayed_sidebar: csperfSidebar +keywords: + - compilersutra-perf 0.1.1 +--- + +# Release 0.1.1 + +**CompilerSutraPerf** 0.1.1 · **Status:** Infrastructure (superseded by [0.2.0](/docs/project/compilersutra-perf/releases/0.2.0/)) + +Package: [PyPI: `compilersutra-perf`](https://pypi.org/project/compilersutra-perf/). + +## Install this release + +```python +pip install 'compilersutra-perf==0.1.1' +``` + +## Highlights + +- Experimental macOS CPU docs and CI coverage +- Native CPU runner more portable (Linux-only affinity isolated) +- Optional OpenCL discovery when dependency missing +- macOS-incompatible Vulkan `libdl` linkage removed +- Improved macOS `powermetrics` diagnostics +- macOS CPU smoke in CI + +## Next / previous + +- Next: [0.1.2](/docs/project/compilersutra-perf/releases/0.1.2/) +- Previous: [0.1.0](/docs/project/compilersutra-perf/releases/0.1.0/) +- [All releases](/docs/project/compilersutra-perf/releases/) +- Hub: [CompilerSutraPerf](/docs/project/compilersutra-perf/) diff --git a/docs/project/compilersutra-perf/releases/0.1.2.md b/docs/project/compilersutra-perf/releases/0.1.2.md new file mode 100644 index 00000000..2e1aec34 --- /dev/null +++ b/docs/project/compilersutra-perf/releases/0.1.2.md @@ -0,0 +1,35 @@ +--- +title: "0.1.2" +sidebar_label: "0.1.2" +description: "compilersutra-perf 0.1.2 — maintenance release (attribution update)." +displayed_sidebar: csperfSidebar +keywords: + - compilersutra-perf 0.1.2 +--- + +# Release 0.1.2 + +**CompilerSutraPerf** 0.1.2 · **Status:** Maintenance (superseded by [0.2.0](/docs/project/compilersutra-perf/releases/0.2.0/)) + +Package: [PyPI: `compilersutra-perf`](https://pypi.org/project/compilersutra-perf/). + +## Install this release + +```python +pip install 'compilersutra-perf==0.1.2' +``` + +Prefer **0.2.0** for new work unless you must reproduce an older experiment. + +## Changelog + +### Changed + +- Author / maintainer attribution updated to Priya Pandey and CompilerSutra (`osc@compilersutra.com`). + +## Next / previous + +- Next: [0.2.0](/docs/project/compilersutra-perf/releases/0.2.0/) +- Previous: [0.1.1](/docs/project/compilersutra-perf/releases/0.1.1/) +- [All releases](/docs/project/compilersutra-perf/releases/) +- Hub: [CompilerSutraPerf](/docs/project/compilersutra-perf/) diff --git a/docs/project/compilersutra-perf/releases/0.2.0.md b/docs/project/compilersutra-perf/releases/0.2.0.md new file mode 100644 index 00000000..6a7c6cfb --- /dev/null +++ b/docs/project/compilersutra-perf/releases/0.2.0.md @@ -0,0 +1,101 @@ +--- +title: "0.2.0" +sidebar_label: "0.2.0 (current)" +description: "compilersutra-perf 0.2.0 — Alpha UX release: quickstart, energy, --diff-optimize, reports, and honest metrics." +displayed_sidebar: csperfSidebar +keywords: + - compilersutra-perf 0.2.0 + - csperf 0.2.0 + - release notes +--- + +# Release 0.2.0 + +**CompilerSutraPerf** 0.2.0 · **Date:** 2026-09-02 · **Status:** Current · Alpha (not v1.0) + +User-experience release: easier day-one path, energy + reports, opt-level sweeps, and clearer measurement provenance. Package: [PyPI: `compilersutra-perf`](https://pypi.org/project/compilersutra-perf/). + +## Install this release + +```python +pip install 'compilersutra-perf==0.2.0' +# or latest 0.2.x: +# pip install 'compilersutra-perf>=0.2.0,<0.3' +csperf --version +csperf quickstart --output-dir results/quickstart +``` + +Extras: `pip install 'compilersutra-perf[excel,pdf,visualize]'` — see [PyPI](https://pypi.org/project/compilersutra-perf/). + +Requires **Python 3.11+**. Linux is primary. Windows → WSL2. + +## Highlights + +| Area | What shipped | +| --- | --- | +| Day one | `csperf quickstart`, stronger `doctor` (real `perf` probe, auto `LD_LIBRARY_PATH`) | +| Compare | `--diff-optimize` (`-O0`…`-O3`) + comparison HTML/PDF | +| Energy | `power` block, `list-energy-backends`, `--energy` / `--no-energy` / `--energy-backend` | +| Reports | Modular HTML/PDF (`list-report-formats`), energy section | +| Honesty | `metrics_availability`, `profiling_status`, `host_platform`; methodology docs | +| Experiments | `--experiment` / `--tile-size` / `--opt-config` compile macros | +| Devices | `list-devices --backend gpu` readiness (`ok` / `partial` / `unavailable`) | + +## Backends in 0.2.0 + +| Backend | Status | +| --- | --- | +| CPU (Linux) | Stable | +| OpenCL / HIP | Experimental | +| Vulkan | Experimental (validate only) | +| macOS CPU | Experimental | +| CUDA / Metal | Planned | + +## Docs for this line + +| Page | Use | +| --- | --- | +| [Tutorial](/docs/project/compilersutra-perf/tutorial/) | Hands-on (perfwiki-style) | +| [Getting started](/docs/project/compilersutra-perf/getting-started/) | First run | +| [Usage](/docs/project/compilersutra-perf/usage/) | CLI + recipes | +| [Methodology](/docs/project/compilersutra-perf/methodology/) | Reading counters | +| [Architecture](/docs/project/compilersutra-perf/architecture/) | Layers / phases | +| [Observatory](/docs/project/compilersutra-perf/observatory/) | Vision + roadmap | +| [Energy & reports](/docs/project/compilersutra-perf/energy-and-reports/) | RAPL / HTML / PDF | +| [Troubleshooting](/docs/project/compilersutra-perf/troubleshooting/) | Failures | + +## Changelog (summary) + +### Added + +- `--experiment` / `--tile-size` / `--opt-config`; `experiment_application` in results +- Separate-execution `perf`/`papi` honors `--cpu-affinity` on Linux +- macOS `powermetrics` while the workload runs +- `docs/METHODOLOGY.md`; troubleshooting + GETTING_STARTED aligned with README +- `execution.metrics_availability` / `profiling_status` +- Dashboard multi-result selector and unmatched CPU-vs-GPU flags +- `list-devices --backend gpu` readiness +- `--diff-optimize` + optional `--report-format` comparison reports +- Energy framework + `power` block (RAPL, macOS package, AMD GPU) +- Modular report export (HTML/PDF) +- `csperf doctor` real `perf` probe; auto `LD_LIBRARY_PATH` for perf +- Compile phase 60s floor vs `--execution-timeout` +- `csperf quickstart` + +### Changed + +- `--cpu-affinity` rejected off Linux +- Clearer exit codes / `error:` + `hint:` on failed runs +- `execution.host_platform` recorded +- Docs/README match rocprofv3 and default `-O3` behavior + +### Deferred + +- Autotune / baseline store / NUMA (post-0.2) +- Experiment plugins (0.3.0) + +## Next / previous + +- Previous: [0.1.2](/docs/project/compilersutra-perf/releases/0.1.2/) +- All releases: [Releases](/docs/project/compilersutra-perf/releases/) +- Hub: [CompilerSutraPerf](/docs/project/compilersutra-perf/) diff --git a/docs/project/compilersutra-perf/releases/index.md b/docs/project/compilersutra-perf/releases/index.md new file mode 100644 index 00000000..f1585ece --- /dev/null +++ b/docs/project/compilersutra-perf/releases/index.md @@ -0,0 +1,22 @@ +--- +title: "Releases" +sidebar_label: "Releases" +description: "All compilersutra-perf releases — open a version for notes, install pin, and highlights." +displayed_sidebar: csperfSidebar +keywords: + - csperf releases + - compilersutra-perf changelog +--- + +# Releases + +**CompilerSutraPerf** release pages ([PyPI: `compilersutra-perf`](https://pypi.org/project/compilersutra-perf/)). Prefer the **current** line unless you are reproducing an older experiment. + +| Release | Status | Open | +| --- | --- | --- | +| **0.2.0** | Current · Alpha | [Release 0.2.0](/docs/project/compilersutra-perf/releases/0.2.0/) | +| 0.1.2 | Maintenance | [Release 0.1.2](/docs/project/compilersutra-perf/releases/0.1.2/) | +| 0.1.1 | Infrastructure | [Release 0.1.1](/docs/project/compilersutra-perf/releases/0.1.1/) | +| 0.1.0 | First MVP | [Release 0.1.0](/docs/project/compilersutra-perf/releases/0.1.0/) | + +Back to [CompilerSutraPerf overview](/docs/project/compilersutra-perf/). diff --git a/docs/project/compilersutra-perf/troubleshooting.md b/docs/project/compilersutra-perf/troubleshooting.md new file mode 100644 index 00000000..f29a50ba --- /dev/null +++ b/docs/project/compilersutra-perf/troubleshooting.md @@ -0,0 +1,98 @@ +--- +title: "Troubleshooting" +description: "Fix common compilersutra-perf failures: doctor, perf, RAPL, GPU, WSL2." +displayed_sidebar: csperfSidebar +keywords: + - csperf troubleshooting + - perf symbol lookup +--- + +# Troubleshooting + +Common first-run issues for **CompilerSutraPerf** 0.2.0 ([PyPI: `compilersutra-perf`](https://pypi.org/project/compilersutra-perf/)). + +## First check + +```python +csperf doctor +csperf quickstart --output-dir results/quickstart +csperf doctor --install +``` + +`quickstart` = doctor + example + profile + HTML report. + +## `perf` symbol lookup (Ubuntu / AMD) + +**Symptom:** `LLVMInitializeVETargetMC` or `symbol lookup error`. + +```python +export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH} +csperf doctor +``` + +On aarch64 use `/usr/lib/aarch64-linux-gnu`. + +Or: + +```python +csperf run --input program.cpp --backend cpu --cpu-profiler papi +``` + +## RAPL / energy unavailable + +```python +sudo sysctl kernel.perf_event_paranoid=-1 +csperf list-energy-backends +``` + +Wall time only: + +```python +csperf run --input program.cpp --backend cpu --no-perf --no-energy +``` + +Energy is separate from `--no-perf` — use `--no-energy` when you only want timing. + +## No OpenCL / HIP devices + +```python +csperf list-devices --backend gpu +csperf doctor --all +``` + +CPU-only work does not need GPU packages. + +## Compile vs run timeout + +`--execution-timeout` bounds **workload execution**, not compilation (compile keeps a ~60s floor). + +## PDF missing + +```python +pip install 'compilersutra-perf[pdf]' +csperf list-report-formats +``` + +## Windows + +Native Windows is unsupported. Use **WSL2**, install Python 3.11+ and a toolchain there, then `pip install compilersutra-perf`. Open JSON/HTML on Windows if you want. + +## macOS + +- `--no-perf` for CPU timing +- powermetrics needs passwordless `sudo` +- `--cpu-affinity` rejected +- Metal not implemented + +## Still stuck? + +1. `csperf --version` / `pip show compilersutra-perf` +2. Re-run `doctor` +3. Check `execution.perf_error`, `profiling_status`, `metrics_availability` +4. [Methodology](/docs/project/compilersutra-perf/methodology/) + +## See also + +- [Getting started](/docs/project/compilersutra-perf/getting-started/) +- [Energy & reports](/docs/project/compilersutra-perf/energy-and-reports/) +- [Usage](/docs/project/compilersutra-perf/usage/) diff --git a/docs/project/compilersutra-perf/tutorial.md b/docs/project/compilersutra-perf/tutorial.md new file mode 100644 index 00000000..36e4756b --- /dev/null +++ b/docs/project/compilersutra-perf/tutorial.md @@ -0,0 +1,341 @@ +--- +title: "Tutorial" +sidebar_label: "Tutorial" +description: "Demonstrate CompilerSutraPerf through example runs — compile, count, and read structured results on Linux-first workflows." +displayed_sidebar: csperfSidebar +keywords: + - csperf tutorial + - perfwiki + - perf stat + - compilersutra-perf +--- + +# Tutorial + +CompilerSutraPerf is a performance experimentation tool for Linux-first workflows that abstracts compile, run, and profile steps for C/C++ programs and GPU kernels (OpenCL, HIP, Vulkan shaders) behind a simple command-line interface. It orchestrates existing toolchains (Clang, GCC, hipcc, and others) and, on Linux, builds on the kernel `perf_events` interface via `perf` (with optional PAPI fallback) so hardware differences are handled by the same counting path you would use with `perf stat`. The CLI entry point is `csperf` ([PyPI: `compilersutra-perf`](https://pypi.org/project/compilersutra-perf/)). This article demonstrates CompilerSutraPerf through example runs. Commands target the **0.2.0** Alpha line (`pip install 'compilersutra-perf==0.2.0'`). For readability, some output and paths are abbreviated with ellipsis (`[...]`). + +:::note Prerequisites +- Linux recommended (CPU + `perf` path) +- Python 3.11+ +- Release notes: [0.2.0](/docs/project/compilersutra-perf/releases/0.2.0/) +::: + +```python +pip install 'compilersutra-perf==0.2.0' +csperf doctor +``` + +--- + +## Introduction + +CompilerSutraPerf accepts a source file, selects a backend, plans or executes a compile-and-run pipeline, and writes structured JSON/CSV artifacts (optional HTML/PDF reports). On the CPU path it is not a separate profiler implementation: timed trials go through a native runner, and hardware counts are collected with Linux `perf` when available. The CLI (`csperf`) is subcommand-based, reminiscent of `git` and of `perf` itself: one tool, several commands (`run`, `profile`, `diff`, …). + +It does not replace vendor compilers or `perf`. It sequences them and records provenance so later comparisons know what was measured. + +--- + +## Commands + +List what your install knows: + +```python +csperf --help +csperf list-backends +``` + +The commands you will use most in this tutorial: + +| Command | Role (perfwiki analogue) | +| --- | --- | +| `doctor` | Sanity check tools (`perf`, compilers, …) | +| `quickstart` | One-shot smoke: doctor + example + report | +| `run` | Compile + execute + optional counters (**like `perf stat`**) | +| `profile` | Summarize a saved JSON | +| `diff` | Compare two result files (**like comparing two `perf` runs**) | +| `visualize` | HTML/PDF report from JSON | +| `list-devices` | What CPU/GPU targets exist | +| `list-energy-backends` | What power collectors are available | +| `list-experiments` | Built-in compile-macro experiments | + +Get help for one command the same way as `perf stat -h`: + +```python +csperf run --help +``` + +--- + +## Events + +Before measuring, know **what** you are counting. The perfwiki tutorial splits events into sources; `csperf` inherits the same ideas when it wraps `perf`. + +| Kind | Meaning | Examples | +| --- | --- | --- | +| **Software** | Kernel counters | context-switches, page-faults | +| **Hardware / PMU** | CPU Performance Monitoring Unit | cycles, instructions, branch-misses | +| **Generic names** | Portable aliases mapped to a real PMU event *if present* | `cycles`, `instructions`, `branches` | +| **Energy / power** | Package or device power domains | RAPL `power/energy-pkg/`, `rocm-smi` | + +Common hardware names you will see: + +| Event | Tutorial meaning | +| --- | --- | +| `cycles` | Total cycles (frequency scaling can confuse interpretation) | +| `ref-cycles` | Cycles not affected by frequency scaling | +| `instructions` | Retired instructions | +| `branches` | Retired branches | +| `branch-misses` | Mispredicted branches | +| `cache-references` / `cache-misses` | Usually LLC-oriented; use as a pair for miss rate | + +Derived metrics (like the `# IPC` line in `perf stat`) include **IPC** and **branch-miss rate**. `csperf profile` surfaces those when the artifact has the raw counts. + +:::caution +Hardware events are **CPU-specific**. A name that works on one machine may be unavailable or mapped differently on another. Always check `execution.metrics_availability` / `profiling_status` in the JSON. +::: + +--- + +## Counting with `csperf run` + +In counting mode, `perf` aggregates events over a run and prints them at the end (`perf stat`). `csperf run` does the same idea, but also **compiles** first and writes JSON. + +### 1. Smoke path (no thinking required) + +```python +csperf quickstart --output-dir results/quickstart +``` + +This runs `doctor`, a bundled CPU example, a profile summary, and (when possible) an HTML report. If this fails, fix the environment before continuing ([Troubleshooting](/docs/project/compilersutra-perf/troubleshooting/)). + +### 2. Your first timed run + +```python +csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu \ + --warmup-runs 1 --repeat-runs 3 \ + --output results/cpu.json + +csperf profile results/cpu.json +``` + +What you should look for (same discipline as reading `perf stat` output): + +1. **Did it execute?** — status / trial results +2. **How long?** — `metrics.execution_time_ms` +3. **Were counters collected?** — `execution.profiling_status`, `metrics_availability` +4. **Same process as the timer?** — `execution.measurement.same_execution` +5. **Can I reproduce?** — `commands`, compiler flags, `tool_version`, `hardware` + +### 3. Time only (no counters) + +Like `perf stat -n` (null run) in spirit — skip hardware counters: + +```python +csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu \ + --no-perf --no-energy \ + --warmup-runs 1 --repeat-runs 3 \ + --output results/time-only.json +``` + +Use this when `perf` is blocked or you only care about wall time. + +### 4. Pin a CPU (reduce noise) + +`perf` can restrict CPUs with `-C`. On Linux, pin the workload: + +```python +csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu \ + --cpu-affinity 0 \ + --warmup-runs 1 --repeat-runs 5 \ + --output results/cpu-pinned.json +``` + +`--cpu-affinity` is **Linux-only** (rejected on macOS). + +--- + +## Options that control what you measure + +### Compiler and flags + +Default CPU compiler is `clang++` / `clang` unless overridden. **`CXXFLAGS` is not read.** + +```python +csperf run --input program.cpp --compiler g++ --output results/gcc.json +csperf run --input program.cpp --compiler clang++ --output results/clang.json + +csperf run --input program.cpp \ + --compiler-flag=-O2 \ + --compiler-flag=-march=native \ + --output results/o2.json +``` + +Default optimization is `-O3` unless you pass another `-O*`. + +### Profiler choice + +```python +csperf run --input program.cpp --backend cpu --cpu-profiler auto +csperf run --input program.cpp --backend cpu --cpu-profiler perf +csperf run --input program.cpp --backend cpu --cpu-profiler papi +``` + +| Mode | Notes | +| --- | --- | +| `auto` | Try `perf`, fall back to PAPI if broken | +| `perf` | Prefer for same-execution Linux counters | +| `papi` / `papi-native` | Alternate paths — see [Methodology](/docs/project/compilersutra-perf/methodology/) | +| `--no-perf` | Disable all hardware counters | + +### Energy (optional, independent of `perf`) + +```python +csperf list-energy-backends +csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu \ + --energy-backend amd-cpu --output results/cpu-energy.json +csperf run --input program.cpp --backend cpu --no-energy +``` + +Package RAPL energy is **not** per-process joules. Details: [Energy & reports](/docs/project/compilersutra-perf/energy-and-reports/). + +### See the plan without running + +```python +csperf run --input program.cpp --backend cpu --plan-only +``` + +--- + +## Multiplexing and “is this count real?” + +The perfwiki tutorial emphasizes: if you ask for **more events than hardware counters**, the kernel **multiplexes** and later **scales** counts: + +```text +final_count ≈ raw_count × (time_enabled / time_running) +``` + +That is an **estimate**. `csperf` inherits this when it asks `perf` for many events. + +Rules of thumb (same as perfwiki): + +1. Prefer a **small** event set for a claim (e.g. branches + branch-misses). +2. Prefer `same_execution: true` in the JSON. +3. Use enough repeats to see variance. +4. Never mix `perf` counters with PAPI / powermetrics as if they were one series. + +Full write-up: [Methodology](/docs/project/compilersutra-perf/methodology/). + +--- + +## Repeated measurement + +`perf stat -r 5` runs the workload several times and reports mean ± spread. In `csperf`: + +```python +csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu \ + --warmup-runs 1 --repeat-runs 5 \ + --output results/cpu-r5.json +csperf profile results/cpu-r5.json +``` + +- **Warmup** trials are discarded from the primary timing summary. +- **Repeat** trials populate `execution.trial_results` and summary stats (min / mean / …). + +Defaults without overrides are higher (`warmup-runs 3`, `repeat-runs 15`). Start small while learning. + +### Opt-level sweep (built-in) + +```python +csperf run --input examples/cpp/matrix_traversal.cpp \ + --diff-optimize --no-perf --report-format both +``` + +Runs `-O0`…`-O3`, writes per-level results under `results/optimize/`, and a comparison report. + +--- + +## Compare two runs + +Like looking at two `perf` experiments side by side: + +```python +csperf run --input program.cpp --compiler g++ --output results/gcc.json +csperf run --input program.cpp --compiler clang++ --output results/clang.json +csperf diff results/gcc.json results/clang.json --csv results/gcc-vs-clang.csv +``` + +`diff` warns when profiler sources or availability differ. Empty or skipped metrics are better than fake zeros. + +--- + +## GPU path (short) + +Same command shape; different middle step: + +```python +csperf list-devices --backend gpu + +csperf run --input path/to/kernel.hip --backend hip \ + --device-index 0 --warmup-runs 1 --repeat-runs 3 \ + --output results/hip.json +``` + +| Backend | Today | +| --- | --- | +| HIP / OpenCL | Experimental execute | +| Vulkan | Validate shader only (`status: partial`) — **not** a full kernel benchmark | + +Do not compare unmatched CPU vs GPU workloads. + +--- + +## How to read a result file + +Open `results/cpu.json` (or use `csperf profile`) and ask, in order: + +| Question | Field | +| --- | --- | +| How long? | `metrics.execution_time_ms` | +| Trial spread? | `execution.trial_results` | +| Same-exec counters? | `execution.measurement.same_execution` | +| What was counted? | metrics + `perf_source` | +| Why missing? | `execution.perf_error`, `profiling_status`, `metrics_availability` | +| Reproduce? | `commands`, flags, `hardware`, `tool_version` | + +That is the same habit as reading a `perf stat` block: elapsed time → counts → derived ratios → caveats. + +--- + +## Common mistakes + +| Mistake | Fix | +| --- | --- | +| `list-devices --backend` with no value | Always pass `cpu` / `gpu` / `hip` / … | +| Treating scaled / separate-execution counts as continuous same-exec truth | Check `same_execution` | +| Publishing OpenCL times without `--opencl-measure` | State the measure mode | +| Vulkan “success” as a speedup | Validation ≠ dispatch | +| Assuming `device-index 0` is the same GPU everywhere | Re-list per backend | +| Package RAPL as “this binary used X J” | Package ≠ process | + +--- + +## Checklist + +- [ ] `csperf doctor` / `quickstart` OK +- [ ] One own `results/*.json` from `csperf run` +- [ ] You can explain wall time vs counters in that file +- [ ] You checked `same_execution` before quoting branch/cache numbers +- [ ] You used repeats (or `--diff-optimize`) before claiming a compiler win + +--- + +## Next + +| Goal | Page | +| --- | --- | +| Flag reference | [Usage](/docs/project/compilersutra-perf/usage/) | +| Events, scaling, publish rules | [Methodology](/docs/project/compilersutra-perf/methodology/) | +| Layers / phases | [Architecture](/docs/project/compilersutra-perf/architecture/) | +| Research vision | [Observatory](/docs/project/compilersutra-perf/observatory/) | +| Failures | [Troubleshooting](/docs/project/compilersutra-perf/troubleshooting/) | +| External deep dive | [perfwiki tutorial](https://perfwiki.github.io/main/tutorial/) | diff --git a/docs/project/compilersutra-perf/usage.md b/docs/project/compilersutra-perf/usage.md new file mode 100644 index 00000000..2ebe7a88 --- /dev/null +++ b/docs/project/compilersutra-perf/usage.md @@ -0,0 +1,217 @@ +--- +title: "Usage" +description: "How-to guides and CLI reference for CompilerSutraPerf (csperf) 0.2.0." +displayed_sidebar: csperfSidebar +keywords: + - csperf CLI + - compiler comparison + - workload manifest +--- + +# Usage + +Task recipes and flags for **CompilerSutraPerf** **0.2.0** ([PyPI](https://pypi.org/project/compilersutra-perf/)). First install: [Getting started](/docs/project/compilersutra-perf/getting-started/) · [Tutorial](/docs/project/compilersutra-perf/tutorial/). + +## Support level (summary) + +- **CPU:** Python orchestrates compile; native C++ runner executes; optional Linux `perf` / PAPI +- **`--backend gpu`:** `.hip` → HIP, `.cl` → OpenCL, shaders → Vulkan +- **HIP:** `hipcc` + ROCm; rocprofv3 → v2 → v1; `kernel_time_ms` ≠ `hip_profiler_*` +- **OpenCL:** build, launch, argument binding, warmup/repeat +- **Vulkan:** GLSL → SPIR-V + shader-module validate (no full dispatch) +- **CUDA / Metal:** planned, not executable +- **macOS CPU:** experimental; use `--no-perf`; powermetrics needs passwordless `sudo` + +## How do I… + +### …run a CPU workload + +```python +csperf run --input program.cpp --backend cpu \ + --warmup-runs 1 --repeat-runs 5 --output results/cpu.json +csperf profile results/cpu.json +``` + +### …choose a compiler + +Order: `CXX_COMPILER` / `C_COMPILER` → `CXX` / `CC` → default `clang++` / `clang`. +HIP always uses `hipcc` when available. **`CXXFLAGS` / `CFLAGS` are not read.** + +```python +csperf run --input program.cpp --compiler g++ --output results/gcc.json +CC=gcc CXX=g++ csperf run --input program.cpp --output results/gcc-env.json +CXX=g++-13 csperf run --input program.cpp --output results/gpp13.json +``` + +### …pass flags + +`-O3` is default unless you pass another `-O*`: + +```python +csperf run --input program.cpp --backend cpu \ + --compiler-flag=-march=native \ + --compiler-flag=-funroll-loops \ + --output results/optimized.json +``` + +### …compare two compilers + +```python +csperf run --input program.cpp --compiler g++ --output results/gcc.json +csperf run --input program.cpp --compiler clang++ --output results/clang.json +csperf diff results/gcc.json results/clang.json \ + --csv results/gcc-vs-clang.csv \ + --derived-config configs/derived_metrics.sample.json +``` + +### …sweep `-O0`…`-O3` + +```python +csperf run --input examples/cpp/matrix_traversal.cpp \ + --diff-optimize --no-perf --report-format both +``` + +### …use a manifest + +```json +{ + "schema_version": 1, + "source": "solver.cpp", + "backend": "cpu", + "compiler": "g++", + "compiler_flags": ["-O3", "-march=native"], + "program_args": ["--iterations", "1000000"], + "stdin_file": "input.txt", + "execution_timeout_s": 30, + "warmup_runs": 1, + "repeat_runs": 5, + "use_perf": true, + "use_energy": true, + "energy_backends": "auto" +} +``` + +```python +csperf run --manifest workloads/solver.json --output results/solver.json +``` + +### …pick a CPU profiler + +```python +csperf run --input program.cpp --backend cpu --cpu-profiler auto +csperf run --input program.cpp --backend cpu --cpu-profiler perf +csperf run --input program.cpp --backend cpu --cpu-profiler papi +csperf run --input program.cpp --backend cpu --cpu-profiler papi-native +``` + +| Mode | Notes | +| --- | --- | +| `auto` | `perf` first; fall back to PAPI if broken | +| `perf` | Prefer for same-execution counters | +| `papi` | Often separate execution | +| `papi-native` | Same-execution when runner built with PAPI | +| `--no-perf` | Disables all hardware counters | + +### …run OpenCL / HIP / Vulkan + +```python +csperf run --input examples/opencl/saxpy.cl --backend opencl --device-index 0 \ + --kernel-name saxpy \ + --kernel-arg buffer:float:read:4096:1.0 \ + --kernel-arg scalar:uint32:4096 \ + --readback-arg 2 + +csperf run --input kernel.hip --backend hip --device-index 0 --output results/hip.json + +csperf run --input examples/shaders/vector_add.comp --backend vulkan \ + --device-index 0 --output results/vulkan.json +``` + +State `--opencl-measure` (`setup` / `steady_state` / `end_to_end`) when publishing OpenCL times. + +### …inspect without executing + +```python +csperf run --input program.cpp --backend cpu --plan-only +``` + +### …list hardware + +```python +csperf list-backends +csperf list-experiments +csperf list-energy-backends +csperf list-report-formats +csperf cpuinfo +csperf gpuinfo +csperf deviceinfo +csperf list-devices --backend gpu +``` + +## CLI cheat sheet + +| Flag | Purpose | +| --- | --- | +| `--warmup-runs N` | default `3` | +| `--repeat-runs N` | default `15` | +| `--no-perf` | skip hardware counters | +| `--plan-only` | print commands only | +| `--build-dir PATH` | binary / IR output dir | +| `--cpu-affinity 0,1` | Linux only | +| `--device-index N` | GPU device | +| `--backend gpu` | auto-select by file type | +| `--compiler=PATH` | CPU compiler | +| `--compiler-flag=FLAG` | repeatable | +| `--program-arg ARG` | workload argv | +| `--stdin-file PATH` | workload stdin | +| `--execution-timeout SECONDS` | bounds **run**, not compile | +| `--diff-optimize` | `-O0`…`-O3` + report | +| `--energy` / `--no-energy` | energy on by default when available | +| `--energy-backend KEY` | `auto`, `amd-cpu`, `amd-gpu`, … | +| `--cpu-profiler` | `auto` \| `perf` \| `papi` \| `papi-native` | +| `--json` | machine-readable stdout | +| `--version` | package version | + +## Experiments + +```python +csperf list-experiments +csperf run --input examples/cpp/matrix_traversal.cpp --experiment row-major +csperf run --input examples/cpp/tiled_matmul.cpp --experiment tiled --tile-size 32 +``` + +Macros: `row-major` → `-DCSPERF_ROW_MAJOR=1`, `column-major` → `-DCSPERF_COLUMN_MAJOR=1`, `tiled` → `-DCSPERF_TILE_SIZE=N`. + +## Result files + +Each run writes `results/.json` and `.csv` (`.xlsx` with `[excel]`). + +Stable sections: `pipeline`, `commands`, `hardware`, `metrics`, `execution`, `artifacts`, optional `power`. +Metadata: `schema_version`, `tool_version`, `generated_at_utc`. + +## Exit codes + +| Code | Meaning | +| --- | --- | +| `0` | `executed`, `partial`, or `planned` | +| `1` | failed / fatal | +| `2` | usage error | + +Incomplete counters can still exit `0` — check `execution.profiling_status` and `metrics_availability`. + +## Batch compiler folder diff + +With a local checkout that includes the script: + +```python +python scripts/compiler_diff_batch.py examples/cpp \ + --config1 configs/compiler_gcc.sample.json \ + --config2 configs/compiler_clang.sample.json +``` + +## Next + +- [Methodology](/docs/project/compilersutra-perf/methodology/) +- [Observatory](/docs/project/compilersutra-perf/observatory/) +- [Energy & reports](/docs/project/compilersutra-perf/energy-and-reports/) +- [Troubleshooting](/docs/project/compilersutra-perf/troubleshooting/) diff --git a/docs/project/Project.md b/docs/project/cpp-project-ideas.md similarity index 96% rename from docs/project/Project.md rename to docs/project/cpp-project-ideas.md index 06677123..421443c3 100644 --- a/docs/project/Project.md +++ b/docs/project/cpp-project-ideas.md @@ -1,82 +1,36 @@ --- title: "C++ Project Ideas: Domain-Specific" -description: "Explore the best C++ project ideas for all levels—beginner, intermediate, and advanced—across different domains like system programming, AI, game development, networking, and more." -keywords: -- C++ projects -- beginner C++ -- advanced C++ -- LLVM -- system programming -- game development -- AI projects -- networking -- C++ programming -- data structures -- algorithms -- multithreading -- cloud computing -- blockchain -- cybersecurity -- embedded systems -- compiler development -- game AI -- machine learning -- deep learning -- robotics -- IoT projects -- performance optimization -- GPU programming -- real-time systems -- operating systems -- high-performance computing -- parallel computing -- numerical computing -- database systems -- financial software -- automotive software -- scientific computing -- autonomous systems -- cloud security -- ethical hacking -- C++ frameworks -- natural language processing -- quantum computing -- DevOps with C++ -- microservices in C++ -- cryptography - +description: "Explore C++ project ideas for beginner, intermediate, and advanced levels across systems, AI, games, networking, compilers, and more." +displayed_sidebar: projectSidebar +keywords: + - C++ projects + - beginner C++ + - advanced C++ + - LLVM + - system programming + - game development + - AI projects + - networking + - compiler development + - GPU programming + - embedded systems tags: -- C++Projects -- LLVM -- Game Development -- Networking -- System Programming -- AI -- Compiler -- Data Structures -- Multithreading -- Blockchain -- Cybersecurity -- Cloud Computing -- Embedded Systems -- Game AI -- Robotics -- IoT -- Machine Learning -- Deep Learning -- GPU Programming -- Quantum Computing -- High-Performance Computing -- Cryptography -- Database Systems -- Financial Software -- Autonomous Systems -- Ethical Hacking - + - C++Projects + - LLVM + - Game Development + - Networking + - System Programming + - AI + - Compiler --- + import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; +:::note Part of the project catalogue +This page is one catalogue entry. For all projects (CompilerSutraPerf, LLVM, automation, and more), start at [Projects](./index). +::: + # 💻 C++ Project Ideas by Domain @@ -1273,29 +1227,8 @@ Here are some projects to build expertise in compiler development: ::: --- +## Back to the catalogue +Looking for other projects (CompilerSutraPerf, LLVM / VELOX, Python automation)? -## LLVM Projects - -If you want a guided LLVM project hub, start here: - -- [LLVM Projects](./llvm) - -The main project in that folder is: - -- [VELOX Compiler Project](./llvm/VELOX/) - -This series covers: - -- compiler theory and architecture -- a custom frontend for `VELOX` -- lowering to LLVM IR -- optimization passes -- a RISC-V backend - ---- - -## 🚀 Get Started! -Choose a project that interests you and start building! Each project offers an opportunity to apply your C++ knowledge and explore various domains. - -Happy Coding! 🎯 +- [Projects catalogue](./index) diff --git a/docs/project/index.md b/docs/project/index.md new file mode 100644 index 00000000..cc1a5ded --- /dev/null +++ b/docs/project/index.md @@ -0,0 +1,77 @@ +--- +title: "Projects" +description: "Catalogue of CompilerSutra projects — tools, LLVM labs, C++ ideas, and automation workspaces." +displayed_sidebar: projectSidebar +keywords: + - CompilerSutra projects + - project catalogue + - CompilerSutraPerf + - LLVM projects + - VELOX + - C++ project ideas + - Python automation +--- + +# Projects + +This is the **project catalogue**. Each entry below is a separate workspace with its own docs. Use this page to pick a project; do not dump new material into one shared file. + +:::tip How this section grows +Add a new row to the catalogue when you start a project. Keep long write-ups under that project's folder (`docs/project//`), not on this page. +::: + +## Project Catalogue + +| Project | What it is | Status | Start here | +| --- | --- | --- | --- | +| [CompilerSutraPerf](/docs/project/compilersutra-perf/) | `csperf` — release-first docs; compile, run, profile, compare · [PyPI](https://pypi.org/project/compilersutra-perf/) | Active · Alpha 0.2.0 | [0.2.0 release](/docs/project/compilersutra-perf/releases/0.2.0/) | +| [LLVM Projects](./llvm/) | Guided LLVM labs, starting with the VELOX compiler | Active | [VELOX](./llvm/VELOX/) | +| [C++ Project Ideas](./cpp-project-ideas) | Domain-specific C++ project ideas by skill level | Reference | [Browse ideas](./cpp-project-ideas) | +| [Python Automation](./python_automation/python_automation) | Automation scripts and productivity projects | Active | [Python Automation](./python_automation/python_automation) | + +## Featured Projects + +### CompilerSutraPerf + +Performance experimentation CLI (**CompilerSutraPerf** / `csperf`) for C/C++, HIP, OpenCL, and Vulkan — with an observatory roadmap for honest compiler research. + +- Install from PyPI, profile with Linux `perf`, export JSON/HTML reports +- Hub: [CompilerSutraPerf](/docs/project/compilersutra-perf/) · [0.2.0](/docs/project/compilersutra-perf/releases/0.2.0/) · [Tutorial](/docs/project/compilersutra-perf/tutorial/) · [Observatory](/docs/project/compilersutra-perf/observatory/) +- Package: [PyPI — `compilersutra-perf`](https://pypi.org/project/compilersutra-perf/) + +### LLVM Projects + +Build real compiler pipelines on LLVM — frontend → IR → optimize → backend → QEMU. + +- Featured series: [VELOX](./llvm/VELOX/) +- Hub: [LLVM Projects](./llvm/) + +### C++ Project Ideas + +Large list of beginner → advanced C++ project ideas across systems, AI, games, networking, and compilers. + +- [C++ Project Ideas](./cpp-project-ideas) + +### Python Automation + +Automation-focused projects and walkthroughs. + +- [Python Automation](./python_automation/python_automation) +- Example: [System Specs Collector](./python_automation/automate_boring_stuff/system-specs-collector/system_spec_collector) + +## Related Hubs (not projects) + +These live outside `/docs/project/` but are often used with the work above: + +| Hub | Link | +| --- | --- | +| Articles | [Articles catalogue](/docs/articles) | +| Labs | [Labs](/docs/labs) | +| Tools | [Tools](/docs/tools) | + +## Adding a New Project + +1. Create `docs/project//index.md` (hub page). +2. Add a row to the **Project Catalogue** table on this page. +3. Register the hub in `sidebars/site.js` under `projectSidebar` (and a dedicated sidebar if the project has many pages). +4. Keep article-length content in that folder — not on this catalogue page. diff --git a/docs/project/llvm/index.md b/docs/project/llvm/index.md index 298c4a81..cf1d8e36 100644 --- a/docs/project/llvm/index.md +++ b/docs/project/llvm/index.md @@ -17,6 +17,10 @@ import TabItem from '@theme/TabItem'; # LLVM Projects +:::note Project catalogue +This is one entry in the [Projects catalogue](../index). See also [CompilerSutraPerf](../compilersutra-perf/) ([PyPI](https://pypi.org/project/compilersutra-perf/)) and other workspaces there. +::: + If you want to build real compiler projects on top of LLVM, this is the right place to start. This folder is organized around one main idea: diff --git a/docusaurus.config.js b/docusaurus.config.js index 14139e8b..f1c250c0 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -49,6 +49,10 @@ const config = { '@docusaurus/plugin-client-redirects', { redirects: [ + { + to: '/docs/project/cpp-project-ideas', + from: ['/docs/project/Project', '/docs/project/Project/'], + }, { to: '/docs/c++/advanced/', from: ['/docs/c++/advance/', '/docs/c++/advance/index', '/docs/c++/advance/intro'] }, { to: '/docs/dsa/foundations/data/', diff --git a/sidebars/site.js b/sidebars/site.js index 4f3cf4e9..5747c152 100644 --- a/sidebars/site.js +++ b/sidebars/site.js @@ -21,18 +21,115 @@ const site = { }, ], - projectSidebar: [ + csperfSidebar: [ { type: 'category', - label: 'Project', + label: 'CompilerSutraPerf', collapsed: false, - items: ['project/Project'], + link: { + type: 'doc', + id: 'project/compilersutra-perf/index', + }, + items: [ + { + type: 'category', + label: 'Releases', + collapsed: false, + link: { + type: 'doc', + id: 'project/compilersutra-perf/releases/index', + }, + items: [ + 'project/compilersutra-perf/releases/0.2.0', + 'project/compilersutra-perf/releases/0.1.2', + 'project/compilersutra-perf/releases/0.1.1', + 'project/compilersutra-perf/releases/0.1.0', + ], + }, + 'project/compilersutra-perf/tutorial', + 'project/compilersutra-perf/getting-started', + 'project/compilersutra-perf/usage', + 'project/compilersutra-perf/methodology', + 'project/compilersutra-perf/observatory', + 'project/compilersutra-perf/architecture', + 'project/compilersutra-perf/energy-and-reports', + 'project/compilersutra-perf/troubleshooting', + ], }, + ], + + projectSidebar: [ { type: 'category', - label: 'LLVM Projects', + label: 'Projects', collapsed: false, - items: ['project/llvm/index'], + link: { + type: 'doc', + id: 'project/index', + }, + items: [ + { + type: 'category', + label: 'CompilerSutraPerf', + collapsed: false, + link: { + type: 'doc', + id: 'project/compilersutra-perf/index', + }, + items: [ + { + type: 'category', + label: 'Releases', + collapsed: false, + link: { + type: 'doc', + id: 'project/compilersutra-perf/releases/index', + }, + items: [ + 'project/compilersutra-perf/releases/0.2.0', + 'project/compilersutra-perf/releases/0.1.2', + 'project/compilersutra-perf/releases/0.1.1', + 'project/compilersutra-perf/releases/0.1.0', + ], + }, + 'project/compilersutra-perf/tutorial', + 'project/compilersutra-perf/getting-started', + 'project/compilersutra-perf/usage', + 'project/compilersutra-perf/methodology', + 'project/compilersutra-perf/observatory', + 'project/compilersutra-perf/architecture', + 'project/compilersutra-perf/energy-and-reports', + 'project/compilersutra-perf/troubleshooting', + ], + }, + { + type: 'category', + label: 'LLVM Projects', + collapsed: true, + link: { + type: 'doc', + id: 'project/llvm/index', + }, + items: [ + 'project/llvm/VELOX/index', + 'project/llvm/VELOX/v1-language-spec', + 'project/llvm/VELOX/creating-your-first-llvm-based-compiler', + ], + }, + 'project/cpp-project-ideas', + { + type: 'category', + label: 'Python Automation', + collapsed: true, + link: { + type: 'doc', + id: 'project/python_automation/python_automation', + }, + items: [ + 'project/python_automation/automate_boring_stuff/system-specs-collector/system_spec_collector', + ], + }, + ], }, ], diff --git a/src/components/clang-flags/PerfReport.jsx b/src/components/clang-flags/PerfReport.jsx index 55d74f1f..a097cbe9 100644 --- a/src/components/clang-flags/PerfReport.jsx +++ b/src/components/clang-flags/PerfReport.jsx @@ -179,7 +179,15 @@ export default function PerfReport({
- Measured with compilersutra-perf + + Measured with{' '} + CompilerSutraPerf + {' '}( + + PyPI + + ) +
{note ?

{note}

: null}
diff --git a/src/pages/tools/clang-flags/flags/flag-2d44.mdx b/src/pages/tools/clang-flags/flags/flag-2d44.mdx index 6052478f..b37bc30b 100644 --- a/src/pages/tools/clang-flags/flags/flag-2d44.mdx +++ b/src/pages/tools/clang-flags/flags/flag-2d44.mdx @@ -51,7 +51,7 @@ This option does not repair invalid source, undefined behavior, data races, or a ## Performance impact -The qualitative impact depends on the target and workload. No performance numbers are included because no controlled CompilerSutra Perf run was performed for this flag. +The qualitative impact depends on the target and workload. No performance numbers are included because no controlled CompilerSutraPerf run was performed for this flag. ## Compatibility diff --git a/src/pages/tools/clang-flags/flags/flag-2d49.mdx b/src/pages/tools/clang-flags/flags/flag-2d49.mdx index e4b66b51..b621f656 100644 --- a/src/pages/tools/clang-flags/flags/flag-2d49.mdx +++ b/src/pages/tools/clang-flags/flags/flag-2d49.mdx @@ -51,7 +51,7 @@ This option does not repair invalid source, undefined behavior, data races, or a ## Performance impact -The qualitative impact depends on the target and workload. No performance numbers are included because no controlled CompilerSutra Perf run was performed for this flag. +The qualitative impact depends on the target and workload. No performance numbers are included because no controlled CompilerSutraPerf run was performed for this flag. ## Compatibility diff --git a/src/pages/tools/clang-flags/flags/flag-2d4f.mdx b/src/pages/tools/clang-flags/flags/flag-2d4f.mdx index cd438be2..ca170d7f 100644 --- a/src/pages/tools/clang-flags/flags/flag-2d4f.mdx +++ b/src/pages/tools/clang-flags/flags/flag-2d4f.mdx @@ -133,7 +133,7 @@ Do not choose `-O3` or `-Ofast` because the number is larger. Benchmark represen ## Performance impact -Higher levels generally cost more compile time. Runtime effects depend on hot loops, inlining, target ISA, code size, and the workload. This page omits `PerfReport` because no controlled CompilerSutra Perf matrix was actually run for this flag. +Higher levels generally cost more compile time. Runtime effects depend on hot loops, inlining, target ISA, code size, and the workload. This page omits `PerfReport` because no controlled CompilerSutraPerf matrix was actually run for this flag. ## Compatibility diff --git a/src/pages/tools/clang-flags/flags/flag-2d4f32.mdx b/src/pages/tools/clang-flags/flags/flag-2d4f32.mdx index 33382497..7c2da16e 100644 --- a/src/pages/tools/clang-flags/flags/flag-2d4f32.mdx +++ b/src/pages/tools/clang-flags/flags/flag-2d4f32.mdx @@ -82,7 +82,7 @@ This option does not repair invalid source, undefined behavior, data races, or a ## Performance impact -The following measurements were produced with [CompilerSutra Perf](https://pypi.org/project/compilersutra-perf/) on the prefix_sum.cpp workload using Ubuntu clang 18.1.3, x86_64-pc-linux-gnu, CPU affinity 0, one warmup run, five measured trials, with profiling disabled: +The following measurements were produced with [CompilerSutraPerf](/docs/project/compilersutra-perf/) ([PyPI](https://pypi.org/project/compilersutra-perf/)) on the prefix_sum.cpp workload using Ubuntu clang 18.1.3, x86_64-pc-linux-gnu, CPU affinity 0, one warmup run, five measured trials, with profiling disabled: Experiments - B — CompilerSutra Perf / csperf (n=100) + B — CompilerSutraPerf / csperf (n=100) summary.json · O1 vs O2 diff