This comparison measures native managed functions generated by the FScript .NET backend against equivalent F# compiled in Release mode. It also dumps both assemblies' IL. It does not measure either interpreter, Native AOT, command startup, or publishing.
Run from the repository root with the .NET 10 SDK:
make benchmark-dotnet-comparison
# Defaults: 500 invocations per batch, 9 batches per variant.
make benchmark-dotnet-comparison iterations=1000 rounds=11 output=out/comparison
# Correctness and IL only; no timing:
make benchmark-dotnet-comparison verify=trueThe Make target builds in Release, then starts a separate process with tiered compilation and ReadyToRun disabled. This makes optimized JIT code available immediately and avoids measuring a tier transition. Both variants use the same runtime and settings. To investigate the normal runtime policy instead:
dotnet build benchmarks/FScript.DotNet.Comparison -c Release
dotnet benchmarks/FScript.DotNet.Comparison/bin/Release/net10.0/FScript.DotNet.Comparison.dll 500 9 out/comparison-defaultDo not run concurrent builds or benchmarks. Repeat the process several times on an idle machine; compare medians and the per-round spread, rather than a single run. These measurements are diagnostic, not performance promises or CI timing thresholds.
workloads.fss and Reference.fs use the same signed 64-bit arithmetic, inputs, and checksums:
| Export | Work per invocation | What it exercises |
|---|---|---|
numeric |
100,000 recurrence steps | Tail recursion, captured partial application, repeated arithmetic, CSE |
pipeline |
5,000 immutable records | Map, filter, fold, record updates, nominal/structural boundaries |
pipeline_nominal |
Same pipeline, explicit record annotations | Additional diagnostic for record conversions; same F# reference |
routing |
5,000 tagged decisions | Payload unions, records, pattern matching, map/fold |
sorting |
Quicksort of 512 integers, weighted checksum | Non-tail recursion, list patterns, filtering, append, tuple accumulators |
There is also an unused helper and record to inspect pruning. Each export accepts a runtime seed and returns a checksum. The harness checks all implementations over 32 fixed seeds, including negative and large values. It asks the CLR to JIT every closed generated method, including helpers outside the executed paths. NUnit runs these checks without a performance threshold.
The F# Lists implementation is the idiomatic source comparison. FScript currently represents compiled lists as typed arrays, so F# Arrays provides another baseline using the same immutable-by-convention operations and record types. It does not replace the algorithms with hand-written mutable loops. Neither baseline uses custom inlining attributes. The quicksort concatenations have different associations: FScript appends the pivot to the left result first, while the F# array reference prepends it to the right result first. Both produce the same values, but intermediate array sizes and copy costs can differ. Compiler before/after comparisons retain their original association. FScript int is compared with F# int64, not F#'s default 32-bit int.
The harness compiles FScript once to a library, loads it, and obtains the exported Func<int64,int64> values. F# methods are also wrapped once as matching delegates. Compilation, reflection, assembly loading, export initialization, validation, and warmup are outside the measured batches. Each variant warms up for at least 500 ms and 20 invocations. The source-to-library compilation time is reported separately; it includes inference, lowering, backend optimization, and Roslyn emission, not an isolated compiler phase.
Each batch calls the cached native delegate with seeds cycling from 42 through 58 and retains a checksum. It does not call Run(), which would mostly measure cached library initialization. Variant order rotates and reverses across rounds. Full collections precede each measured batch; collection and allocation costs during the batch remain included. Allocation uses GC.GetAllocatedBytesForCurrentThread, appropriate for these synchronous workloads. Timing and allocation are reported per invocation.
results.json records each batch, checksums, median/minimum/maximum, framework/OS/architecture, JIT settings, source SHA-256, compilation duration, and IL summaries. Process-wide allocations, peak memory, startup, and hardware counters are not measured. The two F# pipeline rows intentionally reuse one implementation; their timings can show ordinary run-to-run variation.
The output directory contains:
FScript.Benchmark.Workloads.dlland the measuredworkloads.fss.fscript.il.txtandfsharp.il.txt, with resolved method operands and branch destinations.results.json, including own-assembly reachability per export, external call sites, backedges, direct-call cycles, delegate calls, boxing, allocation sites, and multiplication counts.
The .NET backend currently optimizes LIR and lowers it to native-oriented C# internally, then uses Roslyn Release emission. F# uses the F# compiler. Both resulting IL programs are subsequently optimized by the same .NET JIT. The benchmark therefore compares complete compilation paths, not two standalone IL optimization passes.
A backedge in the numeric kernel and no direct recursive call cycle provide evidence of tail-loop lowering. Quicksort still has pending append work and remains recursive. A zero self-call count alone is insufficient: recursion can span lifted helpers. Delegate calls can also hide recursion, so the cycle detector reports only direct call edges.
Reachability follows own-assembly method operands and constructed closure Invoke methods. It is an inspection aid, not a complete dynamic call graph: unknown delegate targets and code inside FSharp.Core, LINQ, and the BCL are excluded. External call sites are listed to make this limitation visible. Instruction counts are static sites, not execution counts. A value-type newobj can construct a tuple on the stack or in registers without a heap allocation; use measured allocation bytes rather than constructor counts to distinguish this. A smaller IL body need not mean faster machine code; the JIT can further inline or eliminate repeated operations.
The F# assembly contains both collection baselines, while the generated FScript assembly contains its five exports. Raw DLL sizes exclude dependencies and are not an apples-to-apples size comparison. Likewise, pruning an unused generated type does not establish whole-program trimming of a regular F# library.
See RESULTS.md for one machine's measurements and the optimization opportunities exposed by the comparison. Re-run before drawing conclusions for another runtime, CPU, or compiler revision.
Keep the workload DLLs from two runs that used the same workloads.fss, then compare them in one process:
DOTNET_gcServer=0 dotnet fsi benchmarks/FScript.DotNet.Comparison/CompareArtifacts.fsx \
out/before/FScript.Benchmark.Workloads.dll \
out/after/FScript.Benchmark.Workloads.dll \
out/paired-comparisonBuild the comparison project in Release first. This script reuses its compiled measurement loop and F# references. It loads both DLLs, checks their exports across the same 32 seeds, JIT-checks their closed methods, and balances before, after, and F# variants in every workload. No compiler execution is measured. It records both binary hashes, the actual GC mode, and all batches in results.json. The FSI host can use a different GC configuration from the standalone executable; the command above selects workstation GC. Repeat in separate processes on an idle machine; keep the same JIT environment for both inputs. The script checks results rather than proving that the supplied artifacts came from identical source, so retain their original source and metadata alongside them.