Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions build/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,25 @@ together. The snapshot tests in
`scripts/pod-scheduler/tests/test_snapshots.py` will fail in CI if the
generated YAML drifts from the configs.

## Build reuse within a pipeline run

The container, gRPC, HttpClient, SignalR, and WebSocket templates pass Crank's
`reuseBuild`, `buildKey`, and `source.sourceKey` options directly. Keys include
`$(Build.BuildId)`: each agent builds once per compatible variant in a pipeline
run, then refreshes sources and builds on the next run. Source and build keys use
separate namespaces because their metadata shares a cache directory.
`--session` alone does not invalidate caches.

Keys distinguish Docker images, gRPC implementations and frameworks, and
WebSocket compression builds. Container scenarios with a database also include
`$(containerDatabaseBuildReuse)`. Docker image names get the build ID appended in
`onConfigure` hooks so interleaved runs cannot overwrite each other's image tags;
the gRPC hooks leave non-Docker jobs unchanged.

Only builds are reused. Applications and database containers still start fresh,
and warmup/measurement durations are unchanged. Caches remain local to each
agent and rely on its existing cleanup/restart policy; no cache TTL is added.

## Minimal Todo

`minimal-todo-scenarios.yml` dispatches `scenarios/minimal-todo/minimal-todo.benchmarks.yml`
Expand Down
24 changes: 12 additions & 12 deletions build/containers-scenarios.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,52 +62,52 @@ parameters:
arguments: --scenario json_aspnet --property scenario=JsonAspNet
condition: Math.round(Date.now() / 43200000) % 8 == 0 # once every 8 half-days
- displayName: Fortunes ASP.NET
arguments: --scenario fortunes_aspnet --property scenario=FortunesAspNet --application.environmentVariables DOTNET_gcServer=1 --application.environmentVariables DOTNET_GCDynamicAdaptationMode=0
arguments: --scenario fortunes_aspnet --property scenario=FortunesAspNet --application.environmentVariables DOTNET_gcServer=1 --application.environmentVariables DOTNET_GCDynamicAdaptationMode=0 $(containerDatabaseBuildReuse)
condition: Math.round(Date.now() / 43200000) % 8 == 0 # once every 8 half-days
- displayName: Fortunes ASP.NET - DATAS
arguments: --scenario fortunes_aspnet --property scenario=FortunesAspNetDATAS --application.environmentVariables DOTNET_GCDynamicAdaptationMode=1
arguments: --scenario fortunes_aspnet --property scenario=FortunesAspNetDATAS --application.environmentVariables DOTNET_GCDynamicAdaptationMode=1 $(containerDatabaseBuildReuse)
condition: Math.round(Date.now() / 43200000) % 8 == 0 # once every 8 half-days

- displayName: Json ASP.NET Alpine
arguments: --scenario json_aspnet_alpine --property scenario=JsonAspNetAlpine
condition: Math.round(Date.now() / 43200000) % 8 == 2 # once every 8 half-days
- displayName: Fortunes ASP.NET Alpine
arguments: --scenario fortunes_aspnet_alpine --property scenario=FortunesAspNetAlpine
arguments: --scenario fortunes_aspnet_alpine --property scenario=FortunesAspNetAlpine $(containerDatabaseBuildReuse)
condition: Math.round(Date.now() / 43200000) % 8 == 2 # once every 8 half-days

- displayName: Json ASP.NET Composite
arguments: --scenario json_aspnet_composite --property scenario=JsonAspNetComposite
condition: Math.round(Date.now() / 43200000) % 8 == 4 # once every 8 half-days
- displayName: Fortunes ASP.NET Composite
arguments: --scenario fortunes_aspnet_composite --property scenario=FortunesAspNetComposite
arguments: --scenario fortunes_aspnet_composite --property scenario=FortunesAspNetComposite $(containerDatabaseBuildReuse)
condition: Math.round(Date.now() / 43200000) % 8 == 4 # once every 8 half-days

- displayName: Json ASP.NET 8.0
arguments: --scenario json_aspnet_current --property scenario=JsonAspNet80
condition: Math.round(Date.now() / 43200000) % 8 == 6 # once every 8 half-days
- displayName: Fortunes ASP.NET 8.0
arguments: --scenario fortunes_aspnet_current --property scenario=FortunesAspNet80
arguments: --scenario fortunes_aspnet_current --property scenario=FortunesAspNet80 $(containerDatabaseBuildReuse)
condition: Math.round(Date.now() / 43200000) % 8 == 6 # once every 8 half-days

- displayName: Json NodeJS
arguments: --scenario json_nodejs --property scenario=JsonNodeJs
arguments: --scenario json_nodejs --property scenario=JsonNodeJs $(containerDatabaseBuildReuse)
condition: Math.round(Date.now() / 43200000) % 10 == 2 # once every 10 half-days
- displayName: Fortunes NodeJS
arguments: --scenario fortunes_nodejs --property scenario=FortunesNodeJs
arguments: --scenario fortunes_nodejs --property scenario=FortunesNodeJs $(containerDatabaseBuildReuse)
condition: Math.round(Date.now() / 43200000) % 10 == 3 # once every 10 half-days

- displayName: Json FastHttp
arguments: --scenario json_fasthttp --property scenario=JsonFastHttp
arguments: --scenario json_fasthttp --property scenario=JsonFastHttp $(containerDatabaseBuildReuse)
condition: Math.round(Date.now() / 43200000) % 10 == 4 # once every 10 half-days
- displayName: Fortunes FastHttp
arguments: --scenario fortunes_fasthttp --property scenario=FortunesFastHttp
arguments: --scenario fortunes_fasthttp --property scenario=FortunesFastHttp $(containerDatabaseBuildReuse)
condition: Math.round(Date.now() / 43200000) % 10 == 5 # once every 10 half-days

- displayName: Json Vertx
arguments: --scenario json_vertx --property scenario=JsonVertx
arguments: --scenario json_vertx --property scenario=JsonVertx $(containerDatabaseBuildReuse)
condition: Math.round(Date.now() / 43200000) % 10 == 8 # once every 10 half-days
- displayName: Fortunes Vertx
arguments: --scenario fortunes_vertx --property scenario=FortunesVertx
arguments: --scenario fortunes_vertx --property scenario=FortunesVertx $(containerDatabaseBuildReuse)
condition: Math.round(Date.now() / 43200000) % 10 == 9 # once every 10 half-days

# Json NTex and Fortunes NTex are disabled: their docker builds
Expand Down Expand Up @@ -145,6 +145,6 @@ steps:
{
"name": "crank",
"condition": "(${{ parameters.condition }}) && (${{ scenario.condition }})",
"args": [ "--session $(session) $(containerMatrixJobs) ${{ scenario.arguments }} ${{ size.arguments }} ${{ rate.arguments }} ${{ parameters.arguments }} $(azureProfile) --no-metadata --no-measurements --load.options.reuseBuild true --command-line-property --table ContainerBenchmarks --sql SQL_CONNECTION_STRING --cert-tenant-id SQL_SERVER_TENANTID --cert-client-id SQL_SERVER_CLIENTID --cert-path SQL_SERVER_CERT_PATH --cert-sni --chart" ]
"args": [ "--session $(session) $(containerMatrixJobs) ${{ scenario.arguments }} ${{ size.arguments }} ${{ rate.arguments }} ${{ parameters.arguments }} $(azureProfile) --application.options.reuseBuild true --application.buildKey containers-build-{{job.Source.DockerImageName}}-$(Build.BuildId) --application.source.sourceKey containers-source-{{job.Source.DockerImageName}}-$(Build.BuildId) --application.onConfigure \"job.DockerImageName += '-$(Build.BuildId)';\" --no-metadata --no-measurements --load.options.reuseBuild true --command-line-property --table ContainerBenchmarks --sql SQL_CONNECTION_STRING --cert-tenant-id SQL_SERVER_TENANTID --cert-client-id SQL_SERVER_CLIENTID --cert-path SQL_SERVER_CERT_PATH --cert-sni --chart" ]
}

20 changes: 17 additions & 3 deletions build/grpc-scenarios.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,21 @@ parameters:

- displayName: "ASP.NET Core - Client: Grpc.Net.Client"
arguments: --scenario grpcaspnetcoreserver-grpcnetclient $(grpcJobs) --property server=grpcaspnetcoreserver --property client=grpcnetclient --application.options.collectCounters true
serverBuildKey: aspnetcore
clientBuildKey: dotnet
- displayName: "ASP.NET Core - Client: Go"
arguments: --scenario grpcaspnetcoreserver-grpcgoclient $(grpcJobs) --property server=grpcaspnetcoreserver --property client=grpcgoclient --application.options.collectCounters true
serverBuildKey: aspnetcore
clientBuildKey: go

- displayName: "Go - Client: Grpc.Net.Client"
arguments: --scenario grpcgoserver-grpcnetclient $(grpcJobs) --property server=grpcgoserver --property client=grpcnetclient
serverBuildKey: go
clientBuildKey: dotnet
- displayName: "Go - Client: Go"
arguments: --scenario grpcgoserver-grpcgoclient $(grpcJobs) --property server=grpcgoserver --property client=grpcgoclient
serverBuildKey: go
clientBuildKey: go

# H2Load scenarios
- name: h2loadscenarios
Expand All @@ -42,8 +50,12 @@ parameters:

- displayName: "ASP.NET Core - Client: h2load"
arguments: --scenario grpcaspnetcoreserver-h2loadclient $(grpcJobs) --property server=grpcaspnetcoreserver --property client=h2loadclient --application.options.collectCounters true
serverBuildKey: aspnetcore
clientBuildKey: h2load
- displayName: "Go - Client: h2load"
arguments: --scenario grpcgoserver-h2loadclient $(grpcJobs) --property server=grpcgoserver --property client=h2loadclient
serverBuildKey: go
clientBuildKey: h2load

# grpc-net only scenarios
- name: grpcnetscenarios
Expand All @@ -52,6 +64,8 @@ parameters:

- displayName: "ASP.NET Core - Client: Grpc.Net.Client"
arguments: --scenario grpcaspnetcoreserver-grpcnetclient $(grpcJobs) --property server=grpcaspnetcoreserver --property client=grpcnetclient --application.options.collectCounters true
serverBuildKey: aspnetcore
clientBuildKey: dotnet

- name: callTypes
type: object
Expand Down Expand Up @@ -107,7 +121,7 @@ steps:
{
"condition": "(${{ parameters.condition }}) && ((new Date().getUTCHours() - 7 + 24) % 24 >= 13)",
"name": "crank",
"args": [ "--application.framework net11.0 --command-line-property --table GrpcBenchmarks --sql SQL_CONNECTION_STRING --cert-tenant-id SQL_SERVER_TENANTID --cert-client-id SQL_SERVER_CLIENTID --cert-path SQL_SERVER_CERT_PATH --cert-sni --chart --session $(session) --no-metadata --no-measurements ${{ parameters.arguments }} ${{ s.arguments }} ${{ stream.arguments }} --variable protocol=h2c --property protocol=h2c --property scenario=unary --property payload=0 --variable body=AAAAAAcKBVdvcmxk --variable path=/grpc.testing.BenchmarkService/UnaryCall" ]
"args": [ "--application.framework net11.0 --command-line-property --table GrpcBenchmarks --sql SQL_CONNECTION_STRING --cert-tenant-id SQL_SERVER_TENANTID --cert-client-id SQL_SERVER_CLIENTID --cert-path SQL_SERVER_CERT_PATH --cert-sni --chart --session $(session) --no-metadata --no-measurements ${{ parameters.arguments }} ${{ s.arguments }} ${{ stream.arguments }} --application.options.reuseBuild true --application.buildKey grpc-server-build-${{ s.serverBuildKey }}-{{job.Framework}}-$(Build.BuildId) --application.source.sourceKey grpc-server-source-${{ s.serverBuildKey }}-$(Build.BuildId) --application.onConfigure \"if (job.DockerImageName) job.DockerImageName += '-$(Build.BuildId)';\" --load.options.reuseBuild true --load.buildKey grpc-client-build-${{ s.clientBuildKey }}-{{job.Framework}}-$(Build.BuildId) --load.source.sourceKey grpc-client-source-${{ s.clientBuildKey }}-$(Build.BuildId) --load.onConfigure \"if (job.DockerImageName) job.DockerImageName += '-$(Build.BuildId)';\" --variable protocol=h2c --property protocol=h2c --property scenario=unary --property payload=0 --variable body=AAAAAAcKBVdvcmxk --variable path=/grpc.testing.BenchmarkService/UnaryCall" ]
}

# h2/h3 HttpClient to ASP.NET Core
Expand All @@ -132,7 +146,7 @@ steps:
{
"condition": "(${{ parameters.condition }}) && ((new Date().getUTCHours() - 7 + 24) % 24 >= 13)",
"name": "crank",
"args": [ "--application.framework net11.0 --load.framework net11.0 --command-line-property --table GrpcBenchmarks --sql SQL_CONNECTION_STRING --cert-tenant-id SQL_SERVER_TENANTID --cert-client-id SQL_SERVER_CLIENTID --cert-path SQL_SERVER_CERT_PATH --cert-sni --chart --session $(session) --no-metadata --no-measurements ${{ parameters.arguments }} ${{ s.arguments }} ${{ callType.arguments }} ${{ payload.arguments }} ${{ stream.arguments }} ${{ protocol.arguments }}" ]
"args": [ "--application.framework net11.0 --load.framework net11.0 --command-line-property --table GrpcBenchmarks --sql SQL_CONNECTION_STRING --cert-tenant-id SQL_SERVER_TENANTID --cert-client-id SQL_SERVER_CLIENTID --cert-path SQL_SERVER_CERT_PATH --cert-sni --chart --session $(session) --no-metadata --no-measurements ${{ parameters.arguments }} ${{ s.arguments }} ${{ callType.arguments }} ${{ payload.arguments }} ${{ stream.arguments }} ${{ protocol.arguments }} --application.options.reuseBuild true --application.buildKey grpc-server-build-${{ s.serverBuildKey }}-{{job.Framework}}-$(Build.BuildId) --application.source.sourceKey grpc-server-source-${{ s.serverBuildKey }}-$(Build.BuildId) --application.onConfigure \"if (job.DockerImageName) job.DockerImageName += '-$(Build.BuildId)';\" --load.options.reuseBuild true --load.buildKey grpc-client-build-${{ s.clientBuildKey }}-{{job.Framework}}-$(Build.BuildId) --load.source.sourceKey grpc-client-source-${{ s.clientBuildKey }}-$(Build.BuildId) --load.onConfigure \"if (job.DockerImageName) job.DockerImageName += '-$(Build.BuildId)';\"" ]
}

# h2c between various clients and servers
Expand All @@ -154,5 +168,5 @@ steps:
{
"condition": "(${{ parameters.condition }}) && ((new Date().getUTCHours() - 7 + 24) % 24 >= 13)",
"name": "crank",
"args": [ "--application.framework net11.0 --load.framework net11.0 --command-line-property --table GrpcBenchmarks --sql SQL_CONNECTION_STRING --cert-tenant-id SQL_SERVER_TENANTID --cert-client-id SQL_SERVER_CLIENTID --cert-path SQL_SERVER_CERT_PATH --cert-sni --chart --session $(session) ${{ parameters.arguments }} --no-metadata --no-measurements ${{ s.arguments }} ${{ callType.arguments }} ${{ payload.arguments }} ${{ stream.arguments }} --variable protocol=h2c --property protocol=h2c" ]
"args": [ "--application.framework net11.0 --load.framework net11.0 --command-line-property --table GrpcBenchmarks --sql SQL_CONNECTION_STRING --cert-tenant-id SQL_SERVER_TENANTID --cert-client-id SQL_SERVER_CLIENTID --cert-path SQL_SERVER_CERT_PATH --cert-sni --chart --session $(session) ${{ parameters.arguments }} --no-metadata --no-measurements ${{ s.arguments }} ${{ callType.arguments }} ${{ payload.arguments }} ${{ stream.arguments }} --application.options.reuseBuild true --application.buildKey grpc-server-build-${{ s.serverBuildKey }}-{{job.Framework}}-$(Build.BuildId) --application.source.sourceKey grpc-server-source-${{ s.serverBuildKey }}-$(Build.BuildId) --application.onConfigure \"if (job.DockerImageName) job.DockerImageName += '-$(Build.BuildId)';\" --load.options.reuseBuild true --load.buildKey grpc-client-build-${{ s.clientBuildKey }}-{{job.Framework}}-$(Build.BuildId) --load.source.sourceKey grpc-client-source-${{ s.clientBuildKey }}-$(Build.BuildId) --load.onConfigure \"if (job.DockerImageName) job.DockerImageName += '-$(Build.BuildId)';\" --variable protocol=h2c --property protocol=h2c" ]
}
4 changes: 2 additions & 2 deletions build/httpclient-scenarios.yml
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ steps:
{
"condition": "(${{ parameters.condition }}) && (Math.round(Date.now() / 43200000) % 2 == 0)",
"name": "crank",
"args": [ "--description \"${{ version.displayName }} ${{ s.displayName }} ${{ response.displayName }} ${{ ct.displayName }} $(System.JobDisplayName)\" ${{ parameters.tfm }} --command-line-property --table HttpClientBenchmarks --sql SQL_CONNECTION_STRING --cert-tenant-id SQL_SERVER_TENANTID --cert-client-id SQL_SERVER_CLIENTID --cert-path SQL_SERVER_CERT_PATH --cert-sni --chart --session $(session) ${{ parameters.arguments }} ${{ parameters.httpClientJobs }} --no-metadata --no-measurements ${{ s.arguments }} ${{ version.arguments }} ${{ ct.arguments }} ${{ response.arguments }}" ]
"args": [ "--description \"${{ version.displayName }} ${{ s.displayName }} ${{ response.displayName }} ${{ ct.displayName }} $(System.JobDisplayName)\" ${{ parameters.tfm }} --command-line-property --table HttpClientBenchmarks --sql SQL_CONNECTION_STRING --cert-tenant-id SQL_SERVER_TENANTID --cert-client-id SQL_SERVER_CLIENTID --cert-path SQL_SERVER_CERT_PATH --cert-sni --chart --session $(session) ${{ parameters.arguments }} ${{ parameters.httpClientJobs }} --no-metadata --no-measurements ${{ s.arguments }} ${{ version.arguments }} ${{ ct.arguments }} ${{ response.arguments }} --server.options.reuseBuild true --server.buildKey httpclient-server-build-{{job.Framework}}-$(Build.BuildId) --server.source.sourceKey httpclient-source-$(Build.BuildId) --client.options.reuseBuild true --client.buildKey httpclient-client-build-{{job.Framework}}-$(Build.BuildId) --client.source.sourceKey httpclient-source-$(Build.BuildId)" ]
}

# POST scenarios
Expand All @@ -116,5 +116,5 @@ steps:
{
"condition": "(${{ parameters.condition }}) && (Math.round(Date.now() / 43200000) % 2 == 0)",
"name": "crank",
"args": [ "--description \"${{ version.displayName }} ${{ s.displayName }} ${{ content.displayName }} ${{ ct.displayName }} $(System.JobDisplayName)\" ${{ parameters.tfm }} --command-line-property --table HttpClientBenchmarks --sql SQL_CONNECTION_STRING --cert-tenant-id SQL_SERVER_TENANTID --cert-client-id SQL_SERVER_CLIENTID --cert-path SQL_SERVER_CERT_PATH --cert-sni --chart --session $(session) ${{ parameters.arguments }} ${{ parameters.httpClientJobs }} --no-metadata --no-measurements ${{ s.arguments }} ${{ version.arguments }} ${{ ct.arguments }} ${{ content.arguments }}" ]
"args": [ "--description \"${{ version.displayName }} ${{ s.displayName }} ${{ content.displayName }} ${{ ct.displayName }} $(System.JobDisplayName)\" ${{ parameters.tfm }} --command-line-property --table HttpClientBenchmarks --sql SQL_CONNECTION_STRING --cert-tenant-id SQL_SERVER_TENANTID --cert-client-id SQL_SERVER_CLIENTID --cert-path SQL_SERVER_CERT_PATH --cert-sni --chart --session $(session) ${{ parameters.arguments }} ${{ parameters.httpClientJobs }} --no-metadata --no-measurements ${{ s.arguments }} ${{ version.arguments }} ${{ ct.arguments }} ${{ content.arguments }} --server.options.reuseBuild true --server.buildKey httpclient-server-build-{{job.Framework}}-$(Build.BuildId) --server.source.sourceKey httpclient-source-$(Build.BuildId) --client.options.reuseBuild true --client.buildKey httpclient-client-build-{{job.Framework}}-$(Build.BuildId) --client.source.sourceKey httpclient-source-$(Build.BuildId)" ]
}
6 changes: 6 additions & 0 deletions build/job-variables.yml
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,12 @@ variables:
value: --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/azure.profile.yml
- name: ciProfile
value: --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml
- name: containerDatabaseBuildReuse
value: >-
--db.options.reuseBuild true
--db.buildKey containers-build-{{job.Source.DockerImageName}}-$(Build.BuildId)
--db.source.sourceKey containers-source-{{job.Source.DockerImageName}}-$(Build.BuildId)
--db.onConfigure \"job.DockerImageName += '-$(Build.BuildId)';\"
- name: grpcJobs
value: --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/grpc.benchmarks.yml
- name: efcoreJobs
Expand Down
Loading
Loading