[CONFIGURATION] Registry has builders for all components and link only what you use

[dbarker]

Background:

The configuration_core library was created to separate the programmatic configuration interface from the YAML parsing components in completion of #4134.

It links to all SDK signal libraries (traces, logs, and metrics). Therefore, if a user of programmatic or yaml configuration desires to only configure one signal provider they must still link all SDK libraries.

This is driven by two things: SdkBuilder, which constructs all signal providers via direct Factory::Create calls in one TU, and ConfiguredSdk, whose header stores all three SDK provider types. This also limits what SDK component builders a user can register for configuration.

Goals:

This ticket has two goals:

1. Allow linking just to the SDK components being configured and
2. Support registering builders for every configurable SDK component, including the signal providers themselves.

Together these would allow linking and registering only the builders that an application supports, providing more control over configurable options, ability to customize the SDK components, and reduce binary size.

Some use cases enabled by this include:

• Registering a custom builder to replace a default SDK component builder
  • A user may define their own implementation of aTracerConfigurator, BatchSpanProcessor, and TracerProvider for example.
• Registering mock component builders for test scenarios
• Intentionally not registering a component's builder to prevent its use in the application

Example:

If the config components are decoupled from the SDK signal providers, the example below should only require the application to link the configuration library, the trace default builders, the gRPC span exporter, and users custom batch processor. No metrics or logs libraries would be linked and attempts to configure them would result in an error.

namespace cfg = opentelemetry::sdk::configuration;

auto otlp_grpc_exporter      = std::make_unique<cfg::OtlpGrpcSpanExporterConfiguration>();
otlp_grpc_exporter->endpoint = "http://localhost:4317";

auto processor               = std::make_unique<cfg::BatchSpanProcessorConfiguration>();
processor->exporter          = std::move(otlp_grpc_exporter);

auto tp = std::make_unique<cfg::TracerProviderConfiguration>();
tp->processors.push_back(std::move(processor));

auto model             = std::make_unique<cfg::Configuration>();
model->tracer_provider = std::move(tp);

// New Registry behavior:
//   Constructing a registry creates an empty registry that brings no
//   SDK dependencies outside of `configuration_core`. A config node 
//   with no registered builder is a error
auto registry = std::make_shared<cfg::Registry>();

// New SDK signal specific method to register all its component builders:
//  Here the trace method registers its builders (provider, processors, samplers, configurator, ...)
//  This is the one call that links opentelemetry_trace into the app.
cfg::RegisterTraceDefaults(registry.get());
opentelemetry::exporter::otlp::OtlpGrpcSpanBuilder::Register(registry.get());

// New registry slots for all configurable SDK components: 
//   Users may register their own builders for the standard SDK components
//   Here their batch processor implementation takes the standard registry slot
my_library::MyBatchSpanProcessorBuilder::Register(registry.get());

// New SdkBuilder behavior:
//   SdkBuilder brings in no signal SDK dependencies (traces/metrics/logs)
//   it only reads the model, then looks up and calls builders from the
//   registry. 
auto sdk = cfg::ConfiguredSdk::Create(registry, model);

// New ConfiguredSdk behavior:
//   ConfiguredSdk gains API-typed provider handles (Install
//   uses only these) and keeps its SDK-typed members declared against
//   forward declarations (set if the default SDK provider was built and null for a custom provider)
// 
sdk->Install();

// Full SDK provider API still available in the default case
sdk->tracer_provider->ForceFlush(std::chrono::milliseconds(100));

Behavioral change:

Constructing a Registry today has some registered default builders and configuration relies on the SdkBuilder to directly create standard SDK components.

The new behavior would be to construct an empty Registry by default and require explicit registration of the builders needed.

Existing users (including all yaml users) may keep the current behavior (Registry is populated with default SDK component builders) through a new RegistryFactory library and Create method:

// Current approach to instantiate a Registry 
//     The registry holds mainly exporters, extensions, and propagator builders
//     The SdkBuilder has the SDK component creators built-in
auto registry = std::make_shared<cfg::Registry>();   

// New approach to instantiate a Registry with all SDK component builders
//     The new RegistryFactory library links all signals and registers their default component builders
auto registry = cfg::RegistryFactory::Create();     

[dbarker]

This is the rough plan for changes needed: (will edit with PR links and update as things change)

1. Add builder interfaces:
   [CONFIGURATION] Add SDK component builder interfaces to the registry #4358
   • New builder abstract classes for all SDK components in the schema not currently in the registry
   • New registry accessors for all configurable components.
2. Create builder libraries per signal and update SdkBuilder to use them:
   [CONFIGURATION] SDK default component builder libraries and example #4367
   • Create default "leaf" component builders in three new targets (configuration_<trace,metrics,logs>_builders)
   • Add per-signal functions to register the new default builders.
   • Update SdkBuilder to use the registered builders for leaf nodes replacing direct Factory::Create calls.
3. Create SDK provider builders and update SdkBuilder to use them:
   • Create the per-signal SDK provider builders by moving the functionality out of SDK Builder
   • SdkBuilder gets the provider builders from the registry
4. Update ConfiguredSdk to own the providers through API-typed handles
   • New test to cover a custom provider builder (ie: MockTracerProvider derived from the API TracerProvider).
5. Drop the signal library links from configuration_core
   • Update CMake/Bazel to remove the links on opentelemetry_<trace,metrics,logs>
   • Update tests and examples to link what they use

[dbarker]

This is what I'm thinking for the configuration libraries end state (CMake targets). The current opentelemetry-cpp::configuration target becomes an interface to bring in the configuration "kitchen sink" (core, all default builders, yaml) and is created when WITH_CONFIGURATION=ON. The yaml components then go to configuration_yaml.

configuration                            (family aggregate, INTERFACE, WITH_CONFIGURATION)
├── configuration_core                   (model + builder interfaces + Registry + SdkBuilder + ConfiguredSdk)
│   ├── PUBLIC  opentelemetry_api
│   ├── PUBLIC  opentelemetry_resources
│   └── PRIVATE opentelemetry_common
├── configuration_yaml                   (yaml parser)
│   ├── PUBLIC  configuration_core
│   └── PRIVATE ryml::ryml
├── configuration_trace_builders         (default trace builders + RegisterTraceDefaults)
│   ├── PUBLIC  opentelemetry_trace      
│   └── PRIVATE configuration_core
├── configuration_metrics_builders       (default metrics builders + RegisterMetricsDefaults)
│   ├── PUBLIC  opentelemetry_metrics    
│   └── PRIVATE configuration_core
├── configuration_logs_builders          (default logs builders + RegisterLogsDefaults)
│   ├── PUBLIC  opentelemetry_logs      
│   └── PRIVATE configuration_core
└── configuration_default_registry       (RegistryFactory makes Registry with all default builders - use when needed)
    ├── PUBLIC  configuration_core
    ├── PUBLIC  configuration_trace_builders
    ├── PUBLIC  configuration_metrics_builders
    └── PUBLIC  configuration_logs_builders

A user that only wants to configure just the tracer provider programmatically just links configuration_core and configuration_trace_builders.

A user that wants full yaml support and all signal providers (with all default builders) links opentelemetry-cpp::configuration (similar to today).