feat: explicit instrumentation opt-in with HARNESS_ env prefix

[shreyas70]

Summary

Makes SDK instrumentation explicit opt-in (was opt-out) and migrates the env prefix to HARNESS_ with backwards compatibility.

• API/HTTP — one toggle HARNESS_ENABLE_API enables all non-AI instrumentation (HTTP servers/clients, gRPC, MySQL/PostgreSQL, botocore, generic OTel contrib fallback).
• AI — per-provider flags: HARNESS_ENABLE_AI_OPENAI, HARNESS_ENABLE_AI_ANTHROPIC, HARNESS_ENABLE_AI_LITELLM, HARNESS_ENABLE_AI_GOOGLE_GENAI, HARNESS_ENABLE_AI_MCP. Each enabled independently.
• Default — nothing is instrumented until a flag is set to true.
• skip_libraries still works and takes precedence over enable flags.

Env prefix migration (backwards compatible)

• SDK settings now resolve HARNESS_ > HA_ > AT_ > TA_. Existing HA_/AT_/TA_ configs keep working.
• The HARNESS_ENABLE_* opt-in flags are strict HARNESS_-only (no legacy aliases).
• Legacy HA_GEN_AI_ENABLED master switch no longer controls instrumentation; use per-provider flags. Its removed per-call/instrument gates were pulled from the openai/anthropic/litellm/google_genai/mcp wrappers.

Key changes

• env.py: get_env_value precedence + is_env_var_present + is_harness_flag_enabled.
• instrumentation_definitions.py: API_LIBRARIES, AI_LIBRARY_ENV_FLAGS, is_library_enabled, is_api_instrumentation_enabled, any_ai_provider_enabled.
• agent.py: instrument() filters libraries by opt-in; generic contrib gated behind HARNESS_ENABLE_API.
• Routed config/environment.py, config.py, litellm raw-capture, and console-exporter checks through the shared resolver.
• Docs: README, sdk-quickstart, AGENTS.md/CLAUDE.md updated with flag matrix, precedence, migration.

Test plan

• New test/instrumentation/test_opt_in_gating.py: default-off, API scope, per-provider independence, strict-true, no-legacy-alias, skip_libraries precedence, Agent integration.
• environment_test.py: prefix precedence + legacy aliases.
• Converted obsolete gen_ai_disabled_passthrough tests to legacy_gen_ai_master_ignored; removed dead MCP disable tests; conftest/botocore updated for opt-in.
• Local run green across config, gating, anthropic/litellm/google_genai/mcp, botocore, fastapi, autoinstrumentation, plugins.
• CI: flask/grpc/aiohttp server suites + MySQL/PostgreSQL Docker integration (hang/skip locally; verify in CI).

Note: pre-existing local failures unrelated to this change (openai client version mismatch, requests test_request_client, server-suite hangs) also fail on main.

Made with Cursor

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}