diff --git a/docs/admin-token-network-policy.md b/docs/admin-token-network-policy.md new file mode 100644 index 00000000000..ddbdaee959b --- /dev/null +++ b/docs/admin-token-network-policy.md @@ -0,0 +1,78 @@ +# Administrative token network policy + +Cloud Controller can restrict validated administrative bearer tokens to approved +source networks. The policy runs before downstream middleware, user creation, +and API authorization, including uploads using the relaxed token-expiry decoder. +It applies to old and new tokens and all API paths. Ordinary tokens and +unauthenticated requests retain their existing behavior. + +```yaml +admin_token_network_policy: + enabled: true + allowed_cidrs: [192.0.2.10/32, '2001:db8:1234::/48'] + trusted_proxies: [10.0.0.10/32] + additional_protected_scopes: [clients.write] +``` + +These are documentation examples, not deployment values. All four fields are +operator-controlled. The policy is disabled by default; allowed networks, trusted +proxies, and additional protected scopes default to empty lists. Enabled with an +empty allowlist means deny-all for protected tokens. Invalid numeric addresses or +CIDRs fail startup, even when disabled. Restart to apply configuration changes. + +Protected scopes/authorities are `cloud_controller.admin`, `cloud_controller.admin_read_only`, +`cloud_controller.global_auditor`, `uaa.admin`, and `zones..admin` (any nonempty +zone ID). Additional scope names extend this set by exact match. Checking both +`scope` and `authorities` prevents a narrowed client-credentials scope from +bypassing protection. The list is not an exhaustive defense against escalation +through other delegation privileges; review and add those explicitly. + +Allowlist/trusted-proxy entries are numeric IPv4/IPv6 hosts or CIDRs with decimal +prefix lengths. Bare addresses match a single host; CIDRs mask host bits. +Hostnames, abbreviated IPv4, dotted netmasks, ports, brackets, whitespace, +interface IDs, and mapped-IPv6 CIDR notation are invalid. IPv4-mapped IPv6 +requests match IPv4 networks. No DNS resolution is performed. + +## Source-address contract + +- TCP requests use the actual Puma socket peer (or `REMOTE_ADDR` on other Rack + servers). Real-IP and forwarded headers from untrusted peers are ignored, + including on loopback TCP. +- Nginx may assert `X-Real-IP` only when the actual Puma socket is Unix-domain and + its local path matches `nginx.instance_socket` with `nginx.use_nginx` enabled. + The BOSH Nginx templates overwrite `X-Real-IP` with `$remote_addr` on public and + internal proxy paths, including upload forwarding, and strip `X-Real_IP` to + prevent a Rack-normalization collision. Deploy the matching Nginx template + changes before enabling the policy. The socket and its parents must be protected + against untrusted local access. Loopback does not prove this transport. +- Only explicitly listed `trusted_proxies` can forward source addresses. Walk + `X-Forwarded-For` from right to left until the first untrusted numeric address, + which is the source. Nginx's `$proxy_add_x_forwarded_for` appends its immediate + peer. Ignore attacker-controlled prefixes beyond the first untrusted hop; + malformed encountered hops, missing addresses, more than 64 entries, and + chains with no untrusted source fail closed. + +List actual router/proxy addresses, **not client networks**, in `trusted_proxies`. +Never broadly trust all private addresses just because your Gorouters have private +IPs. Every ingress hop must sanitize or append forwarded headers correctly, and +direct backend access must be protected. Other Rack servers must ensure +`REMOTE_ADDR` remains an authoritative peer address. + +## Denial and coordinated rollout + +Denials return HTTP 403 using the existing `CF-NotAuthorized` v2/v3 error format, +clear the validated security context, and do not invoke downstream user creation +or API handling. The fixed denial log message includes no credentials or tokens. + +Use matching allowed networks and protected additions in UAA's +`uaa.admin_token_network_policy` to restrict issuance, refresh, and UAA usage. +In capi-release, the properties have the `cc.admin_token_network_policy` prefix +with `enabled`, `allowed_cidrs`, `trusted_proxies`, and +`additional_protected_scopes` suffixes. Release wiring requires application +versions that implement these settings; old binaries do not gain protection +by rendering configuration. Other offline JWT consumers need their own policy. + +Before enabling, inventory administrator VPN/NAT egress and internal automation, +test public/internal ingress and forged headers, and verify issuance on an allowed +network followed by use/refresh on a denied network. Keep a tested operator +configuration rollback procedure. diff --git a/lib/cloud_controller/config_schemas/api_schema.rb b/lib/cloud_controller/config_schemas/api_schema.rb index acb517879da..a5b170ce85e 100644 --- a/lib/cloud_controller/config_schemas/api_schema.rb +++ b/lib/cloud_controller/config_schemas/api_schema.rb @@ -8,6 +8,12 @@ class ApiSchema < VCAP::Config define_schema do { external_port: Integer, + optional(:admin_token_network_policy) => { + enabled: bool, + allowed_cidrs: [String], + optional(:trusted_proxies) => [String], + optional(:additional_protected_scopes) => [String] + }, external_domain: String, temporary_disable_deployments: bool, optional(:temporary_disable_v2_staging) => bool, diff --git a/lib/cloud_controller/rack_app_builder.rb b/lib/cloud_controller/rack_app_builder.rb index cb8b4f5176c..2566f660f6d 100644 --- a/lib/cloud_controller/rack_app_builder.rb +++ b/lib/cloud_controller/rack_app_builder.rb @@ -13,12 +13,14 @@ require 'block_v3_only_roles' require 'below_min_cli_warning' require 'user_context_setter' +require 'cloud_controller/security/admin_token_network_policy' module VCAP::CloudController class RackAppBuilder def build(config, request_metrics, request_logs) token_decoder = VCAP::CloudController::UaaTokenDecoder.new(config.get(:uaa)) configurer = VCAP::CloudController::Security::SecurityContextConfigurer.new(token_decoder) + admin_token_network_policy = admin_token_network_policy(config) concurrency_rate_limiter_config = concurrency_rate_limiter_config(config) rate_limiter_config = rate_limiter_config(config) @@ -31,7 +33,7 @@ def build(config, request_metrics, request_logs) use CloudFoundry::Middleware::VcapRequestContextSetter use CloudFoundry::Middleware::BelowMinCliWarning if config.get(:warn_if_below_min_cli_version) use CloudFoundry::Middleware::NewRelicCustomAttributes if config.get(:newrelic_enabled) - use CloudFoundry::Middleware::SecurityContextSetter, configurer + use CloudFoundry::Middleware::SecurityContextSetter, configurer, admin_token_network_policy use CloudFoundry::Middleware::Zipkin use CloudFoundry::Middleware::ConcurrencyRateLimiter, concurrency_rate_limiter_config if config.get(:concurrency_rate_limiter, :enabled) @@ -63,6 +65,13 @@ def build(config, request_metrics, request_logs) private + def admin_token_network_policy(config) + Security::AdminTokenNetworkPolicy.new( + config.get(:admin_token_network_policy) || {}, + nginx_socket: config.get(:nginx, :use_nginx) ? config.get(:nginx, :instance_socket) : nil + ) + end + def access_log(config) return unless !config.get(:nginx, :use_nginx) && config.get(:logging, :file) diff --git a/lib/cloud_controller/security/admin_token_network_policy.rb b/lib/cloud_controller/security/admin_token_network_policy.rb new file mode 100644 index 00000000000..d7a39284210 --- /dev/null +++ b/lib/cloud_controller/security/admin_token_network_policy.rb @@ -0,0 +1,93 @@ +require 'ipaddr' +require 'socket' + +module VCAP + module CloudController + module Security + class AdminTokenNetworkPolicy + ADMIN_SCOPES = %w[uaa.admin cloud_controller.admin cloud_controller.admin_read_only cloud_controller.global_auditor].freeze + + def initialize(options={}, nginx_socket: nil) + @enabled = options.fetch(:enabled, false) + @networks = options.fetch(:allowed_cidrs, []).map { |cidr| network(cidr) } + @trusted_proxies = options.fetch(:trusted_proxies, []).map { |cidr| network(cidr) } + @additional_scopes = options.fetch(:additional_protected_scopes, []) + @nginx_socket = nginx_socket + end + + def allowed?(token, env) + return true unless @enabled && protected_token?(token) + + address = source_address(env) + address && @networks.any? { |cidr| cidr.include?(address) } + end + + private + + def protected_token?(token) + %w[scope authorities].any? do |claim| + values = token[claim].is_a?(String) ? token[claim].split(/[ ,]+/) : Array(token[claim]) + values.any? { |scope| ADMIN_SCOPES.include?(scope) || @additional_scopes.include?(scope) || (scope.is_a?(String) && scope.match?(/\Azones\..+\.admin\z/)) } + end + end + + def source_address(env) + peer = peer_address(env) + return nil unless peer + return peer unless trusted?(peer) + + forwarded_header = env.fetch('HTTP_X_FORWARDED_FOR', '') + return nil unless forwarded_header.is_a?(String) + + forwarded = forwarded_header.split(',', -1) + return nil if forwarded.size > 64 + + forwarded.reverse_each do |value| + address = literal(value.strip) + return nil unless address + return address unless trusted?(address) + end + nil + end + + def peer_address(env) + socket = env['puma.socket'] + # Only Nginx on the configured, access-controlled Unix socket may assert X-Real-IP. + # A loopback REMOTE_ADDR alone is NOT evidence of Unix-socket transport. + if @nginx_socket && socket && socket.local_address.unix? && socket.local_address.unix_path == @nginx_socket + literal(env['HTTP_X_REAL_IP']) + elsif socket + literal(socket.remote_address.ip_address) + else + literal(env['REMOTE_ADDR']) + end + rescue IOError, SystemCallError, SocketError + nil + end + + def trusted?(address) + @trusted_proxies.any? { |cidr| cidr.include?(address) } + end + + def literal(value) + return nil unless value.is_a?(String) && value.match?(/\A[0-9a-fA-F:.]+\z/) + + address = IPAddr.new(value) + address.ipv4_mapped? ? address.native : address + rescue IPAddr::Error + nil + end + + def network(cidr) + parts = cidr.split('/', -1) + address = literal(parts.first) + valid_prefix = parts.size == 1 || (parts.size == 2 && parts.last.match?(/\A[0-9]{1,3}\z/) && + parts.last.to_i <= (address&.ipv4? ? 32 : 128) && !(parts.first.include?(':') && address&.ipv4?)) + raise ArgumentError.new("Invalid administrative token network CIDR: #{cidr}") unless address && valid_prefix + + parts.size == 1 ? address : address.mask(parts.last.to_i) + end + end + end + end +end diff --git a/middleware/security_context_setter.rb b/middleware/security_context_setter.rb index ec5a7020324..7405ff0e9d0 100644 --- a/middleware/security_context_setter.rb +++ b/middleware/security_context_setter.rb @@ -10,9 +10,11 @@ class SecurityContextSetter ] ) - def initialize(app, security_context_configurer) + def initialize(app, security_context_configurer, admin_token_network_policy) @app = app @security_context_configurer = security_context_configurer + @admin_token_network_policy = admin_token_network_policy + raise ArgumentError.new('Administrative token network policy is required') unless @admin_token_network_policy end def call(env) @@ -32,6 +34,11 @@ def call(env) security_context_configurer.configure_token_only(header_token) if VCAP::CloudController::SecurityContext.valid_token? + unless @admin_token_network_policy.allowed?(VCAP::CloudController::SecurityContext.token, env) + logger.warn('Administrative token network policy denied request') + VCAP::CloudController::SecurityContext.clear + return [403, { 'Content-Type' => 'application/json', 'Cache-Control' => 'no-store' }, [error_message(env, 'NotAuthorized')]] + end env['cf.user_guid'] = id_from_token env['cf.user_name'] = VCAP::CloudController::SecurityContext.token['user_name'] ::VCAP::Request.user_guid = id_from_token @@ -76,7 +83,7 @@ def error_message(env, error_name) if version == '/v2' ErrorPresenter.new(api_error, Rails.env.test?, V2ErrorHasher.new(api_error)).to_json - elsif version == '/v3' + else ErrorPresenter.new(api_error, Rails.env.test?, V3ErrorHasher.new(api_error)).to_json end end diff --git a/spec/unit/lib/cloud_controller/security/admin_token_network_policy_spec.rb b/spec/unit/lib/cloud_controller/security/admin_token_network_policy_spec.rb new file mode 100644 index 00000000000..2772d4384a9 --- /dev/null +++ b/spec/unit/lib/cloud_controller/security/admin_token_network_policy_spec.rb @@ -0,0 +1,85 @@ +require 'rspec' +require 'cloud_controller/security/admin_token_network_policy' + +RSpec.describe VCAP::CloudController::Security::AdminTokenNetworkPolicy do + subject(:policy) { described_class.new(options, nginx_socket: '/private/cloud_controller.sock') } + + let(:options) { { enabled: true, allowed_cidrs: ['192.0.2.0/24', '2001:db8::/32'], trusted_proxies: ['10.0.0.0/24'] } } + let(:token) { { 'scope' => ['cloud_controller.admin'] } } + let(:env) { { 'REMOTE_ADDR' => '192.0.2.1' } } + + %w[uaa.admin cloud_controller.admin cloud_controller.admin_read_only cloud_controller.global_auditor zones.tenant.admin zones.*.admin].each do |scope| + it "protects #{scope} in scopes and authorities" do + expect(policy).not_to be_allowed({ 'scope' => [scope] }, { 'REMOTE_ADDR' => '198.51.100.1' }) + expect(policy).not_to be_allowed({ 'scope' => ['openid'], 'authorities' => [scope] }, { 'REMOTE_ADDR' => '198.51.100.1' }) + end + end + + %w[192.0.2.0 192.0.2.255 2001:db8::1 ::ffff:192.0.2.10].each do |address| + it "accepts allowed numeric address #{address}" do + expect(policy).to be_allowed(token, { 'REMOTE_ADDR' => address }) + end + end + + ['192.0.3.0', '2001:db9::1', 'localhost', '127.1', '192.0.2.1/32', '[2001:db8::1]', 'fe80::1%lo0', '', nil].each do |address| + it "fails closed for outside or malformed address #{address.inspect}" do + expect(policy).not_to be_allowed(token, { 'REMOTE_ADDR' => address }) + end + end + + ['localhost', '127.1', '192.0.2.1/33', '2001:db8::/129', '192.0.2.1/', '192.0.2.1/-1', + '192.0.2.1/255.255.255.0', '192.0.2.1/24/1', ' 192.0.2.1', '::ffff:192.0.2.1/120', ''].each do |cidr| + it "rejects invalid configuration #{cidr.inspect} even when disabled" do + expect { described_class.new({ enabled: false, allowed_cidrs: [cidr] }) }.to raise_error(ArgumentError) + expect { described_class.new({ enabled: false, trusted_proxies: [cidr] }) }.to raise_error(ArgumentError) + end + end + + it 'does not trust forwarded or real-IP headers on direct requests, including loopback TCP' do + spoofed = { 'REMOTE_ADDR' => '127.0.0.1', 'HTTP_X_REAL_IP' => '192.0.2.1', 'HTTP_X_FORWARDED_FOR' => '192.0.2.1' } + expect(policy).not_to be_allowed(token, spoofed) + end + + it 'uses the actual TCP socket peer rather than an overridden Rack address' do + socket = double(:socket, local_address: Addrinfo.tcp('127.0.0.1', 3000), remote_address: Addrinfo.tcp('198.51.100.1', 5000)) + expect(policy).not_to be_allowed(token, env.merge('puma.socket' => socket)) + end + + it 'trusts the overwritten Nginx peer only on the configured Unix socket' do + socket = double(:socket, local_address: Addrinfo.unix('/private/cloud_controller.sock')) + request = { 'puma.socket' => socket, 'HTTP_X_REAL_IP' => '10.0.0.1', + 'HTTP_X_FORWARDED_FOR' => '198.51.100.10, 192.0.2.1, 10.0.0.1' } + expect(policy).to be_allowed(token, request) + request['HTTP_X_FORWARDED_FOR'] = '192.0.2.1, 198.51.100.10, 10.0.0.1' + expect(policy).not_to be_allowed(token, request) + request['HTTP_X_REAL_IP'] = '198.51.100.10' + expect(policy).not_to be_allowed(token, request) + end + + it 'does not accept headers on a different Unix socket' do + socket = double(:socket, local_address: Addrinfo.unix('/other.sock'), remote_address: Addrinfo.unix('/client.sock')) + expect(policy).not_to be_allowed(token, env.merge('puma.socket' => socket, 'HTTP_X_REAL_IP' => '192.0.2.1')) + end + + ['192.0.2.1, garbage', '192.0.2.1,', '', '10.0.0.1, 10.0.0.2'].each do |forwarded| + it "fails closed on unresolved trusted-proxy chain #{forwarded.inspect}" do + expect(policy).not_to be_allowed(token, { 'REMOTE_ADDR' => '10.0.0.1', 'HTTP_X_FORWARDED_FOR' => forwarded }) + end + end + + it 'bounds forwarded-chain parsing' do + expect(policy).not_to be_allowed(token, { 'REMOTE_ADDR' => '10.0.0.1', 'HTTP_X_FORWARDED_FOR' => (['192.0.2.1'] * 65).join(',') }) + end + + it 'preserves ordinary tokens and disabled policy, but enabled empty allowlists deny administrative tokens' do + expect(policy).to be_allowed({ 'scope' => ['openid', 'zones.read', 'not.uaa.admin'] }, {}) + expect(described_class.new).to be_allowed(token, {}) + expect(described_class.new({ enabled: true, allowed_cidrs: [] })).not_to be_allowed(token, env) + end + + it 'supports additional exact protected scope names without replacing built-ins' do + additional = described_class.new({ enabled: true, additional_protected_scopes: ['clients.write'] }) + expect(additional).not_to be_allowed({ 'scope' => ['clients.write'] }, env) + expect(additional).not_to be_allowed(token, env) + end +end diff --git a/spec/unit/middleware/security_context_setter_spec.rb b/spec/unit/middleware/security_context_setter_spec.rb index 8acb5bbd2f4..f0d6e11fc4e 100644 --- a/spec/unit/middleware/security_context_setter_spec.rb +++ b/spec/unit/middleware/security_context_setter_spec.rb @@ -1,10 +1,12 @@ require 'spec_helper' require 'security_context_setter' +require 'cloud_controller/security/admin_token_network_policy' module CloudFoundry module Middleware RSpec.describe SecurityContextSetter do - let(:middleware) { SecurityContextSetter.new(app, security_context_configurer) } + let(:policy) { VCAP::CloudController::Security::AdminTokenNetworkPolicy.new } + let(:middleware) { SecurityContextSetter.new(app, security_context_configurer, policy) } let(:app) { double(:app, call: [200, {}, 'a body']) } let(:path_info) { '/v2/foo' } let(:env) do @@ -35,6 +37,32 @@ module Middleware expect(header).to eq({ 'X-USER-GUID' => 'user-id-1' }) end + context 'with administrative token network policy enabled' do + let(:policy) { VCAP::CloudController::Security::AdminTokenNetworkPolicy.new({ enabled: true, allowed_cidrs: ['192.0.2.0/24'] }) } + let(:middleware) { SecurityContextSetter.new(app, security_context_configurer, policy) } + let(:token_information) { { 'user_id' => 'user-id-1', 'scope' => ['cloud_controller.admin'] } } + + it 'denies before calling downstream middleware or creating users and clears validated context' do + status, headers, body = middleware.call(env.merge('REMOTE_ADDR' => '198.51.100.1')) + expect(status).to eq(403) + expect(headers).not_to include('X-USER-GUID') + expect(Oj.load(body.first)['error_code']).to eq('CF-NotAuthorized') + expect(app).not_to have_received(:call) + expect(VCAP::CloudController::SecurityContext.token).to be_nil + end + + it 'allows validated administrative tokens from allowed addresses' do + expect(middleware.call(env.merge('REMOTE_ADDR' => '192.0.2.1')).first).to eq(200) + expect(app).to have_received(:call) + end + + it 'applies the same policy to v3 requests' do + status, _, body = middleware.call(env.merge('PATH_INFO' => '/v3/apps', 'REMOTE_ADDR' => '198.51.100.1')) + expect(status).to eq(403) + expect(Oj.load(body.first)['errors'].first['title']).to eq('CF-NotAuthorized') + end + end + context 'when given a UAA user token' do it 'sets user name and guid on the env' do middleware.call(env) @@ -231,6 +259,20 @@ module Middleware middleware.call(upload_env) end + + context 'with administrative token network policy enabled' do + let(:policy) { VCAP::CloudController::Security::AdminTokenNetworkPolicy.new({ enabled: true, allowed_cidrs: ['192.0.2.0/24'] }) } + let(:token_information) { { 'user_id' => 'user-id-1', 'scope' => ['cloud_controller.admin'] } } + + it 'denies administrative tokens even when relaxed expiry decoding succeeds' do + expect(VCAP::CloudController::UaaTokenDecoder).to receive(:new). + with(anything, alternate_reference_time: 123).and_return(token_decoder) + + expect(middleware.call(upload_env.merge('REMOTE_ADDR' => '198.51.100.1')).first).to eq(403) + expect(app).not_to have_received(:call) + expect(VCAP::CloudController::SecurityContext.token).to be_nil + end + end end context 'when upload_start_time in the POST body is in the future' do