Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 78 additions & 0 deletions docs/admin-token-network-policy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Administrative token network policy

Cloud Controller can restrict validated administrative bearer tokens to approved
source networks. The policy runs before downstream middleware, user creation,
and API authorization, including uploads using the relaxed token-expiry decoder.
It applies to old and new tokens and all API paths. Ordinary tokens and
unauthenticated requests retain their existing behavior.

```yaml
admin_token_network_policy:
enabled: true
allowed_cidrs: [192.0.2.10/32, '2001:db8:1234::/48']
trusted_proxies: [10.0.0.10/32]
additional_protected_scopes: [clients.write]
```

These are documentation examples, not deployment values. All four fields are
operator-controlled. The policy is disabled by default; allowed networks, trusted
proxies, and additional protected scopes default to empty lists. Enabled with an
empty allowlist means deny-all for protected tokens. Invalid numeric addresses or
CIDRs fail startup, even when disabled. Restart to apply configuration changes.

Protected scopes/authorities are `cloud_controller.admin`, `cloud_controller.admin_read_only`,
`cloud_controller.global_auditor`, `uaa.admin`, and `zones.<id>.admin` (any nonempty
zone ID). Additional scope names extend this set by exact match. Checking both
`scope` and `authorities` prevents a narrowed client-credentials scope from
bypassing protection. The list is not an exhaustive defense against escalation
through other delegation privileges; review and add those explicitly.

Allowlist/trusted-proxy entries are numeric IPv4/IPv6 hosts or CIDRs with decimal
prefix lengths. Bare addresses match a single host; CIDRs mask host bits.
Hostnames, abbreviated IPv4, dotted netmasks, ports, brackets, whitespace,
interface IDs, and mapped-IPv6 CIDR notation are invalid. IPv4-mapped IPv6
requests match IPv4 networks. No DNS resolution is performed.

## Source-address contract

- TCP requests use the actual Puma socket peer (or `REMOTE_ADDR` on other Rack
servers). Real-IP and forwarded headers from untrusted peers are ignored,
including on loopback TCP.
- Nginx may assert `X-Real-IP` only when the actual Puma socket is Unix-domain and
its local path matches `nginx.instance_socket` with `nginx.use_nginx` enabled.
The BOSH Nginx templates overwrite `X-Real-IP` with `$remote_addr` on public and
internal proxy paths, including upload forwarding, and strip `X-Real_IP` to
prevent a Rack-normalization collision. Deploy the matching Nginx template
changes before enabling the policy. The socket and its parents must be protected
against untrusted local access. Loopback does not prove this transport.
- Only explicitly listed `trusted_proxies` can forward source addresses. Walk
`X-Forwarded-For` from right to left until the first untrusted numeric address,
which is the source. Nginx's `$proxy_add_x_forwarded_for` appends its immediate
peer. Ignore attacker-controlled prefixes beyond the first untrusted hop;
malformed encountered hops, missing addresses, more than 64 entries, and
chains with no untrusted source fail closed.

List actual router/proxy addresses, **not client networks**, in `trusted_proxies`.
Never broadly trust all private addresses just because your Gorouters have private
IPs. Every ingress hop must sanitize or append forwarded headers correctly, and
direct backend access must be protected. Other Rack servers must ensure
`REMOTE_ADDR` remains an authoritative peer address.

## Denial and coordinated rollout

Denials return HTTP 403 using the existing `CF-NotAuthorized` v2/v3 error format,
clear the validated security context, and do not invoke downstream user creation
or API handling. The fixed denial log message includes no credentials or tokens.

Use matching allowed networks and protected additions in UAA's
`uaa.admin_token_network_policy` to restrict issuance, refresh, and UAA usage.
In capi-release, the properties have the `cc.admin_token_network_policy` prefix
with `enabled`, `allowed_cidrs`, `trusted_proxies`, and
`additional_protected_scopes` suffixes. Release wiring requires application
versions that implement these settings; old binaries do not gain protection
by rendering configuration. Other offline JWT consumers need their own policy.

Before enabling, inventory administrator VPN/NAT egress and internal automation,
test public/internal ingress and forged headers, and verify issuance on an allowed
network followed by use/refresh on a denied network. Keep a tested operator
configuration rollback procedure.
6 changes: 6 additions & 0 deletions lib/cloud_controller/config_schemas/api_schema.rb
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,12 @@ class ApiSchema < VCAP::Config
define_schema do
{
external_port: Integer,
optional(:admin_token_network_policy) => {
enabled: bool,
allowed_cidrs: [String],
optional(:trusted_proxies) => [String],
optional(:additional_protected_scopes) => [String]
},
external_domain: String,
temporary_disable_deployments: bool,
optional(:temporary_disable_v2_staging) => bool,
Expand Down
11 changes: 10 additions & 1 deletion lib/cloud_controller/rack_app_builder.rb
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,14 @@
require 'block_v3_only_roles'
require 'below_min_cli_warning'
require 'user_context_setter'
require 'cloud_controller/security/admin_token_network_policy'

module VCAP::CloudController
class RackAppBuilder
def build(config, request_metrics, request_logs)
token_decoder = VCAP::CloudController::UaaTokenDecoder.new(config.get(:uaa))
configurer = VCAP::CloudController::Security::SecurityContextConfigurer.new(token_decoder)
admin_token_network_policy = admin_token_network_policy(config)

concurrency_rate_limiter_config = concurrency_rate_limiter_config(config)
rate_limiter_config = rate_limiter_config(config)
Expand All @@ -31,7 +33,7 @@ def build(config, request_metrics, request_logs)
use CloudFoundry::Middleware::VcapRequestContextSetter
use CloudFoundry::Middleware::BelowMinCliWarning if config.get(:warn_if_below_min_cli_version)
use CloudFoundry::Middleware::NewRelicCustomAttributes if config.get(:newrelic_enabled)
use CloudFoundry::Middleware::SecurityContextSetter, configurer
use CloudFoundry::Middleware::SecurityContextSetter, configurer, admin_token_network_policy
use CloudFoundry::Middleware::Zipkin

use CloudFoundry::Middleware::ConcurrencyRateLimiter, concurrency_rate_limiter_config if config.get(:concurrency_rate_limiter, :enabled)
Expand Down Expand Up @@ -63,6 +65,13 @@ def build(config, request_metrics, request_logs)

private

def admin_token_network_policy(config)
Security::AdminTokenNetworkPolicy.new(
config.get(:admin_token_network_policy) || {},
nginx_socket: config.get(:nginx, :use_nginx) ? config.get(:nginx, :instance_socket) : nil
)
end

def access_log(config)
return unless !config.get(:nginx, :use_nginx) && config.get(:logging, :file)

Expand Down
93 changes: 93 additions & 0 deletions lib/cloud_controller/security/admin_token_network_policy.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
require 'ipaddr'
require 'socket'

module VCAP
module CloudController
module Security
class AdminTokenNetworkPolicy
ADMIN_SCOPES = %w[uaa.admin cloud_controller.admin cloud_controller.admin_read_only cloud_controller.global_auditor].freeze

def initialize(options={}, nginx_socket: nil)
@enabled = options.fetch(:enabled, false)
@networks = options.fetch(:allowed_cidrs, []).map { |cidr| network(cidr) }
@trusted_proxies = options.fetch(:trusted_proxies, []).map { |cidr| network(cidr) }
@additional_scopes = options.fetch(:additional_protected_scopes, [])
@nginx_socket = nginx_socket
end

def allowed?(token, env)
return true unless @enabled && protected_token?(token)

address = source_address(env)
address && @networks.any? { |cidr| cidr.include?(address) }
end

private

def protected_token?(token)
%w[scope authorities].any? do |claim|
values = token[claim].is_a?(String) ? token[claim].split(/[ ,]+/) : Array(token[claim])
values.any? { |scope| ADMIN_SCOPES.include?(scope) || @additional_scopes.include?(scope) || (scope.is_a?(String) && scope.match?(/\Azones\..+\.admin\z/)) }
end
end

def source_address(env)
peer = peer_address(env)
return nil unless peer
return peer unless trusted?(peer)

forwarded_header = env.fetch('HTTP_X_FORWARDED_FOR', '')
return nil unless forwarded_header.is_a?(String)

forwarded = forwarded_header.split(',', -1)
return nil if forwarded.size > 64

forwarded.reverse_each do |value|
address = literal(value.strip)
return nil unless address
return address unless trusted?(address)
end
nil
end

def peer_address(env)
socket = env['puma.socket']
# Only Nginx on the configured, access-controlled Unix socket may assert X-Real-IP.
# A loopback REMOTE_ADDR alone is NOT evidence of Unix-socket transport.
if @nginx_socket && socket && socket.local_address.unix? && socket.local_address.unix_path == @nginx_socket
literal(env['HTTP_X_REAL_IP'])
elsif socket
literal(socket.remote_address.ip_address)
else
literal(env['REMOTE_ADDR'])
end
rescue IOError, SystemCallError, SocketError
nil
end

def trusted?(address)
@trusted_proxies.any? { |cidr| cidr.include?(address) }
end

def literal(value)
return nil unless value.is_a?(String) && value.match?(/\A[0-9a-fA-F:.]+\z/)

address = IPAddr.new(value)
address.ipv4_mapped? ? address.native : address
rescue IPAddr::Error
nil
end

def network(cidr)
parts = cidr.split('/', -1)
address = literal(parts.first)
valid_prefix = parts.size == 1 || (parts.size == 2 && parts.last.match?(/\A[0-9]{1,3}\z/) &&
parts.last.to_i <= (address&.ipv4? ? 32 : 128) && !(parts.first.include?(':') && address&.ipv4?))
raise ArgumentError.new("Invalid administrative token network CIDR: #{cidr}") unless address && valid_prefix

parts.size == 1 ? address : address.mask(parts.last.to_i)
end
end
end
end
end
11 changes: 9 additions & 2 deletions middleware/security_context_setter.rb
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,11 @@ class SecurityContextSetter
]
)

def initialize(app, security_context_configurer)
def initialize(app, security_context_configurer, admin_token_network_policy)
@app = app
@security_context_configurer = security_context_configurer
@admin_token_network_policy = admin_token_network_policy
raise ArgumentError.new('Administrative token network policy is required') unless @admin_token_network_policy
end

def call(env)
Expand All @@ -32,6 +34,11 @@ def call(env)
security_context_configurer.configure_token_only(header_token)

if VCAP::CloudController::SecurityContext.valid_token?
unless @admin_token_network_policy.allowed?(VCAP::CloudController::SecurityContext.token, env)
logger.warn('Administrative token network policy denied request')
VCAP::CloudController::SecurityContext.clear
return [403, { 'Content-Type' => 'application/json', 'Cache-Control' => 'no-store' }, [error_message(env, 'NotAuthorized')]]
end
env['cf.user_guid'] = id_from_token
env['cf.user_name'] = VCAP::CloudController::SecurityContext.token['user_name']
::VCAP::Request.user_guid = id_from_token
Expand Down Expand Up @@ -76,7 +83,7 @@ def error_message(env, error_name)

if version == '/v2'
ErrorPresenter.new(api_error, Rails.env.test?, V2ErrorHasher.new(api_error)).to_json
elsif version == '/v3'
else
ErrorPresenter.new(api_error, Rails.env.test?, V3ErrorHasher.new(api_error)).to_json
end
end
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
require 'rspec'
require 'cloud_controller/security/admin_token_network_policy'

RSpec.describe VCAP::CloudController::Security::AdminTokenNetworkPolicy do
subject(:policy) { described_class.new(options, nginx_socket: '/private/cloud_controller.sock') }

let(:options) { { enabled: true, allowed_cidrs: ['192.0.2.0/24', '2001:db8::/32'], trusted_proxies: ['10.0.0.0/24'] } }
let(:token) { { 'scope' => ['cloud_controller.admin'] } }
let(:env) { { 'REMOTE_ADDR' => '192.0.2.1' } }

%w[uaa.admin cloud_controller.admin cloud_controller.admin_read_only cloud_controller.global_auditor zones.tenant.admin zones.*.admin].each do |scope|
it "protects #{scope} in scopes and authorities" do
expect(policy).not_to be_allowed({ 'scope' => [scope] }, { 'REMOTE_ADDR' => '198.51.100.1' })
expect(policy).not_to be_allowed({ 'scope' => ['openid'], 'authorities' => [scope] }, { 'REMOTE_ADDR' => '198.51.100.1' })
end
end

%w[192.0.2.0 192.0.2.255 2001:db8::1 ::ffff:192.0.2.10].each do |address|
it "accepts allowed numeric address #{address}" do
expect(policy).to be_allowed(token, { 'REMOTE_ADDR' => address })
end
end

['192.0.3.0', '2001:db9::1', 'localhost', '127.1', '192.0.2.1/32', '[2001:db8::1]', 'fe80::1%lo0', '', nil].each do |address|
it "fails closed for outside or malformed address #{address.inspect}" do
expect(policy).not_to be_allowed(token, { 'REMOTE_ADDR' => address })
end
end

['localhost', '127.1', '192.0.2.1/33', '2001:db8::/129', '192.0.2.1/', '192.0.2.1/-1',
'192.0.2.1/255.255.255.0', '192.0.2.1/24/1', ' 192.0.2.1', '::ffff:192.0.2.1/120', ''].each do |cidr|
it "rejects invalid configuration #{cidr.inspect} even when disabled" do
expect { described_class.new({ enabled: false, allowed_cidrs: [cidr] }) }.to raise_error(ArgumentError)
expect { described_class.new({ enabled: false, trusted_proxies: [cidr] }) }.to raise_error(ArgumentError)
end
end

it 'does not trust forwarded or real-IP headers on direct requests, including loopback TCP' do
spoofed = { 'REMOTE_ADDR' => '127.0.0.1', 'HTTP_X_REAL_IP' => '192.0.2.1', 'HTTP_X_FORWARDED_FOR' => '192.0.2.1' }
expect(policy).not_to be_allowed(token, spoofed)
end

it 'uses the actual TCP socket peer rather than an overridden Rack address' do
socket = double(:socket, local_address: Addrinfo.tcp('127.0.0.1', 3000), remote_address: Addrinfo.tcp('198.51.100.1', 5000))
expect(policy).not_to be_allowed(token, env.merge('puma.socket' => socket))
end

it 'trusts the overwritten Nginx peer only on the configured Unix socket' do
socket = double(:socket, local_address: Addrinfo.unix('/private/cloud_controller.sock'))
request = { 'puma.socket' => socket, 'HTTP_X_REAL_IP' => '10.0.0.1',
'HTTP_X_FORWARDED_FOR' => '198.51.100.10, 192.0.2.1, 10.0.0.1' }
expect(policy).to be_allowed(token, request)
request['HTTP_X_FORWARDED_FOR'] = '192.0.2.1, 198.51.100.10, 10.0.0.1'
expect(policy).not_to be_allowed(token, request)
request['HTTP_X_REAL_IP'] = '198.51.100.10'
expect(policy).not_to be_allowed(token, request)
end

it 'does not accept headers on a different Unix socket' do
socket = double(:socket, local_address: Addrinfo.unix('/other.sock'), remote_address: Addrinfo.unix('/client.sock'))
expect(policy).not_to be_allowed(token, env.merge('puma.socket' => socket, 'HTTP_X_REAL_IP' => '192.0.2.1'))
end

['192.0.2.1, garbage', '192.0.2.1,', '', '10.0.0.1, 10.0.0.2'].each do |forwarded|
it "fails closed on unresolved trusted-proxy chain #{forwarded.inspect}" do
expect(policy).not_to be_allowed(token, { 'REMOTE_ADDR' => '10.0.0.1', 'HTTP_X_FORWARDED_FOR' => forwarded })
end
end

it 'bounds forwarded-chain parsing' do
expect(policy).not_to be_allowed(token, { 'REMOTE_ADDR' => '10.0.0.1', 'HTTP_X_FORWARDED_FOR' => (['192.0.2.1'] * 65).join(',') })
end

it 'preserves ordinary tokens and disabled policy, but enabled empty allowlists deny administrative tokens' do
expect(policy).to be_allowed({ 'scope' => ['openid', 'zones.read', 'not.uaa.admin'] }, {})
expect(described_class.new).to be_allowed(token, {})
expect(described_class.new({ enabled: true, allowed_cidrs: [] })).not_to be_allowed(token, env)
end

it 'supports additional exact protected scope names without replacing built-ins' do
additional = described_class.new({ enabled: true, additional_protected_scopes: ['clients.write'] })
expect(additional).not_to be_allowed({ 'scope' => ['clients.write'] }, env)
expect(additional).not_to be_allowed(token, env)
end
end
Loading
Loading