[9.4](backport #7009) fix: enforce policy-based access control on artifact downloads

[mergify]

What is the problem this PR solves?

The artifact download endpoint (/api/fleet/artifacts/{id}/{sha256}) only validates the agent's API key but never checks whether the requested artifact belongs to the agent's assigned policy. This means an agent enrolled under one policy can download artifacts belonging to a different policy if it knows the artifact ID and SHA256 hash. For example, an agent enrolled under a policy with no integrations can retrieve Elastic Defend trust lists, exception lists, and other security artifacts from another policy.

How does this PR solve the problem?

Implements the authorizeArtifact() function (previously a no-op that returned nil) to enforce policy-based access control:

1. Adds a GetPolicy(ctx, policyID) method to the policy.Monitor interface that returns the cached policy for a given ID (reloads from ES on cache miss).
2. In authorizeArtifact, fetches the agent's policy via the monitor using agent.AgentPolicyID and verifies that the requested artifact (identifier + decoded_sha256) appears in the policy's inputs[].artifact_manifest.artifacts.
3. Returns 403 Forbidden (ErrUnauthorizedArtifact) if the artifact is not listed in the agent's assigned policy.

How to test this PR locally

1. Set up Fleet Server with Elasticsearch and Kibana
2. Create two agent policies: Victim-Policy with Elastic Defend integration (add a trusted application), and Attacker-Policy with no integrations
3. Create an enrollment token for Attacker-Policy and enroll an agent
4. Attempt to download an artifact belonging to Victim-Policy using the attacker agent's API key — should now receive 403 Forbidden instead of the artifact contents
5. Verify that an agent enrolled under Victim-Policy can still download its own artifacts normally (200 OK)

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

  ---

  This is an automatic backport of pull request fix: enforce policy-based access control on artifact downloads #7009 done by Mergify.

[github-actions]

TL;DR

The Buildkite E2E step failed (.buildkite/scripts/e2e_test.sh), but the captured log excerpt does not contain the actual failing assertion/test; it only shows a final package-level FAIL after one suite summary that is entirely PASS. Immediate action: rerun the E2E step with full go test output preserved (or attach build/test-e2e-*.out) so the first failing test line is visible.

Remediation

• Re-run the failing step and capture the full output of mage test:e2e (the first --- FAIL: / panic line is missing in the current artifact).
• Run targeted repro locally/CI for artifact-related E2E tests introduced in this PR:
  • TEST_RUN='TestStandAloneCurrentAPI/TestArtifact' mage test:e2e
  • TEST_RUN='TestStandAlone20230601API/TestArtifact' mage test:e2e
• If the failure is a timeout/flaky 403 loop, add explicit logging of response codes in the retry path and fail with the last response body to make future CI failures diagnosable.

Investigation details

Root Cause

I could not conclusively determine a single root cause from the provided Buildkite log file because the failing assertion is not present in the available output.

What is verifiable:

• Build step runs mage test:e2e test:junitReport in .buildkite/scripts/e2e_test.sh:16.
• E2E execution itself is go test ... -timeout 30m -race -p 1 ./... from magefile.go:2120 and magefile.go:2162.
• The captured log shows TestStandAloneRunningSuite subtests all passing, then package-level FAIL for github.com/elastic/fleet-server/testing/e2e.

Given this PR’s scope, the most likely affected surface is the updated artifact E2E flow:

• testing/e2e/api_version/client_api_current.go:407
• testing/e2e/api_version/client_api_2023_06_01.go:307
• testing/e2e/scaffold/scaffold.go:599

These paths now depend on eventual consistency (policy artifact manifest visibility + retry-on-403), which is a common source of CI-only flakiness/timeouts when logs are truncated.

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/15017
• Job/step: E2E Test (.buildkite/scripts/e2e_test.sh)
• Key log excerpt:
  • --- PASS: TestStandAloneRunningSuite ...
  • FAIL github.com/elastic/fleet-server/testing/e2e 1378.250s
  • no --- FAIL: block or panic for the actual failing test is present in the provided log artifact.

Verification

• Not run in this environment (Docker-in-Docker unavailable), so verification is based on Buildkite artifacts and repository source inspection only.

Follow-up

• I checked for matching flaky-test issues for TestWithElasticsearchConnectionFailures, TestWithElasticsearchConnectionFlakyness, and artifact API suite names; no direct match was found.

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

• #3909 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[mergify]

This pull request has not been merged yet. Could you please review and merge it @ycombinator? 🙏

[mergify]

This pull request has not been merged yet. Could you please review and merge it @ycombinator? 🙏

[mergify]

This pull request has not been merged yet. Could you please review and merge it @ycombinator? 🙏

[ebeahan]

@ycombinator do we need this backport?

[mergify]

This pull request has not been merged yet. Could you please review and merge it @ycombinator? 🙏