Skip to content

Gateway can roll onto an empty resources file (deleted Secret, chart Secret adoption) #7

Open

Description

@shreemaan-abhishek

Problem

Two paths roll every gateway pod onto an empty file, a full outage of all models.

(a) Deleted Secret plus a failing document. With no target Secret and Outcome.Document == nil, the reconciler seeds an empty file and stamps its hash (reconciler.go#L130-L143). Pods with an empty file still pass readiness. The status message is an empty (file): , and the log line says "keeping the served version".

Related: deleting the Secret also loses the record of served entries, so a resource kept on its old version (invalid new spec) is dropped and rolled out even when the rest of the document is valid.

(b) Chart Secret collision. The chart names its inline Secret <fullname>-resources, which is aisix-resources for release aisix or fullnameOverride=aisix, the controller's default.

  • Controller first: helm install fails loudly on ownership metadata.
  • Helm first (migrating an existing inline release): the controller adopts the Secret, replaces it with an empty document, flips managed-by, and stamps the empty hash, so pods roll to empty. Each later helm upgrade writes it back and the next reconcile overwrites it again.

Scores

Reproducibility Likelihood Severity Existing mitigations Detectability Test coverage Fix cost vs benefit
10 4 9 3 3 2 8

Scores are out of 10. Higher is worse for likelihood and severity. Higher is better for mitigations, detectability, test coverage and cost vs benefit (10 = cheap fix, high value).

Fix

  1. If the target Secret exists without app.kubernetes.io/managed-by=aisix-controller, skip the write and the stamp, set Programmed=False with reason SecretNotOwned and a hint to use standalone.existingSecret, and resync. The chart labels its Secret managed-by: Helm, and Secrets this controller created already carry the label.
  2. Seed only when the Deployment is missing or not yet stamped. When it is stamped and the document fails, write nothing, skip the stamp, and report the failure with its real message.
  3. Optional: keep the last written document and index in memory as the fallback when the Secret is missing.

Tests to add

Adopting a foreign Secret; a deleted Secret plus a document-wide failure; a deleted Secret plus a resource kept on its old version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions