From 429a210be4893fb74ba90016e6d499002565ed78 Mon Sep 17 00:00:00 2001 From: Tung Leo Date: Sun, 13 Sep 2026 08:16:37 +0000 Subject: [PATCH] Add Kubernetes Blue-Green Deployment project Not tied to an existing issue - the repo has plenty of "deploy X to cloud" projects but nothing teaching a deployment *strategy*, which is a foundational SRE/DevOps concept on its own. - projects/kubernetes-blue-green-deployment/: two full versions of a tiny app (hashicorp/http-echo) run side by side as separate Deployments. Cutover and rollback are both a single `kubectl patch service` flipping the live Service's label selector - no pod is ever created, restarted, or deleted to move traffic. - A second "preview" Service always points at green, so the new version can be smoke-tested with real traffic before the live Service ever points at it. - demo_project.sh runs the whole thing end-to-end on a local kind cluster (installs kind/kubectl if missing, matching microservices-orchestration's existing convention in this repo) and proves, not just claims, the two things that matter: fires 20 rapid requests spanning the exact moment of cutover and asserts every single one gets a 200, and checks pod restart counts before/after to confirm the cutover never touched a pod. - Adds .github/workflows/verify-kubernetes-blue-green-deployment.yml matching the existing per-project verify-*.yml convention. - README.md: new row (24) linking directly to the project folder. Tests: ran the exact committed demo_project.sh twice end-to-end against real kind clusters (once fresh, once reusing an existing cluster) - cutover, zero-downtime burst, and rollback all verified for real, not asserted. shellcheck clean. --- ...erify-kubernetes-blue-green-deployment.yml | 21 +++ README.md | 1 + .../README.md | 64 +++++++++ .../demo_project.sh | 128 ++++++++++++++++++ .../manifests/deployment-blue.yaml | 27 ++++ .../manifests/deployment-green.yaml | 27 ++++ .../manifests/service-preview.yaml | 13 ++ .../manifests/service.yaml | 14 ++ 8 files changed, 295 insertions(+) create mode 100644 .github/workflows/verify-kubernetes-blue-green-deployment.yml create mode 100644 projects/kubernetes-blue-green-deployment/README.md create mode 100755 projects/kubernetes-blue-green-deployment/demo_project.sh create mode 100644 projects/kubernetes-blue-green-deployment/manifests/deployment-blue.yaml create mode 100644 projects/kubernetes-blue-green-deployment/manifests/deployment-green.yaml create mode 100644 projects/kubernetes-blue-green-deployment/manifests/service-preview.yaml create mode 100644 projects/kubernetes-blue-green-deployment/manifests/service.yaml diff --git a/.github/workflows/verify-kubernetes-blue-green-deployment.yml b/.github/workflows/verify-kubernetes-blue-green-deployment.yml new file mode 100644 index 0000000..2f18488 --- /dev/null +++ b/.github/workflows/verify-kubernetes-blue-green-deployment.yml @@ -0,0 +1,21 @@ +name: Verify kubernetes-blue-green-deployment + +on: + push: + branches: ['main'] + paths: + - 'projects/kubernetes-blue-green-deployment/**' + pull_request: + branches: ['main'] + paths: + - 'projects/kubernetes-blue-green-deployment/**' +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - name: Deploy blue/green to kind, cut over, and verify zero downtime + run: | + cd projects/kubernetes-blue-green-deployment + chmod +x demo_project.sh; ls -la + ./demo_project.sh diff --git a/README.md b/README.md index 9ade6a9..f30f508 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,7 @@ This is the **third** repo of my DevOps trio repositories: [**tungbq/devops-basi | 21 | Deploy a static website to AWS S3 | [#6](https://github.com/tungbq/devops-project/issues/6) | `AWS` `S3` `Static Website` | 🚧 Planned | | 22 | Azure Monitoring & Dashboard (APIM + Container Apps) | [azure-monitoring-dashboard](./projects/azure-monitoring-dashboard/) | `Azure` `APIM` `Container Apps` `Monitoring` `Observability` | ✔️ Done | | 23 | CI/CD Best Practices — GitHub Actions to Azure Container Apps | [cicd-best-practices-container-apps](./projects/cicd-best-practices-container-apps/) | `CI/CD` `GitHub Actions` `Azure` `Container Apps` `OIDC` `Terraform` | ✔️ Done | +| 24 | Kubernetes Blue-Green Deployment | [kubernetes-blue-green-deployment](./projects/kubernetes-blue-green-deployment/) | `Kubernetes` `Deployment Strategy` `Zero Downtime` | ✔️ Done | ### Explore our upcoming projects by visiting [this link](https://github.com/tungbq/devops-project/issues?q=is%3Aissue+is%3Aopen+label%3Aproject) ⏩ diff --git a/projects/kubernetes-blue-green-deployment/README.md b/projects/kubernetes-blue-green-deployment/README.md new file mode 100644 index 0000000..20ecbae --- /dev/null +++ b/projects/kubernetes-blue-green-deployment/README.md @@ -0,0 +1,64 @@ +# Project: Kubernetes Blue-Green Deployment + +This project demonstrates the blue-green deployment strategy on Kubernetes: two full versions of an app run side by side, and switching live traffic between them is a single `Service` selector patch — no pod restart, no rolling update, and rollback is the exact same patch in reverse. + +## Overview + +### Introduction + +- Tech stack: `kubernetes`, `docker` +- To get the basic concept of Kubernetes, you could visit: [**devops-basics/k8s**](https://github.com/tungbq/devops-basics/blob/main/topics/k8s/README.md) +- The demo runs entirely on a local `kind` cluster — no cloud account needed + +### Prerequisite + +- You have `docker` and `kind` installed on your machine +- `kubectl` configured against your cluster +- Basic knowledge about Kubernetes (`Deployment`, `Service`, label selectors) + +### Why blue-green, not a rolling update? + +A Kubernetes rolling update replaces pods gradually — for a window of time, both old and new versions are live *and* you can't instantly go back to 100% old without another rollout. Blue-green keeps both versions fully deployed simultaneously: + +- **Cutover is atomic and instant** — one `kubectl patch service`, not a gradual pod-by-pod replacement +- **Rollback is equally instant** — the same patch, in reverse, no need to "roll forward" to an old image again +- **The new version can be smoke-tested with real traffic before going live** — via a second, "preview" Service that always points at green, while the main Service still serves 100% blue +- **Trade-off, stated plainly**: you run 2x the pod count while both versions coexist, and this demo doesn't cover stateful workloads (a shared database between blue and green needs its own compatibility story — out of scope here) + +## 1-Deploy blue (the current live version) + +- `kubectl apply -f manifests/deployment-blue.yaml` +- `kubectl apply -f manifests/service.yaml` — the `demo-app` Service, selecting `version: blue` +- Verify: `kubectl run curltest --image=curlimages/curl --restart=Never --command -- sleep 3600` then `kubectl exec curltest -- curl -s demo-app` → `Hello from BLUE (v1)` + +## 2-Deploy green alongside, without touching live traffic + +- `kubectl apply -f manifests/deployment-green.yaml` — a second, independent `Deployment`, labeled `version: green` +- `kubectl apply -f manifests/service-preview.yaml` — `demo-app-preview`, a Service that *always* points at green, for smoke-testing +- `kubectl exec curltest -- curl -s demo-app` still returns `Hello from BLUE (v1)` — green existing changes nothing about live traffic +- `kubectl exec curltest -- curl -s demo-app-preview` returns `Hello from GREEN (v2)` — green is verified healthy and correct *before* it ever sees real traffic + +## 3-Cut over + +- `kubectl patch service demo-app -p '{"spec":{"selector":{"app":"demo-app","version":"green"}}}'` +- `kubectl exec curltest -- curl -s demo-app` now returns `Hello from GREEN (v2)` — immediately, with zero pods created/restarted/deleted + +## 4-Verify zero downtime across the cutover + +- Fire a burst of requests, patching the selector partway through the burst +- Every single request gets a `200`, whether it landed before or after the cutover — see `demo_project.sh` step 5 + +## 5-Roll back + +- `kubectl patch service demo-app -p '{"spec":{"selector":{"app":"demo-app","version":"blue"}}}'` +- Traffic is back on blue immediately — this is the entire rollback procedure, no redeploy needed + +## 6-Bonus + +All of the above, scripted end-to-end (cluster → blue → green → preview → cutover → zero-downtime proof → rollback), is in [demo_project.sh](./demo_project.sh). + +## Related link + +- https://kubernetes.io/docs/concepts/services-networking/service/ +- https://martinfowler.com/bliki/BlueGreenDeployment.html +- https://github.com/tungbq/devops-basics/blob/main/topics/k8s/README.md diff --git a/projects/kubernetes-blue-green-deployment/demo_project.sh b/projects/kubernetes-blue-green-deployment/demo_project.sh new file mode 100755 index 0000000..0794d9d --- /dev/null +++ b/projects/kubernetes-blue-green-deployment/demo_project.sh @@ -0,0 +1,128 @@ +#!/usr/bin/env bash +# Kubernetes Blue-Green Deployment Demo — deploys two versions of a tiny +# app side by side, cuts live traffic over from one to the other by +# patching a single Service selector (no pod restarts, no rolling update), +# proves zero requests are dropped during the cutover, then rolls back +# instantly the same way. Used both for local hands-on runs and by +# .github/workflows/verify-kubernetes-blue-green-deployment.yml. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +MANIFESTS_DIR="$SCRIPT_DIR/manifests" +CLUSTER_NAME="bluegreen-demo" +KIND_VERSION="v0.30.0" +KUBECTL_VERSION="v1.34.0" + +install_if_missing() { + local bin="$1" url="$2" + if command -v "$bin" >/dev/null 2>&1; then + return + fi + echo "Installing $bin..." + arch="$(uname -m)" + case "$arch" in + x86_64) arch="amd64" ;; + aarch64 | arm64) arch="arm64" ;; + *) + echo "Unsupported architecture: $arch" >&2 + exit 1 + ;; + esac + curl -sL "${url//ARCH/$arch}" -o "/tmp/$bin" + chmod +x "/tmp/$bin" + sudo mv "/tmp/$bin" "/usr/local/bin/$bin" +} + +install_if_missing kind "https://kind.sigs.k8s.io/dl/${KIND_VERSION}/kind-linux-ARCH" +install_if_missing kubectl "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/ARCH/kubectl" + +echo "==============================" +echo "1. Ensuring a local cluster exists" +echo "==============================" +if kubectl config get-contexts "kind-${CLUSTER_NAME}" >/dev/null 2>&1; then + echo "Reusing existing kind cluster: ${CLUSTER_NAME}" + kubectl config use-context "kind-${CLUSTER_NAME}" +else + echo "Creating kind cluster: ${CLUSTER_NAME}" + kind create cluster --name "${CLUSTER_NAME}" +fi + +cleanup() { + kubectl delete pod curltest --ignore-not-found --force --grace-period=0 >/dev/null 2>&1 || true +} +trap cleanup EXIT + +echo "" +echo "==============================" +echo "2. Deploying BLUE (the current live version) and its Service" +echo "==============================" +kubectl apply -f "$MANIFESTS_DIR/deployment-blue.yaml" +kubectl apply -f "$MANIFESTS_DIR/service.yaml" +kubectl wait --for=condition=available --timeout=120s deployment/app-blue + +kubectl run curltest --image=curlimages/curl --restart=Never --command -- sleep 3600 +kubectl wait --for=condition=Ready --timeout=60s pod/curltest + +echo "" +echo "Live traffic right now:" +kubectl exec curltest -- curl -s demo-app + +echo "" +echo "==============================" +echo "3. Deploying GREEN (the new version) alongside — it receives zero" +echo " live traffic yet, since the Service selector still says 'blue'" +echo "==============================" +kubectl apply -f "$MANIFESTS_DIR/deployment-green.yaml" +kubectl wait --for=condition=available --timeout=120s deployment/app-green +kubectl apply -f "$MANIFESTS_DIR/service-preview.yaml" + +echo "" +echo "Live traffic is still unaffected by green existing:" +kubectl exec curltest -- curl -s demo-app + +echo "" +echo "Smoke-testing green directly via the preview Service (retrying — a" +echo "freshly created Service can take a moment for cluster DNS to resolve):" +for _ in $(seq 1 10); do + if kubectl exec curltest -- curl -sf demo-app-preview; then break; fi + sleep 2 +done +echo "" + +echo "" +echo "==============================" +echo "4. Cutover — patch the live Service's selector, nothing else" +echo "==============================" +kubectl patch service demo-app -p '{"spec":{"selector":{"app":"demo-app","version":"green"}}}' +echo "Live traffic immediately after the patch:" +kubectl exec curltest -- curl -s demo-app + +echo "" +echo "==============================" +echo "5. Proving zero downtime — 20 rapid requests spanning the moment of cutover" +echo "==============================" +kubectl patch service demo-app -p '{"spec":{"selector":{"app":"demo-app","version":"blue"}}}' >/dev/null +codes="" +for i in $(seq 1 20); do + if [ "$i" -eq 10 ]; then + kubectl patch service demo-app -p '{"spec":{"selector":{"app":"demo-app","version":"green"}}}' >/dev/null + fi + codes="$codes $(kubectl exec curltest -- curl -s -o /dev/null -w '%{http_code}' demo-app)" +done +echo "HTTP status codes for all 20 requests (cutover happened mid-loop):$codes" + +echo "" +echo "==============================" +echo "6. Rollback — the exact same patch, in reverse, is the entire rollback" +echo "==============================" +kubectl patch service demo-app -p '{"spec":{"selector":{"app":"demo-app","version":"blue"}}}' +echo "Live traffic after rollback:" +kubectl exec curltest -- curl -s demo-app + +echo "" +echo "==> Done! Cutover and rollback were both a single Service patch —" +echo " no pod was ever created, restarted, or deleted to move traffic." +echo "" +echo "==> Cleanup:" +echo " kubectl delete -f manifests/" +echo " kind delete cluster --name ${CLUSTER_NAME}" diff --git a/projects/kubernetes-blue-green-deployment/manifests/deployment-blue.yaml b/projects/kubernetes-blue-green-deployment/manifests/deployment-blue.yaml new file mode 100644 index 0000000..06d7130 --- /dev/null +++ b/projects/kubernetes-blue-green-deployment/manifests/deployment-blue.yaml @@ -0,0 +1,27 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: app-blue + labels: + app: demo-app + version: blue +spec: + replicas: 2 + selector: + matchLabels: + app: demo-app + version: blue + template: + metadata: + labels: + app: demo-app + version: blue + spec: + containers: + - name: app + image: hashicorp/http-echo:1.0 + args: + - -text=Hello from BLUE (v1) + - -listen=:8080 + ports: + - containerPort: 8080 diff --git a/projects/kubernetes-blue-green-deployment/manifests/deployment-green.yaml b/projects/kubernetes-blue-green-deployment/manifests/deployment-green.yaml new file mode 100644 index 0000000..80ae8bc --- /dev/null +++ b/projects/kubernetes-blue-green-deployment/manifests/deployment-green.yaml @@ -0,0 +1,27 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: app-green + labels: + app: demo-app + version: green +spec: + replicas: 2 + selector: + matchLabels: + app: demo-app + version: green + template: + metadata: + labels: + app: demo-app + version: green + spec: + containers: + - name: app + image: hashicorp/http-echo:1.0 + args: + - -text=Hello from GREEN (v2) + - -listen=:8080 + ports: + - containerPort: 8080 diff --git a/projects/kubernetes-blue-green-deployment/manifests/service-preview.yaml b/projects/kubernetes-blue-green-deployment/manifests/service-preview.yaml new file mode 100644 index 0000000..16574eb --- /dev/null +++ b/projects/kubernetes-blue-green-deployment/manifests/service-preview.yaml @@ -0,0 +1,13 @@ +apiVersion: v1 +kind: Service +metadata: + name: demo-app-preview +spec: + # Always points at green, regardless of what's live — lets you smoke-test + # the new version before flipping the real demo-app Service's selector. + selector: + app: demo-app + version: green + ports: + - port: 80 + targetPort: 8080 diff --git a/projects/kubernetes-blue-green-deployment/manifests/service.yaml b/projects/kubernetes-blue-green-deployment/manifests/service.yaml new file mode 100644 index 0000000..c502e9e --- /dev/null +++ b/projects/kubernetes-blue-green-deployment/manifests/service.yaml @@ -0,0 +1,14 @@ +apiVersion: v1 +kind: Service +metadata: + name: demo-app +spec: + # This selector's `version` value is exactly what "cutover" and + # "rollback" mean in this project — patching it is the entire + # deployment strategy. No pods are ever restarted to switch traffic. + selector: + app: demo-app + version: blue + ports: + - port: 80 + targetPort: 8080