Mohamad bader AA

.dockerignore

@@ -0,0 +1,8 @@
+astro
+.git
+.env
+airflow_settings.yaml
+logs/
+.venv
+airflow.db
+airflow.cfg

.hyf/grader_lib.sh

@@ -7,11 +7,9 @@
 # and a set of common static-analysis checks derived from recurring
 # PR review patterns across cohort c55.
 #
-# blocker(): use for leaked-secret findings (a committed profiles.yml/.env,
-# a hardcoded password/connection string). It behaves like fail() for the
-# printed report, but also flips a flag that forces write_score() to report
-# pass=false regardless of the earned point total -- a leaked secret must
-# be fixed before the PR can pass, it cannot be "pointed around."
+# blocker(): use for findings that must fail the PR regardless of points
+# (leaked secrets, missing required evidence like screenshots). Behaves like
+# fail() in the printed report, but forces write_score() to pass=false.
 
 _grader_details=()
 _grader_blocker=false
@@ -38,7 +36,7 @@ write_score() {
   [[ "$score" -ge "$passing" ]] && pass_flag="true"
   if [[ "$_grader_blocker" == true ]]; then
     pass_flag="false"
-    echo "🚫 A blocker was found (leaked secret) -- forcing pass=false regardless of score." >&2
+    echo "🚫 A blocker was found -- forcing pass=false regardless of score." >&2
   fi
   cat > "$outfile" << JSON
 {

.hyf/test.sh

@@ -3,8 +3,9 @@
 # The DAG needs a running Astro/Airflow stack and a live Azure PostgreSQL
 # connection that CI cannot reach, so this checks file presence and code
 # patterns in dags/taxi_pipeline.py and the docs. The actual green run,
-# Screenshot files are presence-checked; content, backfill idempotency, and
-# shared-Airflow deploy are reviewed by a teacher.
+# Screenshot files are required (≥3): missing screenshots force pass=false.
+# Content of those shots, backfill idempotency, and shared-Airflow deploy
+# are still reviewed by a teacher.
 # Total points: 100. Passing score: 60.
 set -euo pipefail
 
@@ -218,9 +219,10 @@ shot_count=$(
 if [[ "$shot_count" -ge 3 ]]; then
   l6=$((l6 + 3)); pass "screenshots: found ${shot_count} image file(s) (need ≥3 for Graph + Grid/run + task log)"
 elif [[ "$shot_count" -gt 0 ]]; then
-  fail "screenshots: only ${shot_count} image file(s) — commit at least 3 (local Graph, green Grid/run, one task log; add shared-UI shot when the VM is up)"
+  # Screenshots are required evidence for teacher review — cannot pass without them.
+  blocker "screenshots: only ${shot_count} image file(s) — commit at least 3 (local Graph, green Grid/run, one task log; add shared-UI shot when the VM is up)"
 else
-  fail "screenshots: none found — commit Graph, Grid/run, and task-log images into the PR (any folder)"
+  blocker "screenshots: none found — commit Graph, Grid/run, and task-log images into the PR (any folder). Screenshots are required; a high code score without them still fails."
 fi
 score=$((score + l6))
 pass "Level 6: documentation + screenshots ($l6/10 pts)"

AI_ASSIST.md

@@ -1,12 +1,285 @@
 # AI assistance log
+## Use 1
 
-<!-- Document at least one point where you used an LLM on this assignment.
-     Never paste connection strings, passwords, or real data. Fill in each field. -->
+**Prompt I sent:** Astro Runtime Version: 3.3-1
 
-## Use 1
+tests/test_dag_integrity.py::test_no_import_errors FAILED                [ 50%]
+tests/test_dag_integrity.py::test_every_dag_has_tags FAILED              [100%]
+
+=================================== FAILURES ===================================
+____________________________ test_no_import_errors _____________________________
+
+    def test_no_import_errors():
+        """Every .py in dags/ must import cleanly."""
+>       dag_bag = DagBag(dag_folder="dags", include_examples=False)
+                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+E       TypeError: DagBag.__init__() got an unexpected keyword argument 'include_examples'
+
+tests/test_dag_integrity.py:20: TypeError
+___________________________ test_every_dag_has_tags ____________________________
+
+    def test_every_dag_has_tags():
+        """Light convention check so DAGs are discoverable via the UI tag filter."""
+>       dag_bag = DagBag(dag_folder="dags", include_examples=False)
+                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+E       TypeError: DagBag.__init__() got an unexpected keyword argument 'include_examples'
+
+tests/test_dag_integrity.py:28: TypeError
+=============================== warnings summary ===============================
+../lib/python3.14/site-packages/_pytest/cacheprovider.py:469
+  /usr/local/lib/python3.14/site-packages/_pytest/cacheprovider.py:469: PytestCacheWarning: could not create cache path /.pytest_cache/v/cache/nodeids: [Errno 13] Permission denied: '/pytest-cache-files-7646t09h'
+    config.cache.set("cache/nodeids", sorted(self.cached_nodeids))
+
+../lib/python3.14/site-packages/_pytest/cacheprovider.py:423
+  /usr/local/lib/python3.14/site-packages/_pytest/cacheprovider.py:423: PytestCacheWarning: could not create cache path /.pytest_cache/v/cache/lastfailed: [Errno 13] Permission denied: '/pytest-cache-files-g152s9jj'
+    config.cache.set("cache/lastfailed", self.lastfailed)
+
+-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
+=========================== short test summary info ============================
+FAILED tests/test_dag_integrity.py::test_no_import_errors - TypeError: DagBag...
+FAILED tests/test_dag_integrity.py::test_every_dag_has_tags - TypeError: DagB...
+======================== 2 failed, 2 warnings in 3.03s =========================
+Error: pytest failed
+
+**What the model answered:** The root cause of your test failure is that Airflow 3 removed the include_examples argument from DagBag.__init__().
+
+Because Astro Runtime 3.3 is built on top of Airflow 3.x, passing include_examples=False to DagBag() causes Python to throw a TypeError.
+How to Fix It
+1. Update tests/test_dag_integrity.py
+
+Remove include_examples=False from your DagBag initializations:
+Python
+
+# Before
+dag_bag = DagBag(dag_folder="dags", include_examples=False)
+
+# After
+dag_bag = DagBag(dag_folder="dags")
+
+In Airflow 3, example DAGs are disabled by default or loaded via configuration (AIRFLOW__CORE__LOAD_EXAMPLES=False), making the parameter unnecessary in the class constructor.
+Secondary Issue: Pytest Cache Permission Warning
+
+Your logs also showed this warning:
+Plaintext
+
+PytestCacheWarning: could not create cache path /.pytest_cache/v/cache/nodeids: [Errno 13] Permission denied: '/pytest-cache-files-7646t09h'
+
+Why it happens
+
+When running inside a container, pytest is trying to write .pytest_cache to the root directory / where the container user doesn't have write permissions.
+Quick Fixes
+
+    Option A: Add -o cache_dir=/tmp/.pytest_cache when running pytest locally:
+    Bash
+
+    astro dev pytest -- -o cache_dir=/tmp/.pytest_cache
+
+    Option B: Update your pyproject.toml file so pytest defaults to a writable directory:
+    Ini, TOML
+
+    [tool.pytest.ini_options]
+    cache_dir = "/tmp/.pytest_cache"
+
+    Summary: Update lines 20 and 28 in tests/test_dag_integrity.py to instantiate DagBag(dag_folder="dags") without include_examples, and your test suite will pass cleanly on Airflow 3 / Astro Runtime 3.x!
+
+**What I kept, changed, or discarded, and why:** removed include examples, ignored cache permession, since its only a warning and wouldnt affect passing or failing
+
+
+
+
+
+
+
+
+
+
+
+second instance> **Prompt I sent:** 
+
+i am creating a runbook # RUNBOOK
+
+
+## How to trigger the DAG manually
+
+astro dev start
+az login
+PGURL
+hover to url then navigate then trigger
+
+## How to run a backfill
+
+
+astro dev run backfill create \
+  --dag-id bader_taxi_pipeline \
+  --from-date 2024-01-01 \
+  --to-date 2024-03-31 \
+  --max-active-runs 1
+
+
+## How to inspect task logs
+
+using url
+
+## Top 3 likely failures and first response
+
+1. forgotten az login to fetch pg config
+2. astro config set postgres.port 5433 on port conflict
+3. forgotten to add PG_URL using astro dev run connections add azure_pg --conn-uri "$PG_URL"
+create runbook.md using information and pictures similiar to https://hub.hackyourfuture.nl/Airflow-Fundamentals-de0ff4a4371946ee9945b61b6f47d3eb
+using only directions to run without learning material. 
+Keep in mind (dag id = bader_taxi_pipeline), (tags: [student:bader](http://c55-data-week-12.localhost:6563/dags?tags=student%3Abader), [taxi](http://c55-data-week-12.localhost:6563/dags?tags=taxi), [week12](http://c55-data-week-12.localhost:6563/dags?tags=week12))
+airflow pipeline name is bader_taxi_pipeline
+populate all steps example: (## How to trigger the DAG manually
+
+astro dev start
+az login
+PGURL
+hover to url then navigate then trigger follow all steps and explain all steps based on URL given from hackyourfuture)
+
+**What the model answered:** # RUNBOOK — bader_taxi_pipeline
+
+**DAG ID:** `bader_taxi_pipeline`
+**Tags:** `student:bader`, `taxi`, `week12`
+**Airflow UI base URL:** `http://c55-data-week-12.localhost:6563`
+
+This document is written so that another student can operate this DAG end‑to‑end
+without reading the Python source. Every step is a command to run or a click
+to make — no background theory.
+
+---
+
+## 0. Prerequisites (one-time setup)
+
+- Astro CLI installed and working (`astro version`)
+- Azure CLI installed (`az version`)
+- Access to the Azure subscription/project this course uses
+- This repo cloned locally, with your terminal `cd`'d into the project folder
+
+---
+
+## 1. How to trigger the DAG manually
+
+1. **Start the local Airflow environment**
+   ```bash
+   astro dev start
+   ```
+   Wait until the terminal reports the webserver and scheduler are up.
+
+2. **Authenticate with Azure**
+   ```bash
+   az login
+   ```
+   This opens a browser window — log in with your course/organization account.
+   Without this step, the next command (fetching the Postgres connection
+   string) will fail with an authentication error.
+
+3. **Fetch the Postgres connection string (`PG_URL`)**
+   ```bash
+   export PG_URL=$(az keyvault secret show \
+     --name pg-connection-string \
+     --vault-name <your-vault-name> \
+     --query value -o tsv)
+   ```
+   > ⚠️ Replace `<your-vault-name>` (and the secret name if different) with
+   > the actual Key Vault used in this course. Ask your instructor/team if
+   > you don't know it — this is the one value in this runbook that depends
+   > on your specific environment.
+
+4. **Register the connection in Airflow**
+   ```bash
+   astro dev run connections add azure_pg --conn-uri "$PG_URL"
+   ```
+   This tells Airflow how to reach the Azure Postgres database under the
+   connection ID `azure_pg`, which the DAG's tasks reference internally.
+
+5. **Open the Airflow UI**
+   Navigate your browser to:
+   ```
+   http://c55-data-week-12.localhost:6563/dags?tags=student%3Abader
+   ```
+   You can also filter by the `taxi` or `week12` tags using the same
+   pattern (`?tags=taxi`, `?tags=week12`), or combine them in the UI's tag
+   filter bar.
+
+6. **Find and trigger the DAG**
+   - Locate `bader_taxi_pipeline` in the DAG list.
+   - Hover over the row — a **Trigger** (▶) button appears on the right side
+     of the row.
+   - Click **Trigger DAG** (the play icon), then confirm in the dialog that
+     pops up (leave config JSON empty unless you specifically need to pass
+     parameters).
+   - The DAG run will appear at the top of the run history with a status of
+     `running`.
+
+---
+
+## 2. How to run a backfill
+
+Use this when you need to (re)process a historical date range instead of
+triggering a single run.
+
+```bash
+astro dev run backfill create \
+  --dag-id bader_taxi_pipeline \
+  --from-date 2024-01-01 \
+  --to-date 2024-03-31 \
+  --max-active-runs 1
+```
+
+Notes:
+- `--from-date` / `--to-date` define the inclusive date range to backfill.
+  Adjust these to the range you actually need.
+- `--max-active-runs 1` forces the backfill to run one date at a time. This
+  is slower but avoids overloading the shared Postgres database — don't
+  raise it unless you know the DB can handle concurrent runs.
+- You still need `az login` and the `azure_pg` connection (steps 2–4 above)
+  set up before running a backfill.
+
+---
+
+## 3. How to inspect task logs
+
+1. Go to the DAG's run history:
+   ```
+   http://c55-data-week-12.localhost:6563/dags?tags=student%3Abader
+   ```
+2. Click on `bader_taxi_pipeline` to open the DAG detail view.
+3. Click on the specific **DAG run** (identified by its execution date/run
+   ID) you want to inspect.
+4. In the Grid or Graph view, click on the individual **task instance**
+   (the colored square/box for that task).
+5. In the panel that opens, click **Logs**. This shows the full stdout/
+   stderr for that task's execution, including tracebacks on failure.
+
+If the UI is unreachable, you can also stream logs directly from the CLI:
+```bash
+astro dev logs --scheduler
+astro dev logs --webserver
+```
+
+---
+
+## 4. Top 3 likely failures and first response
+
+| # | Symptom | First check / fix |
+|---|---------|--------------------|
+| 1 | `astro dev start` fails with `port is already allocated` (Postgres port conflict) | Another local Postgres/container is using the default port. Set a different port and restart: `astro config set postgres.port 5433`, then `astro dev start` again. |
+| 2 | Task fails with `Connection azure_pg not found` / DB connection error | You forgot to register the connection. Re-run steps 3–4 above: fetch `PG_URL` (requires `az login` first), then `astro dev run connections add azure_pg --conn-uri "$PG_URL"`. |
+| 3 | `az`/Azure calls fail with `Please run 'az login'` or a 401/403 error when fetching `PG_URL` | You forgot to authenticate before fetching config. Run `az login`, confirm with `az account show`, then retry step 3. |
+
+---
+
+## 5. Quick reference
 
-**Prompt I sent:** _Replace this section._
+```bash
+# Full manual trigger sequence, start to finish
+astro dev start
+az login
+export PG_URL=$(az keyvault secret show --name pg-connection-string --vault-name <your-vault-name> --query value -o tsv)
+astro dev run connections add azure_pg --conn-uri "$PG_URL"
+# then open http://c55-data-week-12.localhost:6563/dags?tags=student%3Abader and trigger via the UI
+```
 
-**What the model answered:** _Replace this section._
 
-**What I kept, changed, or discarded, and why:** _Replace this section._
+**What I kept, changed, or discarded, and why:**
+edited links and infor and populated runbook.md accordingly after few changes