MohammedA

.hyf/grader_lib.sh

@@ -7,11 +7,9 @@
 # and a set of common static-analysis checks derived from recurring
 # PR review patterns across cohort c55.
 #
-# blocker(): use for leaked-secret findings (a committed profiles.yml/.env,
-# a hardcoded password/connection string). It behaves like fail() for the
-# printed report, but also flips a flag that forces write_score() to report
-# pass=false regardless of the earned point total -- a leaked secret must
-# be fixed before the PR can pass, it cannot be "pointed around."
+# blocker(): use for findings that must fail the PR regardless of points
+# (leaked secrets, missing required evidence like screenshots). Behaves like
+# fail() in the printed report, but forces write_score() to pass=false.
 
 _grader_details=()
 _grader_blocker=false
@@ -38,7 +36,7 @@ write_score() {
   [[ "$score" -ge "$passing" ]] && pass_flag="true"
   if [[ "$_grader_blocker" == true ]]; then
     pass_flag="false"
-    echo "🚫 A blocker was found (leaked secret) -- forcing pass=false regardless of score." >&2
+    echo "🚫 A blocker was found -- forcing pass=false regardless of score." >&2
   fi
   cat > "$outfile" << JSON
 {

.hyf/test.sh

@@ -3,8 +3,9 @@
 # The DAG needs a running Astro/Airflow stack and a live Azure PostgreSQL
 # connection that CI cannot reach, so this checks file presence and code
 # patterns in dags/taxi_pipeline.py and the docs. The actual green run,
-# Screenshot files are presence-checked; content, backfill idempotency, and
-# shared-Airflow deploy are reviewed by a teacher.
+# Screenshot files are required (≥3): missing screenshots force pass=false.
+# Content of those shots, backfill idempotency, and shared-Airflow deploy
+# are still reviewed by a teacher.
 # Total points: 100. Passing score: 60.
 set -euo pipefail
 
@@ -218,9 +219,10 @@ shot_count=$(
 if [[ "$shot_count" -ge 3 ]]; then
   l6=$((l6 + 3)); pass "screenshots: found ${shot_count} image file(s) (need ≥3 for Graph + Grid/run + task log)"
 elif [[ "$shot_count" -gt 0 ]]; then
-  fail "screenshots: only ${shot_count} image file(s) — commit at least 3 (local Graph, green Grid/run, one task log; add shared-UI shot when the VM is up)"
+  # Screenshots are required evidence for teacher review — cannot pass without them.
+  blocker "screenshots: only ${shot_count} image file(s) — commit at least 3 (local Graph, green Grid/run, one task log; add shared-UI shot when the VM is up)"
 else
-  fail "screenshots: none found — commit Graph, Grid/run, and task-log images into the PR (any folder)"
+  blocker "screenshots: none found — commit Graph, Grid/run, and task-log images into the PR (any folder). Screenshots are required; a high code score without them still fails."
 fi
 score=$((score + l6))
 pass "Level 6: documentation + screenshots ($l6/10 pts)"

AI_ASSIST.md

@@ -1,12 +1,75 @@
 # AI assistance log
 
-<!-- Document at least one point where you used an LLM on this assignment.
-     Never paste connection strings, passwords, or real data. Fill in each field. -->
-
 ## Use 1
 
-**Prompt I sent:** _Replace this section._
+**Prompt I sent:** I ran:
+
+`astro dev pytest tests/test_dag_integrity.py --args "-v"`
+
+and both tests failed:
+
+Running your test suite…
+✔ Project image has been updated
+fd6478823d7fccb6939547ace51d22b6bb76e2b49f35db4bb544bcce82e19a85
+Successfully copied 4.3kB (transferred 6.66kB) to astro-pytest:/usr/local/airflow/
+Successfully copied 1.02kB (transferred 3.07kB) to astro-pytest:/usr/local/airflow/
+Successfully copied 8.44kB (transferred 25.1kB) to astro-pytest:/usr/local/airflow/
+Successfully copied 36B (transferred 2.56kB) to astro-pytest:/usr/local/airflow/
+Astro Runtime Version: 3.3-1
+============================= test session starts ==============================
+platform linux -- Python 3.14.6, pytest-9.1.1, pluggy-1.6.0 -- /usr/local/bin/python
+cachedir: .pytest_cache
+rootdir: /
+configfile: pyproject.toml
+plugins: anyio-4.13.0
+collecting ... collected 2 items
+
+tests/test_dag_integrity.py::test_no_import_errors FAILED [ 50%]
+tests/test_dag_integrity.py::test_every_dag_has_tags FAILED [100%]
+
+=================================== FAILURES ===================================
+****\*\*\*\*****\_\_\_\_****\*\*\*\***** test_no_import_errors ******\*\*******\_******\*\*******
+
+    def test_no_import_errors():
+        """Every .py in dags/ must import cleanly."""
+
+>       dag_bag = DagBag(dag_folder="dags", include_examples=False)
+
+                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+E TypeError: DagBag.**init**() got an unexpected keyword argument 'include_examples'
+
+tests/test_dag_integrity.py:20: TypeError
+****\*\*\*\*****\_\_\_****\*\*\*\***** test_every_dag_has_tags ****\*\*\*\*****\_\_\_\_****\*\*\*\*****
+
+    def test_every_dag_has_tags():
+        """Light convention check so DAGs are discoverable via the UI tag filter."""
+
+>       dag_bag = DagBag(dag_folder="dags", include_examples=False)
+
+                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+E TypeError: DagBag.**init**() got an unexpected keyword argument 'include_examples'
+
+tests/test_dag_integrity.py:28: TypeError
+=============================== warnings summary ===============================
+../lib/python3.14/site-packages/\_pytest/cacheprovider.py:469
+/usr/local/lib/python3.14/site-packages/\_pytest/cacheprovider.py:469: PytestCacheWarning: could not create cache path /.pytest_cache/v/cache/nodeids: [Errno 13] Permission denied: '/pytest-cache-files-h40bpd4s'
+config.cache.set("cache/nodeids", sorted(self.cached_nodeids))
+
+../lib/python3.14/site-packages/\_pytest/cacheprovider.py:423
+/usr/local/lib/python3.14/site-packages/\_pytest/cacheprovider.py:423: PytestCacheWarning: could not create cache path /.pytest_cache/v/cache/lastfailed: [Errno 13] Permission denied: '/pytest-cache-files-sr6_svbo'
+config.cache.set("cache/lastfailed", self.lastfailed)
+
+-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
+=========================== short test summary info ============================
+FAILED tests/test_dag_integrity.py::test_no_import_errors - TypeError: DagBag...
+FAILED tests/test_dag_integrity.py::test_every_dag_has_tags - TypeError: DagB...
+======================== 2 failed, 2 warnings in 1.62s =========================
+Error: pytest failed
+
+The same TypeError occurs in both tests. What is causing it, and what is the minimum change needed to make the starter tests compatible without changing my DAG?
 
-**What the model answered:** _Replace this section._
+**What the model answered:** The installed Airflow version no longer accepts `include_examples` in the `DagBag` constructor. It recommended removing `include_examples=False` from both `DagBag` calls in `tests/test_dag_integrity.py`.
 
-**What I kept, changed, or discarded, and why:** _Replace this section._
+**What I kept, changed, or discarded, and why:** I kept the minimum change and removed only the unsupported argument. I did not change the DAG because the failure came from the test API, not the pipeline. I reran the test and confirmed that both integrity tests passed. I ignored the pytest cache warning because it did not cause the tests to fail.

ASSIGNMENT_REPORT.md

@@ -1,31 +1,80 @@
 # Assignment report
 
-<!-- Fill in every section below. Keep it short: a few sentences each. -->
-
 ## Schedule choice and reason
 
-_Replace this section._
+The DAG uses @monthly because the NYC green-taxi source files and ingestion logic are partitioned by month. A daily schedule would unnecessarily process the same monthly file repeatedly. Normal catchup is disabled with catchup=False.
 
 ## Task dependency graph
 
-_Replace: describe ingest -> dbt_run -> dbt_test and why order matters._
+The tasks run in this strict order:
+
+ingest_taxi_month -> dbt_run -> dbt_test
+
+Ingestion must finish before dbt transforms the data, and the models must build successfully before their tests run. A failed task therefore prevents downstream tasks from processing incomplete data.
 
 ## dbt project used
 
-_Replace: your Week 10 project or the class reference?_
+I copied my Week 10 dbt project into include/dbt_project/. Airflow runs it with uvx --python 3.11 so dbt works with the Astro image.
 
 ## One debugging case I resolved
 
-_Replace: what failed, how you found the cause in the logs, and the fix._
+The shared Airflow ingestion task failed with permission denied for schema airflow_mohammedalfakih. I found the cause in the task log and reported the shared connection problem to the teacher. The shared Airflow connection was corrected, and the teacher reran the DAG. ingest_taxi_month, dbt_run, and dbt_test then completed successfully.
+
+## Parameterized execution
+
+Airflow passes {{ ds }} to ingest_taxi_month. The task takes the year and month from this logical date and uses it to select the parquet URL, filter the rows, and delete and replace only that month in raw_trips. It does not use datetime.now().
+
+## Backfill and idempotency
+
+I ran the initial seven-month backfill with:
+
+astro dev run backfill create \
+ --dag-id taxi_pipeline \
+ --from-date 2024-01-01 \
+ --to-date 2024-07-31 \
+ --max-active-runs 1
+
+I recorded these row counts:
+
+2024-01: 56,549
+
+2024-02: 53,571
+
+2024-03: 57,447
+
+2024-04: 56,467
+
+2024-05: 60,994
+
+2024-06: 54,735
+
+2024-07: 51,811
+
+I then reran the same range with:
+
+astro dev run backfill create \
+ --dag-id taxi_pipeline \
+ --from-date 2024-01-01 \
+ --to-date 2024-07-31 \
+ --max-active-runs 1 \
+ --reprocess-behavior completed
 
-## Parameterized runs and backfill
+All seven reruns succeeded. The monthly counts remained identical, proving that rerunning a partition does not create duplicate rows.
 
-_Replace: how {{ ds }} / logical date drives the partition; the exact backfill create command you ran (with --max-active-runs 1)._
+## Shared Airflow deployment
 
-## Idempotency row counts (before / after re-run)
+Merged deployment PR: https://github.com/lassebenni/c55-shared-airflow/pull/5
 
-_Replace: paste monthly counts before the re-run, then after. They must match._
+The deployed DAG is named mohammedalfakih_taxi_pipeline and tagged student:mohammedalfakih.
 
-## Shared Airflow deploy proof (if VM online)
+Evidence:
 
-_Replace: merged c55-shared-airflow PR URL + path to your shared-UI screenshot in this repo._
+- Successful shared run: `screenshots/shared-airflow-green.png`
+- Shared DAG filtered by student tag: `screenshots/shared-airflow-tag-filter.png`
+- Local successful run: `screenshots/airflow-ui.png`
+- DAG dependency graph: `screenshots/dag-graph.png`
+- Task log snippet: `screenshots/ingest-log.png`
+- Backfill evidence: `screenshots/backfill-green.png`
+- Idempotency rerun: `screenshots/backfill-rerun-green.png`
+- Row counts before rerun: `screenshots/idempotency-before.png`
+- Row counts after rerun: `screenshots/idempotency-after.png`

RUNBOOK.md

@@ -1,22 +1,59 @@
 # RUNBOOK
 
-<!-- Fill in every section below. Another student should be able to
-     operate your DAG from this file alone, without reading your Python. -->
-
 ## How to trigger the DAG manually
 
-_Replace this section._
+1. Start the local Astro environment:
+
+```bash
+astro dev start
+```
+
+2. Confirm that `include/dbt_project/profiles.yml` exists. For a fresh clone,
+   copy it from `include/dbt_project/profiles.yml.example`.
+3. Confirm that the `azure_pg` Airflow connection exists.
+4. Open the Airflow UI and unpause `taxi_pipeline`.
+5. Trigger a historical month whose TLC parquet file exists:
+
+```bash
+astro dev run dags trigger taxi_pipeline \
+  --logical-date 2024-08-01T00:00:00+00:00
+```
+
+6. In the Airflow UI, confirm that `ingest_taxi_month`, `dbt_run`, and `dbt_test` finish successfully.
 
 ## How to run a backfill
 
-_Replace this section._
+Run the seven monthly partitions sequentially to prevent concurrent dbt runs:
+
+```bash
+astro dev run backfill create \
+  --dag-id taxi_pipeline \
+  --from-date 2024-01-01 \
+  --to-date 2024-07-31 \
+  --max-active-runs 1
+```
+
+To rerun completed partitions for an idempotency check:
+
+```bash
+astro dev run backfill create \
+  --dag-id taxi_pipeline \
+  --from-date 2024-01-01 \
+  --to-date 2024-07-31 \
+  --max-active-runs 1 \
+  --reprocess-behavior completed
+```
 
 ## How to inspect task logs
 
-_Replace this section._
+1. Open `taxi_pipeline` in the Airflow UI.
+2. Select **Runs** and open the required DAG run.
+3. Select the failed or successful task.
+4. Open the **Logs** tab.
+5. Read the final error message and traceback before retrying or clearing the task.
 
 ## Top 3 likely failures and first response
 
-1. _Replace: symptom, first check, fix_
-2. _Replace this section._
-3. _Replace this section._
+1. **TLC parquet file is unavailable** — `ingest_taxi_month` reports an HTTP error and retries. Check the logical date in the run. Trigger a historical month with an available parquet file instead of a future or unavailable month.
+2. **PostgreSQL connection fails** — tasks report that `azure_pg` is missing or authentication failed. Check the `azure_pg` connection in Airflow, including its host, database, login, port, and `sslmode=require`. Never commit its password.
+3. **dbt run or test fails** — `dbt_run` or `dbt_test` turns red. Inspect its log, confirm that the project exists under `include/dbt_project`, and check `profiles.yml`, source names, and `PG_SCHEMA`. Keep dbt running through the configured `uvx --python 3.11` command.