diff --git a/.gitignore b/.gitignore index d083004..5f1af3c 100644 --- a/.gitignore +++ b/.gitignore @@ -3,6 +3,7 @@ target/ dbt_packages/ logs/ profiles.yml +.user.yml # System files .DS_Store diff --git a/AI_ASSIST.md b/AI_ASSIST.md index 76f6a37..70472bb 100644 --- a/AI_ASSIST.md +++ b/AI_ASSIST.md @@ -4,31 +4,81 @@ Document one place you used an LLM during this assignment. ## The problem - +tried to run jinja syntax inside dbeaver -TODO ## The prompt - +select + pickup_datetime, + dropoff_datetime, + pickup_location_id, + dropoff_location_id, + fare_amount, + tip_amount, + trip_distance, + {{ safe_divide('tip_amount', 'fare_amount') }} as tip_pct +from {{ source('nyc_taxi', 'raw_trips') }} +where pickup_location_id is not null +and fare_amount > 0 +SQL Error [42601]: ERROR: syntax error at or near "{" + Position: 164 + +Error position: line: 9 pos: 163 + -TODO ## The response - +The error you are seeing happens because your SQL client (like DBeaver, pgAdmin, or Datagrip) is trying to execute this query directly against your database, but it doesn't understand the `{{ ... }}` syntax. + +Those double curly braces are **Jinja templates**, which are used by tools like **dbt (Data Build Tool)**. Your database just sees them as a syntax error. + +Here is how to fix this depending on what you are trying to do: + +### Scenario 1: You are working in a dbt project + +If this is part of a dbt project, you cannot run this raw script directly in a standard SQL editor. + +* **The Fix:** Run the model through your terminal using `dbt run` or `dbt compile`. +* If you are using an IDE extension (like dbt Power User in VS Code) or dbt Cloud, use their specific **"Preview"** or **"Compile"** buttons instead of your database client's standard "Execute" button. + +### Scenario 2: You just want to run this quick query in your SQL client + +If you aren't using dbt, or you just want to test this query right now in your SQL tool, you need to replace the Jinja code with standard SQL. + +Here is the clean SQL version of your query: + +```sql +select + pickup_datetime, + dropoff_datetime, + pickup_location_id, + dropoff_location_id, + fare_amount, + tip_amount, + trip_distance, + -- Replaced safe_divide macro with standard SQL handling for division by zero + case + when fare_amount = 0 then 0 + else tip_amount / fare_amount + end as tip_pct +-- Replaced the source() macro with your actual schema and table name +from nyc_taxi.raw_trips +where pickup_location_id is not null +and fare_amount > 0; + +``` + +> **Note:** Since you already have `where fare_amount > 0` at the bottom of your query, a simple `tip_amount / fare_amount` would technically be safe from division-by-zero errors in this specific run, but using the `CASE` statement mimics what `safe_divide` does behind the scenes! + -TODO ## Reflection - +realized i should remove jinja and got the actual dbeaver query from target>compiled after running dbt compile + -TODO --- diff --git a/docs/lineage.png b/docs/lineage.png new file mode 100644 index 0000000..63af118 Binary files /dev/null and b/docs/lineage.png differ diff --git a/macros/safe_divide.sql b/macros/safe_divide.sql index e30c8fa..8695fe0 100644 --- a/macros/safe_divide.sql +++ b/macros/safe_divide.sql @@ -1,10 +1,6 @@ --- safe_divide(numerator, denominator) --- Returns numerator / denominator, or NULL when denominator is 0 or NULL. --- Use for tip_pct = tip_amount / fare_amount and similar ratio columns. - {% macro safe_divide(numerator, denominator) %} - -- TODO: implement the macro body. - -- Use NULLIF(denominator, 0) to avoid division-by-zero errors. - -- Return only the SQL expression (no SELECT, no semicolon). - NULL -{% endmacro %} + case + when {{ denominator }} > 0 then round(({{ numerator }} / {{ denominator }})::numeric, 4) + else null + end +{% endmacro %} \ No newline at end of file diff --git a/models/marts/_fct_daily_borough_stats.yml b/models/marts/_fct_daily_borough_stats.yml index 15bba90..716bc4e 100644 --- a/models/marts/_fct_daily_borough_stats.yml +++ b/models/marts/_fct_daily_borough_stats.yml @@ -2,23 +2,35 @@ version: 2 models: - name: fct_daily_borough_stats - description: > - TODO: state the grain (one row per ___), the source lineage - (built from ___ and ___), and at least one known caveat - (e.g. rows dropped in staging, any WARN-severity tests). + description: "One row per completed NYC green taxi trip in January 2024, with + pickup/dropoff zone attributes folded in (OBT-style mart). Queried + directly by dashboards and ad-hoc analysis. + + **Grain:** one row per trip. + **Source:** `public.raw_trips` joined to `public.raw_zones` on + `pickup_location_id` and `dropoff_location_id`. + **Not included:** trips where `pickup_location_id` is NULL (dropped + in `stg_trips`); duplicate rows from the TLC source are kept as-is + and surfaced by `dbt_utils.unique_combination_of_columns`." # TODO: Task 5 -- add the compound uniqueness test on the mart's primary # key (pickup_borough, pickup_date). You need the dbt_utils package for # this: declare it in packages.yml and run `dbt deps` first. + tests: + - dbt_utils.unique_combination_of_columns: + combination_of_columns: + - pickup_borough + - pickup_date + severity: warn columns: - name: pickup_borough - description: "TODO: explain what this column contains and where it comes from" + description: "TLC borough where the trip started" - name: pickup_date - description: "TODO: explain units and how it is derived" + description: "the day the trip started (date only, no time)" - name: trip_count - description: "TODO: explain what is counted (unit: number of trips)" + description: "total trips that started in this borough on this day" - name: total_fare - description: "TODO: explain units (USD) and what fare_amount represents" + description: "total revenue from trips that started in this borough on this day in USD" - name: avg_tip_pct - description: "TODO: explain the ratio (tip_amount / fare_amount)" + description: "average tip percentage for trips that started in this borough on this day, expressed as a decimal (e.g. 0.15 = 15%)" - name: avg_trip_distance - description: "TODO: explain units (miles, from TLC source data)" + description: "average distance of trips that started in this borough on this day, in miles" diff --git a/models/marts/fct_daily_borough_stats.sql b/models/marts/fct_daily_borough_stats.sql index bf47070..7c56b93 100644 --- a/models/marts/fct_daily_borough_stats.sql +++ b/models/marts/fct_daily_borough_stats.sql @@ -14,25 +14,15 @@ zones AS ( ) SELECT - -- TODO: join trips to zones on pickup_location_id = location_id. - -- Use an INNER JOIN: a few trips have a pickup_location_id with no matching - -- zone (e.g. 999). INNER JOIN drops those so pickup_borough is never NULL and - -- can serve as part of the mart's primary key (your not_null test needs this). - -- TODO: aggregate to grain (pickup_borough, pickup_date) - -- Required output columns: - -- pickup_borough TEXT - z.borough - -- pickup_date DATE - pickup_datetime::date - -- trip_count BIGINT - count(*) - -- total_fare NUMERIC - sum(fare_amount) - -- avg_tip_pct NUMERIC - avg(tip_pct) - -- avg_trip_distance NUMERIC - avg(trip_distance) - NULL AS pickup_borough, - NULL AS pickup_date, - NULL AS trip_count, - NULL AS total_fare, - NULL AS avg_tip_pct, - NULL AS avg_trip_distance + + z.borough::text AS pickup_borough, + t.pickup_datetime::date AS pickup_date, + COUNT(*) AS trip_count, + SUM(t.fare_amount)::numeric(10,2) AS total_fare, + AVG(t.tip_pct)::numeric(10,2) AS avg_tip_pct, + AVG(t.trip_distance)::numeric(10,2) AS avg_trip_distance FROM trips t --- TODO: add JOIN to zones here --- TODO: add GROUP BY here +INNER JOIN zones z + ON t.pickup_location_id = z.location_id +GROUP BY pickup_borough, pickup_date diff --git a/models/staging/_sources.yml b/models/staging/_sources.yml index 4bbee9c..e5bfc3c 100644 --- a/models/staging/_sources.yml +++ b/models/staging/_sources.yml @@ -2,9 +2,9 @@ version: 2 sources: - name: nyc_taxi - schema: nyc_taxi # TODO: confirm this matches the schema where raw_trips and raw_zones live + schema: nyc_taxi tables: - name: raw_trips - description: "TODO: one sentence on what this table contains and its grain" + description: "One row per green taxi trip for January 2024 (~57K rows)." - name: raw_zones - description: "TODO: one sentence on what this table contains" + description: "NYC taxi zone lookup (265 rows mapping location IDs to boroughs)." diff --git a/models/staging/_stg_trips.yml b/models/staging/_stg_trips.yml index 14829a8..5a947ef 100644 --- a/models/staging/_stg_trips.yml +++ b/models/staging/_stg_trips.yml @@ -2,19 +2,29 @@ version: 2 models: - name: stg_trips - description: "TODO: state the grain (one row per ___) and what source this reads from" + description: "Cleaned green taxi trips, one row per trip. This reads from the raw_trips source." columns: - name: pickup_datetime - description: "TODO" - # TODO: Task 5 -- add not_null tests on every column used as a join or - # group-by key. See the chapter's dbt Tests section for the syntax. + description: "when the trip started" + tests: + - not_null - name: pickup_location_id - description: "TODO" + description: "TLC zone id where the trip started" + tests: + - not_null + - relationships: + to: ref('stg_zones') + field: location_id + config: + severity: warn - name: fare_amount - description: "TODO" + description: "cost of the trip in USD" - name: tip_amount - description: "TODO" + description: "tip paid for the trip" + - name: trip_distance - description: "TODO" + description: "distance of the trip in miles" + - name: tip_pct - description: "TODO" + description: "percentage of tip relative to trip cost" + diff --git a/models/staging/_stg_zones.yml b/models/staging/_stg_zones.yml index c4b785d..13c591d 100644 --- a/models/staging/_stg_zones.yml +++ b/models/staging/_stg_zones.yml @@ -2,11 +2,14 @@ version: 2 models: - name: stg_zones - description: "TODO: state the grain and what source this reads from" + description: "One row per TLC taxi zone (265 zones total)" columns: - name: location_id - description: "TODO" - # TODO: Task 5 -- this column is the join key. Which two generic tests - # guarantee a clean one-to-many join from stg_trips? + description: TLC zone ID. + tests: + - unique + - not_null - name: borough - description: "TODO" + description: NYC borough (Manhattan, Brooklyn, Queens, Bronx, Staten Island, EWR, Unknown). + tests: + - not_null \ No newline at end of file diff --git a/models/staging/stg_trips.sql b/models/staging/stg_trips.sql index b5f7b81..b613728 100644 --- a/models/staging/stg_trips.sql +++ b/models/staging/stg_trips.sql @@ -3,11 +3,14 @@ -- Downstream: fct_daily_borough_stats joins this to stg_zones. SELECT - -- TODO: select the columns you need for the mart: - -- pickup_datetime, pickup_location_id, fare_amount, tip_amount, trip_distance - -- - -- TODO: add tip_pct using {{ safe_divide('tip_amount', 'fare_amount') }} - -- - -- TODO: filter out rows where pickup_location_id IS NULL or fare_amount < 0 - + pickup_datetime, + dropoff_datetime, + pickup_location_id, + dropoff_location_id, + fare_amount, + tip_amount, + trip_distance, + {{ safe_divide('tip_amount', 'fare_amount') }} as tip_pct FROM {{ source('nyc_taxi', 'raw_trips') }} +WHERE pickup_location_id IS NOT NULL + AND fare_amount >= 0 \ No newline at end of file diff --git a/models/staging/stg_zones.sql b/models/staging/stg_zones.sql index cc34d23..a44c667 100644 --- a/models/staging/stg_zones.sql +++ b/models/staging/stg_zones.sql @@ -2,6 +2,7 @@ -- Exposes location_id and borough for use as a lookup in the mart. SELECT - -- TODO: select location_id and borough from {{ source('nyc_taxi', 'raw_zones') }} + location_id, + borough FROM {{ source('nyc_taxi', 'raw_zones') }} diff --git a/package-lock.yml b/package-lock.yml new file mode 100644 index 0000000..1ce78fc --- /dev/null +++ b/package-lock.yml @@ -0,0 +1,5 @@ +packages: + - name: dbt_utils + package: dbt-labs/dbt_utils + version: 1.4.1 +sha1_hash: e6424ba9e5a22487e47f023803aa4f0411946808 diff --git a/packages.yml b/packages.yml index bbb2357..71fdb1a 100644 --- a/packages.yml +++ b/packages.yml @@ -1,5 +1,3 @@ -# TODO: Task 5 -- declare the dbt-labs/dbt_utils package here, then run -# `dbt deps` to install it. You need it for the compound uniqueness test -# on the mart. See https://hub.getdbt.com/dbt-labs/dbt_utils/latest/ -# for the package block syntax. -packages: [] +packages: + - package: dbt-labs/dbt_utils + version: [">=1.1.0", "<2.0.0"] \ No newline at end of file diff --git a/profiles.yml.example b/profiles.yml.example index 82c6ae0..f5102c0 100644 --- a/profiles.yml.example +++ b/profiles.yml.example @@ -8,8 +8,9 @@ nyc_taxi_borough_daily: user: "{{ env_var('PG_USER') }}" password: "{{ env_var('PG_PASSWORD') }}" dbname: "{{ env_var('PG_DBNAME', 'postgres') }}" - schema: "dev_" # TODO: replace with your first name (the schema you already own) + schema: "dev_bader" threads: 1 + sslmode: require # Copy this file to profiles.yml (same directory), fill in your name, and ensure # PG_HOST, PG_USER, PG_PASSWORD, and PG_DBNAME are set in your environment. diff --git a/reports/answers.md b/reports/answers.md index 8fb8cf2..dea1e8c 100644 --- a/reports/answers.md +++ b/reports/answers.md @@ -8,11 +8,17 @@ Queries run against `dev_.fct_daily_borough_stats`. ```sql -- TODO: query fct_daily_borough_stats grouped by pickup_borough, sum total_fare, order DESC +select +pickup_borough, +SUM(total_fare) as total_fares +from dev_bader.fct_daily_borough_stats +group by pickup_borough +order by total_fares desc ``` -**Result:** TODO +**Result:** Manhatten 493955.62 -**Interpretation:** TODO (one sentence) +**Interpretation:** highest total revenue per borough is manhatten --- @@ -22,11 +28,18 @@ Queries run against `dev_.fct_daily_borough_stats`. ```sql -- TODO: query fct_daily_borough_stats grouped by pickup_date, sum trip_count, order DESC LIMIT 1 +select +pickup_date, +SUM(trip_count) as total_trips +from dev_bader.fct_daily_borough_stats +group by pickup_date +order by total_trips desc +limit 1 ``` -**Result:** TODO +**Result:** 17-01-2024 -**Interpretation:** TODO (one sentence) +**Interpretation:** highest trips count day is 17-01-2024 --- @@ -35,12 +48,15 @@ Queries run against `dev_.fct_daily_borough_stats`. **SQL:** ```sql --- TODO: query fct_daily_borough_stats order by avg_tip_pct DESC LIMIT 5 +select * +from dev_bader.fct_daily_borough_stats +order by avg_tip_pct desc +limit 5 ``` -**Result:** TODO +**Result:** unknown borough -**Interpretation:** TODO — note whether any avg_tip_pct > 1 rows appear and what causes them +**Interpretation:** shows highest average tip precentages per borough/date. any average tips precentages above 1 do show, because we didnt add where tip_pct > 1 when building mart and we used a warn assert test instead which keeps them in the data --- @@ -50,8 +66,14 @@ Queries run against `dev_.fct_daily_borough_stats`. ```sql -- TODO: use percentile_cont(0.5) WITHIN GROUP (ORDER BY trip_count) filtered by borough +SELECT + pickup_borough, + percentile_cont(0.5) WITHIN GROUP (ORDER BY trip_count) AS median_daily +FROM dev_bader.fct_daily_borough_stats +WHERE pickup_borough IN ('Manhattan', 'Brooklyn') +GROUP BY pickup_borough; ``` -**Result:** TODO +**Result:** 248 for brooklyn and 1169.5 for manhatten -**Interpretation:** TODO (one sentence on the ratio) +**Interpretation:** manhatten has almost 4.7 times the trip count volume as brooklyn diff --git a/tests/assert_avg_tip_pct_within_bounds.sql b/tests/assert_avg_tip_pct_within_bounds.sql index b0fab32..054736d 100644 --- a/tests/assert_avg_tip_pct_within_bounds.sql +++ b/tests/assert_avg_tip_pct_within_bounds.sql @@ -15,5 +15,8 @@ -- TODO: write the SELECT here. -- Query {{ ref('fct_daily_borough_stats') }} and return rows where avg_tip_pct > 1. -SELECT NULL AS pickup_borough, NULL AS pickup_date, NULL AS avg_tip_pct -WHERE FALSE -- TODO: replace with the real query + +{{ config(severity='warn') }} +select pickup_borough, pickup_date, avg_tip_pct +from {{ ref('fct_daily_borough_stats') }} +where avg_tip_pct > 1 \ No newline at end of file