Skip to content

fix CSCL LDF - #2592

Open
damonmcc wants to merge 4 commits into
mainfrom
dm-cscl-ldf
Open

fix CSCL LDF#2592
damonmcc wants to merge 4 commits into
mainfrom
dm-cscl-ldf

Conversation

@damonmcc

@damonmcc damonmcc commented Aug 31, 2026

Copy link
Copy Markdown
Member

related to #2522
all builds on this branch

  • LDF header release IDs and dates come from recipe.yml. All four were unset, so int__ldf_header raised at compile time. Output now matches prod's published 26b header exactly: 26A / 02092026 / 26B / 04222026.
  • The build loads its own LDF inputs. Previous-release LION, the previous LDF header, and prod's LDF were only ever loaded by hand into the shared production_outputs schema, so nothing kept them there.
  • Citywide prod LION can be loaded again. load has aborted on a deliberate assert False since July: the citywide union reads bronx_lion, but the tables are named bronx_lion_dat. Nine QA models source the table it builds.
  • The LDF is excluded from output file validation. Positions 91-100 carry a cumulative record number that diverges from prod's (CSCL-LDF-03), so a whole-line diff flags every record after the first divergence. qa__ldf_summary already compares positions 1-90. Both validation paths read the exclusion from recipe.yml, so they can't disagree about what gets compared.
  • A missing S3 object reports s3://bucket/key instead of ClientError: 404 HeadObject. object_exists no longer reports a 403 as "not found". Eight tests added.

notes

production_outputs is a single shared schema in db-cscl, and the build now rewrites three of its tables with DROP TABLE ... CASCADE. Two concurrent CSCL builds will race. Giving each build its own copy means splitting the source group, which is more than this change warrants.

@damonmcc damonmcc changed the title Dm cscl ldf fix CSCL LDF Aug 31, 2026
@codecov

codecov Bot commented Aug 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.63%. Comparing base (03cb07f) to head (114f400).

Files with missing lines Patch % Lines
dcpy/utils/s3.py 92.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
Files with missing lines Coverage Δ
dcpy/utils/s3.py 82.27% <92.00%> (+1.82%) ⬆️

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@damonmcc
damonmcc marked this pull request as ready for review August 31, 2026 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant