Let the ingestion write to your own disk while you scope a source - #17
Open
lassebenni wants to merge 1 commit into
Open
Let the ingestion write to your own disk while you scope a source#17lassebenni wants to merge 1 commit into
lassebenni wants to merge 1 commit into
Conversation
…a source Pointing the pipeline at a new API starts with finding out what it returns, and until now the only way to see that was to land it in ADLS, which means a storage account, a container grant and a round trip before you can read a single field. `--local` fetches and validates exactly as a real run does and writes the file to your disk instead. Deliberately not a pipeline stage, and the code and the log line both say so: the SQL warehouse cannot read your laptop, so dbt will never see the file and there is no land-locally-then-upload path. Drop the flag and the same command writes the dev container. Same bytes either way. Both destinations share one _ndjson() helper, including the empty-batch guard, so a shape you settle on locally is the shape dbt gets. A local run also no longer demands STORAGE_ACCOUNT, since it opens no connection to Azure. Verified against the real Arbeitnow API with STORAGE_ACCOUNT unset: 176 records written to local-landing/alex/postings/2026-08-13.json, one JSON object per line. Three tests added, 15 pass.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an opt-in local landing mode for the ingestion pipeline (--local [DIR]) so developers can fetch + validate against a real source and inspect the raw NDJSON on their own disk without needing Azure storage credentials, while keeping the on-disk bytes consistent with the ADLS landing format.
Changes:
- Introduces shared
_ndjson()serialization and a newland_local_json()sink in ingestion storage. - Extends the ingestion pipeline CLI/config to support
--localand to skip requiringSTORAGE_ACCOUNTin that mode. - Adds tests for local landing behavior, plus docs and
.gitignoreentries for local output.
Reviewed changes
Copilot reviewed 4 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| data/src/ingestion/storage.py | Adds _ndjson() helper and land_local_json() for local NDJSON output. |
| data/src/ingestion/pipeline.py | Adds --local [DIR] flag, local-mode config loading, and conditional landing behavior. |
| data/tests/ingestion/test_storage.py | Adds tests asserting local output matches landing format and creates needed directories. |
| data/README.md | Documents local scoping workflow using --local. |
| .gitignore | Ignores local-landing/ outputs (repo root and data/). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+83
to
+87
| payload = _ndjson(records) | ||
|
|
||
| destination = directory / path | ||
| destination.parent.mkdir(parents=True, exist_ok=True) | ||
| destination.write_bytes(payload) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
--local [DIR]to the ingestion job. It fetches and validates exactly as a real run does, then writes the file to your own disk instead of the landing zone.uv run python -m src.ingestion.pipeline --local # writes local-landing/Why
Pointing the pipeline at a new API starts with finding out what it returns. Until now the only way to see that was to land it in ADLS, which means a storage account, a container grant and a round trip before you can read a single field. A trainee scoping a source in week 1 may not have any of that yet, so a local run needs no
STORAGE_ACCOUNTat all.What it is not
Not a pipeline stage, and both the docstring and the log line say so. The SQL warehouse cannot read your laptop, so dbt will never see this file and there is no land-locally-then-upload path. Drop the flag and the same command writes the
devcontainer.Same bytes either way
Both destinations share one
_ndjson()helper, including the empty-batch guard, so a shape you settle on locally is the shape dbt gets.Verified
Run against the real Arbeitnow API with
STORAGE_ACCOUNTunset: 176 records written tolocal-landing/alex/postings/2026-08-13.json, one JSON object per line. Three tests added, 15 pass,ruffclean.🤖 Generated with Claude Code