feat(posts): add blog post on losing our tests migrating to Databricks - #489
Open
AbbadB wants to merge 3 commits into
Open
feat(posts): add blog post on losing our tests migrating to Databricks#489AbbadB wants to merge 3 commits into
AbbadB wants to merge 3 commits into
Conversation
First article from the data engineering chapter: three generations of pipelines, property-based testing, self-discovering CI gates, and AI-assisted development with guardrails. Four diagrams included. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
This pull request is automatically being deployed by Amplify Hosting (learn more). |
…nail Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
First article from the data engineering chapter 🎉
"How we lost our tests migrating to Databricks (and how we're getting them back)" — the story of our three pipeline generations: the well-tested Scala + Airflow stack, the Databricks rebuild that improved the architecture but lost the tests along the way, and the platform we're building now to keep the best of both worlds. Deep dives: property-based testing (a real one-character regex bug months of green CI never caught), CI gates that discover their own scope, and why that verification infrastructure is what makes AI-assisted development safe for a data team.
Content was reviewed by the chapter in a Google Doc round beforehand (validated by our manager; naming/confidentiality pass done: no customer names, no absolute financial figures, ratios only).
Checklist
b_abbadadded to_data/authors.yml)🤖 Generated with Claude Code