[Website] Add blog post on Python UnboundedSource and Watch - #40091
[Website] Add blog post on Python UnboundedSource and Watch#40091Eliaaazzz wants to merge 9 commits into
Conversation
Eliaaazzz
changed the title
Eliaaazzz
changed the title
|
Assigning reviewers: R: @damccorm for label website. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
| email: dhalperi@apache.org | ||
| twitter: | ||
| eliaaazzz: | ||
| name: Elia Liu |
There was a problem hiding this comment.
Are you comfortable including an email? In general, we try to have at least that information so that people can reach out if they have thoughts or questions (admittedly this is rare)
There was a problem hiding this comment.
Happy to include it. Added elialiulzy@gmail.com to the author entry.
| limitations under the License. | ||
| --> | ||
|
|
||
| The Apache Beam Python SDK now includes an `UnboundedSource` API for custom |
There was a problem hiding this comment.
In your intro, I think it would be nice to mention that this is a Google Summer of Code project - I think it both gives useful context and highlights your contribution a little bit better
There was a problem hiding this comment.
Also, it would be nice to address the "why?" here - what problem(s) were you trying to address
There was a problem hiding this comment.
Good idea, thank you. The opening paragraph now says I built both APIs during my Google Summer of Code 2026 project with Beam, mentored by Yi Hu.
There was a problem hiding this comment.
Thank you, Danny. I've updated the PR to explain the problems I was trying to solve.
Name the Google Summer of Code project in the intro and add a motivation section describing the gaps the two APIs fill. Describe what the MatchContinuously refactor changed about its per-file state. List a contact email on the author entry.
|
Thank you for the quick review, Danny, and for bringing in Yi Hu. All three comments are addressed. |
Say what the two APIs let a Python developer build: a source for their own message queue or database change feed, and reusable polling for an input that keeps growing. Credit the splittable DoFn support that already existed, and note that checkpoint finalization is best effort.
An unbounded splittable DoFn could already do this. UnboundedSource exists so a source author does not have to learn restrictions, resumption, and watermark estimators first.
Abacn
left a comment
Abacn
left a comment
There was a problem hiding this comment.
Thanks! This LGTM, just had one comment
|
|
||
| <!--more--> | ||
|
|
||
| This post describes the implementation on Beam's `master` branch as of September |
There was a problem hiding this comment.
blog audiences of blog post are mainly current or provisional Beam users. Consider phrase it as
This post describes the status up-to-date as of Beam 2.77.0
There was a problem hiding this comment.
Makes sense, thanks. Changed it to "This post describes both APIs as of Beam 2.77.0".
Readers track Beam releases, so name 2.77.0 instead of the master branch.
3 participants
Adapt the GSoC final report into a concise blog post about the Python
UnboundedSourceAPI andWatchtransform. Cover their design, runner validation, and local benchmarks, with a link to the full report.Validation: checked against the report and current APIs; Hugo 0.117.0 production build, browser preview, and
git diff --checkpassed.