Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 24 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,42 +1,42 @@
# Model Promotion Pipeline Example

A fraud detection pipeline that demonstrates three key `obproject` features working together:
A fraud detection pipeline demonstrating CI-driven model promotion across branches.

| Feature | Where | What it does |
|---|---|---|
| `@project_schedule` | TrainFlow | Cron schedule on main only; other branches deploy without a schedule |
| `@project_trigger` | EvaluateFlow | Automatically runs when TrainFlow publishes `training_complete` |
| `promote_assets` | EvaluateFlow | Copies the model asset from a feature branch to main after validation |
| `promote_assets` | CI workflow | Promotes assets from feature branch to main on PR merge |

## Pipeline

```
TrainFlow (scheduled weekday mornings on main)
|
| publishes "training_complete" event
v
EvaluateFlow (triggered by event)
|
| if accuracy >= 0.85 and not on main: promote model to main
| trigger_on_finish
v
ReportFlow (prints latest model summary)
Feature branch:
TrainFlow (manual trigger) → EvaluateFlow (pass/fail gate) → ReportFlow

On PR merge:
CI promotes assets to main with @candidate → teardown feature branch

Main branch:
TrainFlow (weekday 8 AM ET cron) → EvaluateFlow → ReportFlow
```

## Main vs Feature Branches

**On a feature branch:**
- TrainFlow deploys but has no cron (triggered manually)
- EvaluateFlow auto-triggers after training, reports pass/fail
- Models stay on the feature branch until PR merge

**On PR merge:**
- CI runs `promote_assets()` to copy assets to main with `@candidate` alias
- CI tears down the feature branch (flows, assets, metadata)

**On main:**
- TrainFlow runs on a cron (`0 8 * * 1-5` ET)
- EvaluateFlow auto-triggers after training
- No promotion happens (already on main)
- ReportFlow prints the latest model

**On a feature branch:**
- TrainFlow deploys but has no cron (can be triggered manually)
- EvaluateFlow auto-triggers after training
- If the model passes evaluation, it is promoted to main
- ReportFlow prints the latest model on the feature branch

## Local Development

```bash
Expand All @@ -52,7 +52,8 @@ METAFLOW_PROFILE=yellow python flows/report/flow.py run

## Deploying

Push to a GitHub repository with the included workflow. The CI pipeline will:
1. Deploy all three flows to the platform
2. On main, TrainFlow gets its cron schedule
3. On branch deletion/merge, teardown removes branch resources
Push to GitHub. The CI pipeline handles three lifecycle events:

1. **Push** — deploys flows to the platform (main gets cron, feature branches don't)
2. **PR merge** — promotes assets to main with `@candidate`, then tears down the feature branch
3. **Branch delete** — tears down branch resources
24 changes: 8 additions & 16 deletions flows/evaluate/flow.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
"""
Evaluate the latest fraud classifier and promote to main if accurate.
Evaluate the latest fraud classifier.

Triggered automatically by the training_complete event from TrainFlow.
On feature branches, promotes the model to main if accuracy exceeds
the threshold. On main, no promotion is needed.
Evaluates the model against a held-out test set and reports pass/fail.
Promotion to main is handled by CI on PR merge, not by this flow.
"""

from metaflow import step, current
from obproject import ProjectFlow, project_trigger, promote_assets
from obproject import ProjectFlow, project_trigger


ACCURACY_THRESHOLD = 0.85
Expand Down Expand Up @@ -57,21 +57,13 @@ def start(self):

@step
def end(self):
branch = self.prj.write_branch
passed = self.eval_accuracy >= ACCURACY_THRESHOLD
print(f"Threshold={ACCURACY_THRESHOLD} passed={passed} branch={branch}")
print(f"Threshold={ACCURACY_THRESHOLD} passed={passed}")

if passed and branch != "main":
project = self.prj.project
result = promote_assets(
project, source=branch, target="main",
asset="fraud_classifier", kinds=["models"],
)
print(f"Promotion result: {result}")
elif not passed:
print("Model did not meet accuracy threshold; skipping promotion.")
if passed:
print("Model passed evaluation. Will be promoted to main on PR merge.")
else:
print("Already on main; no promotion needed.")
print("Model did not meet accuracy threshold.")


if __name__ == "__main__":
Expand Down
1 change: 1 addition & 0 deletions models/fraud-classifier/asset_config.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ description = "Trained fraud detection model (sklearn)"
[tags]
framework = "sklearn"
domain = "fraud"
status = "baseline"
Loading