diff --git a/README.md b/README.md index 6522f517..afa7624d 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,14 @@ We have taken these ideas a step further by introducing the concept of the `Mode We aim to provide additional (and optional) tools for workflow orchestration on top of the configuration framework. -[More information is available in our wiki](https://github.com/Point72/ccflow/wiki) +## Documentation + +Our [wiki](https://github.com/Point72/ccflow/wiki) is organized along the four kinds of documentation: + +- **[Tutorials](https://github.com/Point72/ccflow/wiki/Tutorials)** — hands-on lessons, from [First Steps](https://github.com/Point72/ccflow/wiki/First-Steps) through building ETL applications to a [configurable calculator driven from the CLI](https://github.com/Point72/ccflow/wiki/Building-a-Configurable-Calculator). +- **[How-to Guides](https://github.com/Point72/ccflow/wiki/How-to-Guides)** — recipes for specific tasks like configuring complex values, caching, and retries. +- **[Reference](https://github.com/Point72/ccflow/wiki/Reference)** — the core types, built-in models, and options. +- **[Explanation](https://github.com/Point72/ccflow/wiki/Explanation)** — the [design goals](https://github.com/Point72/ccflow/wiki/Design-Goals), [core concepts](https://github.com/Point72/ccflow/wiki/Core-Concepts), and [why ccflow leans on Hydra](https://github.com/Point72/ccflow/wiki/Configuration-and-Hydra). ## Installation diff --git a/ccflow/examples/calculator/__init__.py b/ccflow/examples/calculator/__init__.py new file mode 100644 index 00000000..991a1a61 --- /dev/null +++ b/ccflow/examples/calculator/__init__.py @@ -0,0 +1,25 @@ +from pathlib import Path +from typing import List, Optional + +from ccflow import RootModelRegistry, load_config as load_config_base + +__all__ = ("load_config",) + + +def load_config( + config_dir: str = "", + config_name: str = "", + overrides: Optional[List[str]] = None, + *, + overwrite: bool = True, + basepath: str = "", +) -> RootModelRegistry: + return load_config_base( + root_config_dir=str(Path(__file__).resolve().parent / "config"), + root_config_name="base", + config_dir=config_dir, + config_name=config_name, + overrides=overrides, + overwrite=overwrite, + basepath=basepath, + ) diff --git a/ccflow/examples/calculator/__main__.py b/ccflow/examples/calculator/__main__.py new file mode 100644 index 00000000..19210cfa --- /dev/null +++ b/ccflow/examples/calculator/__main__.py @@ -0,0 +1,33 @@ +import hydra + +from ccflow.utils.hydra import cfg_run + +__all__ = ("main",) + + +@hydra.main(config_path="config", config_name="base", version_base=None) +def main(cfg): + cfg_run(cfg) + + +# Run the default calculator (add) on some input: +# python -m ccflow.examples.calculator +context.values=[1,2,3] +# +# Swap the calculation (function is in the defaults list, so no `+`): +# python -m ccflow.examples.calculator function=scale +context.values=[1,2,3] +# +# Configure the selected function's field: +# python -m ccflow.examples.calculator function=scale function.factor=10 +context.values=[1,2,3] +# python -m ccflow.examples.calculator function=power function.exponent=3 +context.values=[1,2,3] +# +# A composed calculation (round the result of power): +# python -m ccflow.examples.calculator function=rounded function.digits=1 +context.values=[1.5,2.5] +# +# A diamond that reuses a shared `mean` node, deduped by the graph + cache evaluator: +# python -m ccflow.examples.calculator function=tail_ratio +context.values=[1,2,3,10] +# +# Inspect the composed configuration without running it: +# python -m ccflow.examples.calculator.explain function=power + +if __name__ == "__main__": + main() diff --git a/ccflow/examples/calculator/config/__init__.py b/ccflow/examples/calculator/config/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/ccflow/examples/calculator/config/base.yaml b/ccflow/examples/calculator/config/base.yaml new file mode 100644 index 00000000..fe75df25 --- /dev/null +++ b/ccflow/examples/calculator/config/base.yaml @@ -0,0 +1,37 @@ +# @package _global_ + +# Base configuration for the functional calculator example. +# +# The `function` config group selects which calculator runs, and the `output` +# config group selects how the result is emitted (print, log, or write to file). +# +# Run the default calculator, printing the result: +# python -m ccflow.examples.calculator +context.values=[1,2,3] +# Swap the calculator (it is in the defaults list, so no `+`): +# python -m ccflow.examples.calculator function=power +context.values=[1,2,3] +# Configure the selected calculator's field: +# python -m ccflow.examples.calculator function=power function.exponent=3 +context.values=[1,2,3] +# Change how the result is emitted: +# python -m ccflow.examples.calculator output=log +context.values=[1,2,3] +# python -m ccflow.examples.calculator output=write +context.values=[1,2,3] +# A diamond that reuses a shared `mean` node (deduped by the graph + cache evaluator below): +# python -m ccflow.examples.calculator function=tail_ratio +context.values=[1,2,3,10] + +defaults: + - _self_ + - function: add + - output: print + +callable: /output + +# Run every workflow under a graph + cache evaluator, so shared nodes in a +# composed graph (e.g. the `mean` inside `tail_ratio`) are evaluated once. +cli: + model: + _target_: ccflow.FlowOptions + evaluator: + _target_: ccflow.evaluators.MultiEvaluator + evaluators: + - _target_: ccflow.evaluators.GraphEvaluator + - _target_: ccflow.evaluators.MemoryCacheEvaluator + cacheable: true diff --git a/ccflow/examples/calculator/config/function/add.yaml b/ccflow/examples/calculator/config/function/add.yaml new file mode 100644 index 00000000..66881077 --- /dev/null +++ b/ccflow/examples/calculator/config/function/add.yaml @@ -0,0 +1,2 @@ +_target_: ccflow.examples.calculator.functions.add +offset: 0.0 diff --git a/ccflow/examples/calculator/config/function/power.yaml b/ccflow/examples/calculator/config/function/power.yaml new file mode 100644 index 00000000..cdead848 --- /dev/null +++ b/ccflow/examples/calculator/config/function/power.yaml @@ -0,0 +1,2 @@ +_target_: ccflow.examples.calculator.functions.power +exponent: 2.0 diff --git a/ccflow/examples/calculator/config/function/rounded.yaml b/ccflow/examples/calculator/config/function/rounded.yaml new file mode 100644 index 00000000..059ba395 --- /dev/null +++ b/ccflow/examples/calculator/config/function/rounded.yaml @@ -0,0 +1,7 @@ +# Composes two functions: round the result of `power`. +# `value` is a regular parameter fed by an upstream calculator model. +_target_: ccflow.examples.calculator.functions.rounded +digits: 2 +value: + _target_: ccflow.examples.calculator.functions.power + exponent: 2.0 diff --git a/ccflow/examples/calculator/config/function/scale.yaml b/ccflow/examples/calculator/config/function/scale.yaml new file mode 100644 index 00000000..1b35b5c3 --- /dev/null +++ b/ccflow/examples/calculator/config/function/scale.yaml @@ -0,0 +1,2 @@ +_target_: ccflow.examples.calculator.functions.scale +factor: 1.0 diff --git a/ccflow/examples/calculator/config/function/tail_ratio.yaml b/ccflow/examples/calculator/config/function/tail_ratio.yaml new file mode 100644 index 00000000..2b3284bf --- /dev/null +++ b/ccflow/examples/calculator/config/function/tail_ratio.yaml @@ -0,0 +1,12 @@ +# A diamond: tail_ratio needs upper_gap and lower_gap, and both need mean. +# Under the graph + cache evaluator (see base.yaml), the shared mean is +# computed once instead of once per branch. +_target_: ccflow.examples.calculator.functions.tail_ratio +upper: + _target_: ccflow.examples.calculator.functions.upper_gap + center: + _target_: ccflow.examples.calculator.functions.mean +lower: + _target_: ccflow.examples.calculator.functions.lower_gap + center: + _target_: ccflow.examples.calculator.functions.mean diff --git a/ccflow/examples/calculator/config/output/log.yaml b/ccflow/examples/calculator/config/output/log.yaml new file mode 100644 index 00000000..652beaf6 --- /dev/null +++ b/ccflow/examples/calculator/config/output/log.yaml @@ -0,0 +1,6 @@ +_target_: ccflow.PublisherModel +model: /function +publisher: + _target_: ccflow.publishers.LogPublisher + level: info +field: value diff --git a/ccflow/examples/calculator/config/output/print.yaml b/ccflow/examples/calculator/config/output/print.yaml new file mode 100644 index 00000000..35065bf1 --- /dev/null +++ b/ccflow/examples/calculator/config/output/print.yaml @@ -0,0 +1,5 @@ +_target_: ccflow.PublisherModel +model: /function +publisher: + _target_: ccflow.publishers.PrintPublisher +field: value diff --git a/ccflow/examples/calculator/config/output/write.yaml b/ccflow/examples/calculator/config/output/write.yaml new file mode 100644 index 00000000..f6e9957d --- /dev/null +++ b/ccflow/examples/calculator/config/output/write.yaml @@ -0,0 +1,7 @@ +_target_: ccflow.PublisherModel +model: /function +publisher: + _target_: ccflow.publishers.GenericFilePublisher + name: result + suffix: .txt +field: value diff --git a/ccflow/examples/calculator/explain.py b/ccflow/examples/calculator/explain.py new file mode 100644 index 00000000..d3ea35d8 --- /dev/null +++ b/ccflow/examples/calculator/explain.py @@ -0,0 +1,13 @@ +from ccflow.utils.hydra import cfg_explain_cli + +from .__main__ import main + +__all__ = ("explain",) + + +def explain(): + cfg_explain_cli(config_path="config", config_name="base", hydra_main=main) + + +if __name__ == "__main__": + explain() diff --git a/ccflow/examples/calculator/functions.py b/ccflow/examples/calculator/functions.py new file mode 100644 index 00000000..f2d72c0a --- /dev/null +++ b/ccflow/examples/calculator/functions.py @@ -0,0 +1,74 @@ +"""Functional calculator stages built with the ``@Flow.model`` API. + +Each calculator reads its input operands from the runtime context and exposes +its own configuration as ordinary bound parameters. This is the split that makes +the functions interchangeable from configuration: the context carries *what to +compute on*, while the bound fields carry *how to compute*. +""" + +from ccflow import ContextBase, Flow, FromContext + +__all__ = ("Numbers", "add", "scale", "power", "rounded", "mean", "upper_gap", "lower_gap", "tail_ratio") + + +class Numbers(ContextBase): + """Runtime input for the calculators: the operands to compute on.""" + + values: list[float] = [] + + +@Flow.model(context_type=Numbers) +def add(values: FromContext[list[float]], offset: float = 0.0) -> float: + """Sum the input values, then add a configurable offset.""" + return sum(values) + offset + + +@Flow.model(context_type=Numbers) +def scale(values: FromContext[list[float]], factor: float = 1.0) -> float: + """Sum the input values, then multiply by a configurable factor.""" + return sum(values) * factor + + +@Flow.model(context_type=Numbers) +def power(values: FromContext[list[float]], exponent: float = 2.0) -> float: + """Raise each input value to a configurable exponent and sum the results.""" + return sum(value**exponent for value in values) + + +@Flow.model +def rounded(value: float, digits: int = 2) -> float: + """Round an upstream calculator's result to a configurable precision. + + ``value`` is a regular parameter, so it can be bound to a literal or fed by + another calculator model wired in through configuration. + """ + return round(value, digits) + + +@Flow.model(context_type=Numbers) +def mean(values: FromContext[list[float]]) -> float: + """Average of the input values.""" + return sum(values) / len(values) + + +@Flow.model(context_type=Numbers) +def upper_gap(values: FromContext[list[float]], center: float) -> float: + """How far the largest value sits above ``center`` (typically the mean).""" + return max(values) - center + + +@Flow.model(context_type=Numbers) +def lower_gap(values: FromContext[list[float]], center: float) -> float: + """How far the smallest value sits below ``center`` (typically the mean).""" + return center - min(values) + + +@Flow.model +def tail_ratio(upper: float, lower: float) -> float: + """Ratio of the upper gap to the lower gap. + + ``upper`` and ``lower`` are regular parameters fed by ``upper_gap`` and + ``lower_gap``, which both depend on ``mean`` — a diamond whose shared node + the graph evaluator computes once. + """ + return upper / lower diff --git a/ccflow/tests/examples/test_calculator.py b/ccflow/tests/examples/test_calculator.py new file mode 100644 index 00000000..761007b3 --- /dev/null +++ b/ccflow/tests/examples/test_calculator.py @@ -0,0 +1,128 @@ +import sys +from unittest.mock import patch + +import pytest + +from ccflow import Flow, FlowOptionsOverride, FromContext, ModelRegistry +from ccflow.evaluators import GraphEvaluator, MemoryCacheEvaluator, MultiEvaluator +from ccflow.examples.calculator import load_config +from ccflow.examples.calculator.__main__ import main +from ccflow.examples.calculator.explain import explain +from ccflow.examples.calculator.functions import Numbers, add, lower_gap, power, rounded, scale, tail_ratio, upper_gap + + +class TestCalculatorFunctions: + def test_add(self): + model = add(offset=1.0) + assert model(Numbers(values=[1, 2, 3])).value == 7.0 + + def test_scale(self): + model = scale(factor=10.0) + assert model(Numbers(values=[1, 2, 3])).value == 60.0 + + def test_power(self): + model = power(exponent=3.0) + assert model(Numbers(values=[1, 2, 3])).value == 36.0 + + def test_context_from_dict(self): + # cfg_run passes the context as a plain dict; it validates to Numbers. + model = add() + assert model({"values": [1, 2, 3]}).value == 6.0 + + def test_composition_derives_deps(self): + # @Flow.model generates __deps__ from the wiring; the graph evaluator runs it. + model = rounded(value=power(exponent=2.0)) + assert hasattr(model, "__deps__") + assert model.flow.inspect().bound_inputs["value"] is not None + evaluator = MultiEvaluator(evaluators=[GraphEvaluator(), MemoryCacheEvaluator()]) + with FlowOptionsOverride(options={"cacheable": True, "evaluator": evaluator}): + assert model({"values": [1.5, 2.5]}).value == 8.5 + + def test_diamond_dedup(self): + # tail_ratio needs upper_gap and lower_gap, which both need the mean: + # a diamond whose shared node the graph evaluator computes once. + calls = {"mean": 0} + + @Flow.model(context_type=Numbers) + def counting_mean(values: FromContext[list[float]]) -> float: + calls["mean"] += 1 + return sum(values) / len(values) + + center = counting_mean() + model = tail_ratio(upper=upper_gap(center=center), lower=lower_gap(center=center)) + + model({"values": [1, 2, 3, 10]}) + assert calls["mean"] == 2 # once per branch under plain evaluation + + calls["mean"] = 0 + evaluator = MultiEvaluator(evaluators=[GraphEvaluator(), MemoryCacheEvaluator()]) + with FlowOptionsOverride(options={"cacheable": True, "evaluator": evaluator}): + assert model({"values": [1, 2, 3, 10]}).value == 2.0 + assert calls["mean"] == 1 # shared node evaluated once + + +class TestCalculatorConfig: + def _run(self, overrides, values): + try: + registry = load_config(overrides=overrides) + return registry["/function"]({"values": values}).value + finally: + ModelRegistry.root().clear() + + def test_default_function(self): + assert self._run([], [1, 2, 3]) == 6.0 + + def test_swap_function(self): + assert self._run(["function=scale", "function.factor=10"], [1, 2, 3]) == 60.0 + + def test_configure_function(self): + assert self._run(["function=power", "function.exponent=3"], [1, 2, 3]) == 36.0 + + def test_composed_function(self): + # `rounded` wraps `power` (exponent 2) as an upstream model input. + assert self._run(["function=rounded", "function.digits=1"], [1.5, 2.5]) == 8.5 + + def test_diamond_function(self): + # tail_ratio composes a shared-mean diamond entirely in YAML. + assert self._run(["function=tail_ratio"], [1, 2, 3, 10]) == 2.0 + + +class TestCalculatorOutput: + def _run_output(self, overrides, values): + try: + registry = load_config(overrides=overrides) + return registry["/output"]({"values": values}).value + finally: + ModelRegistry.root().clear() + + def test_output_print(self, capsys): + result = self._run_output(["output=print", "function=power", "function.exponent=3"], [1, 2, 3]) + assert result == 36.0 + assert "36.0" in capsys.readouterr().out + + def test_output_log(self): + result = self._run_output(["output=log"], [1, 2, 3]) + assert result == 6.0 + + def test_output_write(self, tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + try: + registry = load_config(overrides=["output=write", "function=scale", "function.factor=10"]) + registry["/output"]({"values": [1, 2, 3]}) + finally: + ModelRegistry.root().clear() + assert (tmp_path / "result.txt").read_text() == "60.0" + + +class TestCalculatorCli: + @pytest.mark.skipif(sys.version_info >= (3, 14), reason="Hydra shell completion help string incompatible with Python 3.14 argparse") + def test_cli(self): + with patch("ccflow.examples.calculator.__main__.cfg_run") as mock_cfg_run: + with patch("sys.argv", ["calculator", "function=power", "+context.values=[1,2,3]"]): + main() + mock_cfg_run.assert_called_once() + + def test_explain(self): + with patch("ccflow.examples.calculator.explain.cfg_explain_cli") as mock_cfg_explain: + explain() + mock_cfg_explain.assert_called_once() diff --git a/docs/wiki/Configuration.md b/docs/wiki/Configuration.md deleted file mode 100644 index 86c9e837..00000000 --- a/docs/wiki/Configuration.md +++ /dev/null @@ -1,989 +0,0 @@ -- [Configuration with `ccflow`](#configuration-with-ccflow) - - [Pydantic for Configuration](#pydantic-for-configuration) - - [Basic Config with BaseModel](#basic-config-with-basemodel) - - [Hierarchical Configs](#hierarchical-configs) - - [Serializing Configs](#serializing-configs) - - [Config Inheritance and Templatization](#config-inheritance-and-templatization) - - [Registering Configurations](#registering-configurations) - - [The Model Registry](#the-model-registry) - - [Dependencies and Dependency Injection](#dependencies-and-dependency-injection) - - [Integration with `hydra`](#integration-with-hydra) - - [Advanced Config Examples](#advanced-config-examples) - - [Custom Validation and Coercion](#custom-validation-and-coercion) - - [Jinja Templates and SQL Queries](#jinja-templates-and-sql-queries) - - [Polars Expressions](#polars-expressions) - - [Numpy Arrays](#numpy-arrays) - - [Arbitrary Types](#arbitrary-types) - - [Loading Objects by Path](#loading-objects-by-path) - - [Using Configuration](#using-configuration) - - [Adding a `__call__` method](#adding-a-__call__-method) - - [Example: Reading a file](#example-reading-a-file) - - [Example: Custom publisher](#example-custom-publisher) - - [Example: Simple data pipeline in polars](#example-simple-data-pipeline-in-polars) - - [Example: Gaussian process regression in sklearn](#example-gaussian-process-regression-in-sklearn) - -# Configuration with `ccflow` - -## Pydantic for Configuration - -Let's dive deeper into some of the ideas from the [First Steps](First-Steps), focusing on how we leverage features of `pydantic` for the purposes of configuration. - -```python -from ccflow import BaseModel, ModelRegistry -from datetime import date -from pathlib import Path -from pprint import pprint -``` - -### Basic Config with BaseModel - -Let's get started with some very simple examples. Pydantic calls their classes "Models", and so we use the same terminology; think of a "Model" as a "Configurable" class. - -The `BaseModel` is our base class for all configuration. We have subclassed Pydantic's `BaseModel` to change some of the default configuration options, and to make the objects play nicer with Hydra and the rest of our framework. However, as they are still Pydantic models, everything you can do with Pydantic's [Models](https://pydantic-docs.helpmanual.io/usage/models/) can be done with these. - -We begin with a dummy example, but one which illustrates how new config schemas and values can be easily defined and manipulated on-the-fly. - -```python -class MyFileConfig(BaseModel): - file: Path - asof: date = date(2024,1,1) - description: str = "" -``` - -This is not very exciting yet, basically just the definition of a config schema, but it already illustrates how Pydantic will conform input data to the right types (i.e. Path, date and str in this case). - -```python -c = MyFileConfig(file="sample.txt") -print(c) -#> MyFileConfig(file=PosixPath('sample.txt'), asof=datetime.date(2024, 1, 1), description='') -``` - -Note that the config object is mutable by default (though they can be frozen too). This makes it easy to change configs, especially once they get nested - -```python -c.description = "Sample description" -print(c) -#> MyFileConfig(file=PosixPath('sample.txt'), asof=datetime.date(2024, 1, 1), description='Sample description') -``` - -Pydantic allows for objects to be created directly from dictionaries - -```python -config = {"file": "sample.txt", "asof": "2024-02-02"} -print(MyFileConfig.model_validate(config)) -#> MyFileConfig(file=PosixPath('sample.txt'), asof=datetime.date(2024, 2, 2), description='') -``` - -Note that Pydantic automatically converted the string path to a `PosixPath` and the string date to a `datetime.date`. - -Pydantic also provides a [JSON schema ](https://json-schema.org/) in standardized format that can users understand the parameters on the config object, though this only works on models that only contain json-compatible types (even though Pydantic supports arbitrary types as we will see later). For example: - -```python -pprint(MyFileConfig.schema()) -#> {'additionalProperties': False, -# 'properties': {'asof': {'default': '2024-01-01', -# 'format': 'date', -# 'title': 'Asof', -# 'type': 'string'}, -# 'description': {'default': '', -# 'title': 'Description', -# 'type': 'string'}, -# 'file': {'format': 'path', 'title': 'File', 'type': 'string'}}, -# 'required': ['file'], -# 'title': 'MyFileConfig', -# 'type': 'object'} -``` - -Pydantic's type validation will catch cases that are incompatible with our schema definition. In fact, Pydantic can be used to place even greater constraints on the values themselves (i.e. asof date must not be in the future) - -```python -try: - c.asof = "foo" -except ValueError as e: - print(e) -#> 1 validation error for MyFileConfig -# asof -# Input should be a valid date or datetime, input is too short [type=date_from_datetime_parsing, input_value='foo', input_type=str] -# For further information visit https://errors.pydantic.dev/2.9/v/date_from_datetime_parsing -``` - -Furthermore, in `ccflow.BaseModel`, we have enabled the option by default to raise exceptions when field names are mis-specified (or extra fields are provided) to catch potential configuration mistakes. - -```python -try: - MyFileConfig(file="sample.txt", AsOf=date(2024,1,1)) -except ValueError as e: - print(e) - -#> 1 validation error for MyFileConfig -# AsOf -# Extra inputs are not permitted [type=extra_forbidden, input_value=datetime.date(2024, 1, 1), input_type=date] -# For further information visit https://errors.pydantic.dev/2.9/v/extra_forbidden -``` - -### Hierarchical Configs - -Hierarchical configs are also easy to work with. Below, we create a new config which consists of two file configs, and we can easily modify nested attributes using standard python syntax. - -```python -class MyTransformConfig(BaseModel): - x: MyFileConfig - y: MyFileConfig = None - param: float = 0. -``` - -The traditional construction of a nested model by object composition would look something -like this - -```python -x = MyFileConfig(file="source1.csv") -y = MyFileConfig(file="source2.csv") -transform = MyTransformConfig(x=x, y=y, param=1.) -print(transform) -#> MyTransformConfig(x=MyFileConfig(file=PosixPath('source1.csv'), asof=datetime.date(2024, 1, 1), description=''), y=MyFileConfig(file=PosixPath('source2.csv'), asof=datetime.date(2024, 1, 1), description=''), param=1.0) -``` - -Note that because of object composition, changing an attribute on the local variable `x` will change it on the object in `transform`: - -```python -x.description="First Source" -print(transform.x.description) -#> 'First Source' -``` - -This becomes important later when we register the config. - -Pydantic also provides the ability to coerce dictionaries recursively into structured types, so long as the types have been declared on the schema. For example, it will automatically create the `MyFileConfig` instance if we just pass a dictionary to `MyTransformConfig`: - -```python -print(MyTransformConfig(x={"file": "source1.csv", "asof": "2024-02-02"})) -#> MyTransformConfig(x=MyFileConfig(file=PosixPath('source1.csv'), asof=datetime.date(2024, 2, 2), description=''), y=None, param=0.0) -``` - -### Serializing Configs - -Pydantic provides a rich set of tools for serializing models, for example - -```python -pprint(transform.model_dump(mode="json")) -#> {'param': 1.0, -# 'type_': '__main__.MyTransformConfig', -# 'x': {'asof': '2024-01-01', -# 'description': 'First Source', -# 'file': 'source1.csv', -# 'type_': '__main__.MyFileConfig'}, -# 'y': {'asof': '2024-01-01', -# 'description': '', -# 'file': 'source2.csv', -# 'type_': '__main__.MyFileConfig'}} -``` - -Note that the custom `BaseModel` in `ccflow` serializes the `type_` of the model in addition to the fields (which does not happen by default in Pydantic). This allows for generic reconstruction of the right config class (or subclass) from the serialized version: - -```python -transform2 = BaseModel.model_validate(transform.model_dump(mode="json")) -assert isinstance(transform2, MyTransformConfig) -``` - -For compatibility with `hydra` (further on in this tutorial), we also alias the `type_` field to `_target_`, i.e. - -```python -pprint(transform.model_dump(mode="json", by_alias=True)) -#> {'param': 1.0, -# '_target_': '__main__.MyTransformConfig', -# 'x': {'asof': '2024-01-01', -# 'description': 'First Source', -# 'file': 'source1.csv', -# '_target_': '__main__.MyFileConfig'}, -# 'y': {'asof': '2024-01-01', -# 'description': '', -# 'file': 'source2.csv', -# '_target_': '__main__.MyFileConfig'}} -``` - -### Config Inheritance and Templatization - -In addition to the composition of configs as illustrated above, there may be cases when inheritance and templatization is required. Pydantic supports both of these out of the box. - -Below we provide an example of multiple inheritance of model objects, which further illustrates the power of having schema classes for configuration over raw dictionaries. - -```python -class DateRangeMixin(BaseModel): - start_date: date - end_date: date - -class RegionMixin(BaseModel): - region: str - -class MyConfig(DateRangeMixin, RegionMixin): - parameter: int - -print(MyConfig(parameter=4, region="US", start_date=date(2022, 1, 1), end_date=date(2023, 1, 1))) -#> MyConfig(region='US', start_date=datetime.date(2022, 1, 1), end_date=datetime.date(2023, 1, 1), parameter=4) -``` - -For examples of templatization, refer to the section of the Pydantic documentation on [Generic Models](https://pydantic-docs.helpmanual.io/usage/models/#generic-models). - -## Registering Configurations - -Next we explain the power of being able to register configurations. - -```python -from ccflow import ModelRegistry -``` - -### The Model Registry - -`ccflow` provides a `ModelRegistry` class which represents a collection of `ccflow.BaseModel` instances (configs). Later we will see how config files can be mapped to a registry, but for now we illustrate how it can be used interactively. - -```python -registry = ModelRegistry(name="My Raw Data") -registry.add( - "source1", - MyFileConfig( - file="source1.csv", description="First" - ), -) -registry.add( - "source2", - MyFileConfig( - file="source2.csv", description="Second" - ), -) -print(registry) -#> ModelRegistry(name='My Raw Data') - -print(list(registry)) -#> ['source1', 'source2'] - -pprint(registry.models) -#> mappingproxy({'source1': MyFileConfig(file=PosixPath('source1.csv'), asof=datetime.date(2024, 1, 1), description='First'), -# 'source2': MyFileConfig(file=PosixPath('source2.csv'), asof=datetime.date(2024, 1, 1), description='Second')}) -``` - -At this point, a `ModelRegistry` just looks and behaves like a dictionary. However, a bit of extra functionality has been built in, such as validation of items that go into the registry to make sure they are config classes, i.e. - -```python -try: - registry.add("bad_data", {"foo": 5, "bar": 6}) -except TypeError as e: - print(e) -#> model must be a child class of , not ''. -``` - -This may seem like an unnecessary restriction, but enforcing that all registry elements are using BaseModel means that we can deliver more powerful functionality over time by extending the BaseModel implementation. - -From any registry, you can access configs directly using `__getitem__` or `get` (which allows for a default value) - -```python -assert "source1" in registry -assert registry["source1"] is registry.get("source1", default=None) -``` - -Note that by default, if you try to add a configuration that already exists, it will raise an error, unless you pass `overwrite=True`: - -```python -try: - registry.add("source1", registry["source1"]) -except ValueError as e: - print(e) -#> Cannot add 'source1' to 'My Raw Data' as it already exists! - -assert registry.add("source1", registry["source1"], overwrite=True) -``` - -As the amount of configuration grows, there is a desire to organize these objects in a hierarchy, and so, the registry class can contain other registries (since they are configuration objects themselves). - -Furthermore, instead of passing various registries around in the code, it is sometimes helpful to have a single registry that is a singleton at the "root" of all these registries. `ccflow` provides this: - -```python -root = ModelRegistry.root() -assert root is ModelRegistry.root() # It is a singleton. -root.add("data", registry, overwrite=True) -print(root) -#> RootModelRegistry() -print(root.models) -#> {'data': ModelRegistry(name='My Raw Data')} -``` - -From the root registry, there are three different ways to get underlying configs, using dictionary syntax, file path or getter syntax: - -```python -root["data"]["source1"] # Dictionary syntax -root["data/source1"] # File path syntax -root.get("data").get("source1") # Getter syntax -``` - -Note that the same object can be registered under multiple different names. If one thinks of a registry as a "catalog" of data or configurations, it makes sense that the same item could be indexed in different ways. - -The registry is a `collections.abc.Mapping` over all the registered models and on any models that belong to sub-registries. In other words, it will return all possible combined keys, i.e.: - -```python -print(list(root)) -#> ['data', 'data/source1', 'data/source2'] -print(len(root)) -#> 3 -``` - -If you only want to access the top level of the registry, use `registry.models`, which in this case returns a single item (the "data" sub-registry): - -```python -print(list(root.models)) -#> ['data'] -``` - -To clear all the entries from a registry, one can execute: - -```python -root.clear() -print(root.models) -#> {} -``` - -### Dependencies and Dependency Injection - -We wish to allow configuration objects to depend on each other in such a way that the linkage is dynamic. In the `ccflow` framework, this is done through object composition (and mutability of configs). However, to make things easier, we allow for configs to be referenced by their name in the **root** registry! For example, we can create a new config object like so: - -```python -root = ModelRegistry.root() -root.add("data", registry, overwrite=True) - -new_config = MyTransformConfig(x="data/source1") -print(new_config) -#> MyTransformConfig(x=MyFileConfig(file=PosixPath('source1.csv'), asof=datetime.date(2024, 1, 1), description='First'), y=None, param=0.0) -``` - -Just like before, if we now change the values on this config model in the registry, they will change in this newly created object as well: - -```python -root["data"]["source1"].description = "Test" -assert new_config.x.description == "Test" -``` - -Note, however, that if we replace the config object in the registry with an entirely new config object, the dependency will still reference the old object. This is why you need to pass `overwrite=True` when adding an object to the registry with a name that already exists. - -With these dependencies set up, we can now register this object as well - -```python -root.add("transform", new_config, overwrite=True) -print(list(root)) -#> ['data', 'data/source1', 'data/source2', 'transform'] -``` - -Even once registered, linkages between objects can be added through simple assignment - -```python -root["transform"].y = "/data/source2" -assert root["transform"].y.file == Path("source2.csv") -``` - -The config objects in `ccflow` can tell you where they are registered (which may be in more than one place), either as a tuple of (registry, name), or as a path by which the object could be accessed. i.e. For the composite configuration `"data config"`, registered in the root registry: - -```python -print(new_config.get_registrations()) -#> [(RootModelRegistry(), 'transform')] -print(new_config.get_registered_names()) -#> ['/transform'] -``` - -Below is an example of that for the file config, as accessed from the transform config: - -```python -print(root["transform"].y.get_registrations()) -#> [(ModelRegistry(name='My Raw Data'), 'source2')] -print(root["transform"].y.get_registered_names()) -#> ['/data/source2'] -``` - -The config objects can also tell you their dependencies on other registered config objects. It will look recursively through the the entire nested configuration structure to find other models that are in the registry (even if some intermediate levels are not registered): - -```python -print(new_config.get_registry_dependencies()) -#> [['/data/source1'], ['/data/source2']] -``` - -When you clear all the models out of the root registry, the registrations and dependencies on previously registered objects are also reset: - -```python -root.clear() -print(new_config.get_registrations()) -#> [] -print(new_config.get_registry_dependencies()) -#> [] -``` - -### Integration with `hydra` - -As shown above, Pydantic provides tools to serialize and deserialize model configurations. The `BaseModel` class in `ccflow` is also compatible with [Hydra](https://hydra.cc/docs/intro/). While hydra integration is optional, it provides powerful tools for working with configuration files and the command line that we can leverage directly. - -```python -from hydra.utils import instantiate -config = { - '_target_': '__main__.MyTransformConfig', - 'param': 1.0, - 'x': {'description': 'First', - 'file': 'source1.csv'}, - 'y': {'file': 'source2.csv'} -} -print(instantiate(config)) -#> MyTransformConfig(x=MyFileConfig(file=PosixPath('source1.csv'), asof=datetime.date(2024, 1, 1), description='First'), y=MyFileConfig(file=PosixPath('source2.csv'), asof=datetime.date(2024, 1, 1), description=''), param=1.0) -``` - -Note that this is the same as calling `BaseModel.model_validate(config)`. - -Furthermore, the `ModelRegistry` class has a convenience method that will take a dictionary of configs, and add them all to the registry. It is even clever enough to interpret nested dictionaries (with no `_target_`) as nested registries, and to resolve dependencies specified as strings (by looking up the string path in the **root** registry). Below is a complete example: - -```python -all_configs = { - "data": { - "source1": { - "_target_": "__main__.MyFileConfig", - "file": "source1.csv", - "description": "First", - }, - "source2": { - "_target_": "__main__.MyFileConfig", - "file": "source2.csv", - "description": "Second", - }, - }, - "transform": { - "_target_": "__main__.MyTransformConfig", - "param": 1.0, - "x": "data/source1", - "y": "data/source2", - }, -} -root = ModelRegistry.root().clear() -root.load_config(all_configs) -print(list(root)) -#> ['data', 'data/source1', 'data/source2', 'transform'] -``` - -In the above, note that the "x" and "y" variables are passed as strings referencing other objects in the registry. When using hydra, something similar could be accomplished by using [variable interpolation](https://omegaconf.readthedocs.io/en/2.3_branch/usage.html#variable-interpolation) in OmegaConf, but there is one key difference. In hydra/omegaconf, variable interpolation makes a copy of the yaml config, so `transform.x` would point to an object that is configured the same way as `data/source1`, but it would not be pointing to the same *instance*. Thus, if changes were made to `data/source1`, they would not be reflected in `transform.x`. - -With the above behavior in place, especially the ability to load dictionaries of configs into the root `ModelRegistry`, the next step is to be able to load the configurations from files. Fortunately, this piece has already been solved by hydra. In particular, it allows for configurations to be split across multiple sub-directories and files, reused in multiple places, and recombined as needed. It also provides advanced command line tools to configure your application. - -While hydra is primarily concerned with loading file-based configurations into command-line applications, their [Compose API](https://hydra.cc/docs/advanced/compose_api/) provides a way to load the configs interactively in a notebook, as a special dictionary type. However, to save people the work of first loading config files into config dictionaries, and then adding those into the registry, we've provided a function to do this directly, shown below. Note that these config files are loading models which are defined in `ccflow` and `ccflow.examples`. - -```python -import ccflow.examples -root = ModelRegistry.root().clear() -absolute_path = Path(ccflow.examples.__file__).parent / "config/conf.yaml" -root.load_config_from_path(path=absolute_path, config_key="registry") -``` - -The "config_key" argument in the function call above points to the subset of the hydra configs to load into the registry, as there may be parts of the config which you do not which to load into the registry (such as configuration of hydra itself, or potentially other global configuration variables that are only meant to exist in the file layer). - -It is out-of-scope for this tutorial to cover the various ways in which hydra can be used to generate configs, but please check out their [documentation](https://hydra.cc/docs/intro/) for more information. - -## Advanced Config Examples - -In this section we cover some more advanced config examples. It can be skipped on the first read if desired. - -### Custom Validation and Coercion - -Pydantic provides a lot of functionality for custom validation. We don't cover all of it in the tutorial, but encourage people to read the section of the Pydantic docs on [validators](https://pydantic-docs.helpmanual.io/usage/validators/). - -One key feature is the ability to coerce non-dictionary objects into Pydantic models, which is very powerful from a configuration perspective. We use this trick in several places in `ccflow` to improve usability, and mention it here in case others find it useful. We illustrate this below: - -```python -from pydantic import model_validator - -class NameConfig(BaseModel): - first_name: str - last_name: str - - @model_validator(mode="wrap") - @classmethod - def coerce_from_string(cls, data, handler): - if isinstance(data, str): - names = data.split(" ") - if len(names) == 2: - data = dict(first_name=names[0], last_name=names[1]) - - return handler(data) - -NameConfig.model_validate("John Doe") -#> NameConfig(first_name='John', last_name='Doe') -``` - -Now, when a string name is passed to a config where `NameConfig` is expected, Pydantic will coerce the data to the correct type. - -```python -class MyConfig(BaseModel): - name: NameConfig - -print(MyConfig(name="John Doe")) -#> MyConfig(name=NameConfig(first_name='John', last_name='Doe')) -``` - -### Jinja Templates and SQL Queries - -Another aspect of configuration that we haven't touched on so far is the need to specify a template document, and then to fill in the data. This occurs commonly when building database queries from parameters or when plugging data into an email template or HTML report. One common solution is to leverage python's string formatting capabilities, but this provides a minimal amount of validation to guard against accidents in the template definition, or malicious users (i.e. SQL injection attacks). The standard solution to this problem is to leverage [Jinja templates](https://jinja.palletsprojects.com/), which are extremely powerful (as they enable some amount of scripting inside the template itself). - -`ccflow` has defined a Pydantic extension type that corresponds to Jinja templates, so that they can be used in configuration objects. We illustrate this below (note that unused template arguments are ignored). - -```python -from ccflow import JinjaTemplate - -class MyTemplateConfig(BaseModel): - greeting: JinjaTemplate - user: str - place: str - -config = MyTemplateConfig( - greeting="Hello {{user|upper}}, welcome to {{place}}!", - user="friend", - place="the tutorial", -) -print(config.greeting) -#> Hello {{user|upper}}, welcome to {{place}}! -print(config.greeting.template.render(config.dict())) -#> Hello FRIEND, welcome to the tutorial! -config.place = "a different configuration" -print(config.greeting.template.render(config.dict())) -#> Hello FRIEND, welcome to a different configuration! -config.greeting = "Dear {{user}}, templates can also be easily switched" -print(config.greeting.template.render(config.dict())) -#> Dear friend, templates can also be easily switched -``` - -While the above example may be useful for a templatized email or report, we provide a more complex and realistic example that illustrates how to easily configure a SQL query: - -```python -from datetime import date -from typing import List - -class MyQueryTemplate(BaseModel): - query: JinjaTemplate - columns: List[str] - where: str = "Test" - query_date: date = date(2024, 1, 1) - filters: List[str] = [] - -query = """select {{columns|join(",\n\t")}} -from MyDatabase -where WhereCol = '{{where}}' - and Date = '{{query_date}}' - {% for filter in filters %}and {{filter}} {% endfor %} -""" - -config = MyQueryTemplate( - query=query, - columns=["Col1", "Col2 as MyOthercol", "SomeID"], -) -print(config.query.template.render(config.dict())) -#> select Col1, -# Col2 as MyOtherCol, -# SomeID -# from MyDatabase -# where WhereCol = 'Test' -# and Date = '2024-01-01' -``` - -Now it's easy to reconfigure the query by, i.e. changing the date and adding filters: - -```python -config.query_date = date(2022, 1, 1) -config.filters = ["SomeID IS NOT NULL", "Col1 in 'blerg'"] -print(config.query.template.render(config.dict())) -#> select Col1, -# Col2 as MyOtherCol, -# SomeID -# from MyDatabase -# where WhereCol = 'Test' -# and Date = '2022-01-01' -# and SomeID IS NOT NULL and Col1 in 'blerg' -``` - -### Polars Expressions - -Instead of configuring SQL queries as shown above, libraries like [Polars](https://docs.pola.rs/) have popularized the use of expressions for defining data transformation. `ccflow` also provides an extension type for polars expressions, so that they can be easily converted from strings (i.e. from a config file): - -```python -from ccflow.exttypes.polars import PolarsExpression -import polars as pl - -class PolarsConfig(BaseModel): - columns: List[PolarsExpression] - where: PolarsExpression - -config = {"columns": ["pl.col('Col1')", "pl.col('Col2').alias('MyOtherCol')", "pl.col('SomeID')"], "where": "pl.col('WhereCol')=='Test'" } -config_model = PolarsConfig.model_validate(config) -assert isinstance(config_model.columns[0], pl.Expr) -assert isinstance(config_model.where, pl.Expr) -``` - -### Numpy Arrays - -Sometimes it is more convenient to work with numpy array objects instead of python lists. `ccflow` provides tools to do this easily, as shown in the following example (which conforms the input data to the declared types automatically). - -```python -from ccflow import NDArray -import numpy as np - -class MyNumpyConfig(BaseModel): - my_array: NDArray[np.float64] - my_list: List[float] - -config = MyNumpyConfig(my_array=[1, 2, 3], my_list=[1, 2, 3]) -assert isinstance(config.my_array, np.ndarray) -assert isinstance(config.my_list, list) -``` - -### Arbitrary Types - -Often the need arises for configuration to create objects that are not built-in types (str, int, float, etc). Pydantic supports a number of additional types (see the [documentation](https://pydantic-docs.helpmanual.io/usage/types/) for a full list), but can also handle completely arbitrary types. We can also use hydra to instantiate these arbitrary types from the configs as well, and Pydantic will validate that the created object is an instance of the desired type. Furthermore, we specify some extension types in `ccflow` that have additional validation and functionality (where `JinjaTemplate`, `PolarsExpression`, and `NDArray` described above are examples). - -First, to illustrate how custom types work, we define our own custom object type, and then a configuration object (i.e. `BaseModel`) that contains that type. To prevent accidental inclusion of custom types, Pydantic must be explicitly told to include them using the `model_config` option. - -```python -from hydra.utils import instantiate - -class MyCustomType: - pass - -class MyConfigWithCustomType(BaseModel): - model_config = {"arbitrary_types_allowed": True} - custom: MyCustomType - -config = { - "_target_": "__main__.MyConfigWithCustomType", - "custom": {"_target_": "__main__.MyCustomType"}, -} -config_model = instantiate(config) -assert isinstance(config_model.custom, MyCustomType) -``` - -### Loading Objects by Path - -Sometimes one needs to refer to an object that is already defined in the codebase by its import name. For example, in some cases, it can be easier easier to construct a config object in python, such as when lots of custom classes, enum types, or lambda functions are involved. For this case, `ccflow` also has a solution that is able to refer to any python object by path, as illustrated below: - -```python -from ccflow import PyObjectPath - -class MyConfigWithPaths(BaseModel): - builtin_func: PyObjectPath = PyObjectPath("builtins.len") - separator: PyObjectPath = PyObjectPath("ccflow.REGISTRY_SEPARATOR") - -config = MyConfigWithPaths() -assert config.builtin_func.object([1, 2, 3]) == 3 -print("Separator: ", config.separator.object) -#> Separator: / -``` - -### Registry Helpers - -ccflow.compose offers small, focused helpers for interacting with the registry and Python-backed defaults: - -- `model_alias(model_name)`: resolve a model instance by string name from the active registry. -- `update_from_template(base, update=None, target_class=None)`: construct a new instance or dict by shallow-copying `base` and applying updates. - - If `base` is a registry alias, you can pass either the alias string directly (e.g., `base: "my_model"`) or use `_target_: ccflow.compose.model_alias` form. - - If `base` is a dict, returns the updated dict. If `target_class` is provided (type or import path), constructs that type with the updated fields. - - Shallow copy ensures nested BaseModel identity is preserved. -- `from_python(py_object_path, indexer=None)`: resolve any Python object by import path. Optionally provide `indexer` (a list of keys) to index into the object in order; no safety checks are performed and indexing errors will propagate. - -## Using Configuration - -With the ability to define arbitrarily complex configuration structures following the examples above, now arises the question of how best to use all that config information. - -One option is simply to access the relevant config information from the root registry whenever it is needed in the code. This method is **fragile** and **strongly discouraged**. It is equivalent to using global variables throughout the code, which has the following problems - -- Causes very tight coupling between parts of the code, and adds dependencies everywhere on, i.e. the naming and structure of the configuration, which may need to change frequently to stay organized -- Since every configuration option is available to every piece of code, it makes it difficult to reason about which code depends on which parts of configuration. One of the original design goals was to make it very easy to remove un-needed configuration classes (i.e. for unused/unsuccessful experimentation) -- Because configurations are mutable (by design), using them everywhere makes it harder to reason about the state of the system (nothing is "pure" or "const correct" any more). It would be better to separate the parts of the code that depend on configuration (and are subject to change) from those that do not (i.e. analytics, pure/idempotent functions, etc) - -### Adding a `__call__` method - -One solution to this problem is to write code that lives as "close" to the relevant configuration as possible, such that the scope is limited to those configuration parameters it needs and the dependencies are more clear. In this way, the registry is not used at all for run-time access, except perhaps as an initial entry-point into the logic. Furthermore, this code can serve as the bridge between the configuration graph (i.e. `ccflow`) and any other computational graphs which will be doing the heavy lifting. For example, the configurations can be used to define tensor graphs (tensorflow, pytorch), event processing graphs (csp, kafka streams), task graphs (ray, dask), etc, which are then executed separately. - -One easy way to bind together user-defined business logic with the configuration classes is simply to add a method on the class. Following the [Single Responsibility Principle](https://en.wikipedia.org/wiki/Single-responsibility_principle), each of these classes should ideally have only one purpose, and hence following the python convention, we can name the primary method that commplishes this `__call__` so that the BaseModel becomes callable. The following examples illustrate this idea. - -### Example: Reading a file - -```python -import pandas as pd - -class MyParquetConfig(BaseModel): - file: Path - description: str = "" - - def __call__(self): - """Read the file as a pandas data frame""" - return pd.read_parquet(self.file) -``` - -```python -config = MyParquetConfig( - file=Path(ccflow.examples.__file__).parent / "example.parquet", description="Example Data" -) -df = config() -df.head() -``` - -Thus, using all the machinery in the previous section, we can define configs that are either pure data containers, or that correspond to some piece of arbitrary user-defined functionality, i.e. a "step" in a workflow. While we do leverage Pydantic for conforming data, there are **no restrictions** on the kind of code that could live inside the `__call__` method (or strictly speaking, how many methods there are or what those methods are called). Here is a slightly more complex example for illustration: - -```python -import pyarrow.parquet as pq -from typing import Literal - -class MyParquetConfig(BaseModel): - """This is an example of a config class.""" - - file: Path - description: str = "" - library: Literal["pandas", "pyarrow"] = "pandas" - - def read_pandas(self): - return pd.read_parquet(self.file) - - def read_arrow(self): - return pq.read_table(self.file) - - def __call__(self): - """Read the file as a pandas data frame or arrow table""" - if self.library == "pandas": - return self.read_pandas() - else: - return self.read_arrow() -``` - -```python -config = MyParquetConfig(file="ccflow/examples/example.parquet", library="pandas") -df = config() -print(type(df)) -#> - -config = MyParquetConfig(file="ccflow/examples/example.parquet", library="pyarrow") -df = config() -print(type(df)) -#> -``` - -### Example: Custom publisher - -A common use case in any research/production framework is to send data from the current process to some other location. In extract-transform-load (ETL) processes, the target location is usually files, an object store or a database. However, automation of research reports containing tables, charts and HTML is another common use case; those may be written to files, sent to an experiment tracking framework or simply emailed to a set of recipients. - -In `ccflow`, we used the principles above to define a very simple interface for "publishers" (`ccflow.publishers.BasePublisher`), which are configurable components that know how to take data and send it somewhere else. The reason for having a `BasePublisher` class is so that publishers can easily be substituted for one another (and validated) as part of the configuration of a larger workflow. If publishers were all implemented a little bit differently, then it would be difficult to switch from, i.e. writing files to sending an email purely based on configuration. - -While `ccflow` provides some implementations out of the box for common use cases (files, email)), custom implementations of the interface are straightforward and encouraged. Below we will create a custom publisher that uses IPython's "display" function to display a list of strings as html. We use Jinja templating as described in a previous section to define the html template. - -```python -from IPython.display import display, HTML -from ccflow import JinjaTemplate, BasePublisher -from typing import List - -class MyPublisher(BasePublisher): - data: List[str] = None - html_template: JinjaTemplate - - def __call__(self): - display(HTML(self.get_name())) - display(HTML(self.html_template.template.render(data="
".join(self.data)))) - -# Create the publisher (i.e. via static configuration) -p = MyPublisher( - name="My {{desc}} publisher:", - html_template="""

{{data}}

""", -) - -# Set the data that we want to publish (i.e. at runtime) -p.name_params = dict(desc="test") -p.data = ["Blue text.", "More blue text."] -p() -``` - -Even though "data" is a standard attribute on BasePublisher, implementations can override it to define more specific data types that the publisher allows, and Pydantic provides the validation. For example, passing a dictionary to "data" in the example above results in an error (before the call to publish is even made): - -```python -p = MyPublisher( - name="My {{desc}} publisher:", - html_template="""

{{data}}

""", -) -try: - p.data = {} -except ValueError as v: - print(v) -``` - -### Example: Simple data pipeline in polars - -In the example below, we use the dataframe library [Polars](https://docs.pola.rs/) along with `ccflow` to configure a very simple data pipeline. For this example to run `polars` must be installed. - -First we define some simple models to read, transform and summarize data - -```python -from ccflow import BaseModel, ModelRegistry -from ccflow.exttypes.polars import PolarsExpression -from pathlib import Path -from typing import List, Optional - -import polars as pl - - -class PolarsCallable(BaseModel): - """A base class for models that return a polars LazyFrame""" - def __call__(self) -> pl.LazyFrame: - ... - -class ParquetDataReader(PolarsCallable): - """A model to read data from a parquet file""" - file: Path - n_rows: Optional[int] = None - row_index_name: Optional[str] = None - # Could expose additional options from the `pl.scan_parquet` interface here - - def __call__(self): - return pl.scan_parquet(self.file, n_rows=self.n_rows, row_index_name=self.row_index_name) - -class FeatureAdder(PolarsCallable): - """This model adds extra columns (features) to the original dataset. - The extra columns are averages over 'average_cols' grouped by 'group_col', and - the absolute difference between the original values and the grouped averages. - """ - data_input: PolarsCallable - group_col: str - average_cols: List[str] - - def __call__(self): - df = self.data_input() - agg_cols = [pl.col(col).mean().alias(f"{col}_mean") for col in self.average_cols] - avg_df = df.group_by(pl.col(self.group_col)).agg(agg_cols) - joined_df = df.join(avg_df, on=self.group_col) - residual_columns = [(pl.col(col) - pl.col(f"{col}_mean")).abs().alias(f"{col}_resid") for col in self.average_cols] - return joined_df.with_columns(residual_columns) - - -class TopKFinder(PolarsCallable): - """This model finds the records corresponding to the max """ - data_input: PolarsCallable - by: str - k: int = 1 - reverse: bool = False - - def __call__(self): - df = self.data_input() - return df.top_k(k=self.k, by=self.by, reverse=self.reverse) - -class ColumnSelector(PolarsCallable): - """This model generically selects column expressions from a source""" - data_input: PolarsCallable - exprs: List[PolarsExpression] - - def __call__(self): - return self.data_input().select(self.exprs) -``` - -With these models, we now register certain configurations in the registry - -```python -root = ModelRegistry.root().clear() -root.add("Raw Data", DataReader(file=Path(ccflow.examples.__file__).parent / "example.parquet")) -root.add("Augmented Data", FeatureAdder(data_input="Raw Data", group_col="State", average_cols=["Sales", "Profit"])) -root.add("TopK Profit residuals", TopKFinder(data_input="Augmented Data", by="Profit_resid", k=3)) -root.add("TopK Sales residuals", TopKFinder(data_input="Augmented Data", by="Sales_resid", k=5)) -root.add("Mean residuals", ColumnSelector(data_input="Augmented Data", exprs=[pl.col("Sales_resid").mean(), pl.col("Profit_resid").mean()])) -``` - -Note that we have registered several different configurations all referencing the same "Augmented Data" component. Two of these are slightly different configurations of `TopKFinder`, but the other is a totally different summary of the data. - -To use these configurations, we can now load a residuals model from the registry and call it. This returns a `polars.LazyFrame`, i.e. we are using `ccflow` to construct the polars graph. We can then call `collect()` on the result to materialize a data frame: - -```python -print(root["TopK Profit residuals"]().collect()) -print(root["Mean residuals"]().collect()) -``` - -Suppose instead we wanted to group by "Region" instead of "State", this can be done by injecting the change directly on "Augmented Data". Note that this change will impact multiple downstream results, as they all depend on the same "Augmented Data" config: - -```python -root["Augmented Data"].group_col = "Region" -print(root["TopK Profit residuals"]().collect()) -print(root["Mean residuals"]().collect()) -``` - -At the moment, "Mean residuals" is taking the mean of the residuals from all the groups output from "Augmented Data". What if we wanted to take the mean of only the top sales residuals? We can reconfigure this easily by pointing the input of "Mean residuals" to "TopK Sales residuals" rather than the original "Augmented Data" - -```python -root["Mean residuals"].data_input = "TopK Sales residuals" -print(root["Mean residuals"]().collect()) -``` - -Suppose that we now need to go back and change the way the data is being read. Often, this might require pointing to a different file or data source, but sometimes it could just be modifying the additional arguments to the additional read call. For example, perhaps we want to change the name of the row index column. We want to be able to do this without changing any code, and especially not by exposing this parameter to any downstream logic (as most data loading interfaces have many different parameters that might need to be configured). Even though this parameter is more than one level down in the call tree, we can inject a new value for it using `ccflow`, and this will still work even if additional processing steps are added in the middle of the workflow: - -```python -root["Raw Data"].row_index_name = "My Index Row" -assert "My Index Row" in root["TopK Profit residuals"]().collect_schema().names() -``` - -### Example: Gaussian process regression in sklearn - -Gaussian Processes (GP) are a generic supervised learning method designed to solve non-linear regression and probabilistic classification problems. The GaussianProcessRegressor in `sklearn` implements Gaussian processes (GP) for regression purposes. For this, the prior of the GP needs to be specified. The prior mean is assumed to be constant and zero (for normalize_y=False) or the training data’s mean (for normalize_y=True). The prior’s covariance is specified by passing a kernel object. - -In the example below, we see how `ccflow` can be used to configure a GP Regression (including the kernel object). Even though the kernel object is not a Pydantic type, we can still configure it in this framework. - -Note that `sklearn` must be installed to run this example. For the meaning of the parameters, refer to the [sklearn documentation](https://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.GaussianProcessRegressor.html#sklearn.gaussian_process.GaussianProcessRegressor); we follow their example for usage of the GP Regressor. - -```python -from ccflow import BaseModel -from sklearn.datasets import make_friedman2 -from sklearn.gaussian_process import GaussianProcessRegressor -from sklearn.gaussian_process.kernels import Kernel -from typing import Callable, Union -from hydra.utils import instantiate - -class GPRegressionModel(BaseModel): - """Wrapping of sklearn's GaussianProcessRegressor for configuration""" - - # Here we tell Pydantic to allow the "Kernel" type, which is not a standard Pydantic type - model_config = {"arbitrary_types_allowed": True} - - kernel: Kernel - alpha: float = 1e-10 - optimizer: Union[str, Callable] = "fmin_l_bfgs_b" - n_restarts_optimizer: int = 0 - normalize_y: bool = False - random_state: int = None - - def __call__(self): - """Build the GP Regressor object""" - # Rather than passing in each attribute individually, we can use the dict representation of the config class (leaving out the "type_" attribute), - # as we named everything consistently with the sklearn parameter names - return GaussianProcessRegressor(**self.model_dump(exclude={"type_"})) - -# Define the config as a dictionary (potentially to live in a config file) -gpr_config = { - "_target_": "__main__.GPRegressionModel", - "kernel": { - "_target_": "sklearn.gaussian_process.kernels.Sum", - "k1": { - "_target_": "sklearn.gaussian_process.kernels.DotProduct", - "sigma_0": 1.0, - }, - "k2": { - "_target_": "sklearn.gaussian_process.kernels.WhiteKernel", - "noise_level": 1.0, - }, - }, - "optimizer": "fmin_l_bfgs_b", - "random_state": 0, -} - -# Load the config in the root registry -root = ModelRegistry.root().clear() -root.load_config({"gpr": gpr_config}) - -# Use the config to train the configured model -X, y = make_friedman2(n_samples=500, noise=0, random_state=0) -gpr = root["gpr"]().fit(X, y) -print(gpr.score(X, y)) - -# We can now change the config interactively to experiment with different kernels, -# without needing to go back to config dictionaries or files -from sklearn.gaussian_process.kernels import RBF, WhiteKernel - -root["gpr"].kernel = RBF() + WhiteKernel() -gpr = root["gpr"]().fit(X, y) -print(gpr.score(X, y)) -``` diff --git a/docs/wiki/ETL.md b/docs/wiki/ETL.md deleted file mode 100644 index 49a971f0..00000000 --- a/docs/wiki/ETL.md +++ /dev/null @@ -1,363 +0,0 @@ -- [Building an ETL System](#building-an-etl-system) - - [Background](#background) - - [Models](#models) - - [Context](#context) - - [Extract](#extract) - - [CLI](#cli) - - [Configuration](#configuration) - - [Remaining Models](#remaining-models) - - [Transform](#transform) - - [Load](#load) - - [Full Configuration](#full-configuration) - - [Running](#running) - - [Visualizing](#visualizing) - - [Appendix - Multiple Files](#appendix---multiple-files) - -# Building an ETL System - -Let's take what we learned in [Workflows](Workflows) and build an end-to-end [ETL](https://en.wikipedia.org/wiki/Extract,_transform,_load) system with `ccflow`. - -The goal is to construct a set of [Callable Models](Workflows#callable-models) into which we can pass [Contexts](Workflows#context) and get back [Results](Workflows#result-type), and to define our workflows via static configuration files using [hydra](https://hydra.cc). - -In order to generically operate on the data that communicated between steps of the workflow, `ccflow` defines a base class for the results that a step returns. -These are also child classes of `BaseModel` so one gets all the type validation and serialization features of pydantic models for free. Insisting on a base class for all result types also allows -the framework to perform additional magic (such as delayed evaluation). - -Some result types are provided with `ccflow`, but it is straightforward to define your own. - -## Background - -Let's start with a simple set of tasks: - -- Extract a website's `html` content and save it -- Given a saved `.html` file, transform it into a `csv` file of link names and corresponding URLs -- Given a saved `.csv` file of names and URLs, load it into a queryable sqlite database - -This is a toy example, but we will use it to put the concepts in [Workflows](Workflows) to practice. -Additionally, we will grow our understanding of [hydra](https://hydra.cc) with a concrete example. - -> [!NOTE] -> Source code is available in-source, in [ccflow/examples/etl](https://github.com/Point72/ccflow/tree/main/ccflow/examples/etl). - -## Models - -We'll define a single [Context](Workflows#context) as the argument to our first model. -This is not strictly necessary, but we'll use it as an example of a context. -In general, contexts are very useful for storing global reference data, like the current date being processed in a backfill pipeline. - -### Context - -```python -from ccflow import ContextBase -from pydantic import Field - -class SiteContext(ContextBase): - site: str = Field(default="https://news.ycombinator.com") -``` - -This class has a single attribute, `site`. -When we finish writing our ETL CLI, we will be able to pass in different contexts at runtime: - -```bash -etl-cli +context=[] # use default hacker news -etl-cli +context=["http://lobste.rs"] # query lobste.rs instead -``` - -### Extract - -For the `extract` stage of our ETL pipeline, let's write a basic model to query a website over REST and return the `HTML` content: - -```python -from typing import Optional -from httpx import Client -from ccflow import CallableModel, Flow, GenericResult, NullContext - - -class RestModel(CallableModel): - @Flow.call - def __call__(self, context: Optional[SiteContext] = None) -> GenericResult[str]: - context = context or SiteContext() - resp = Client().get(context.site, follow_redirects=True) - return GenericResult[str](value=resp.text) -``` - -There are a few key elements here: - -- **CallableModel**: Our class inherits from `CallableModel`, which means it is expected to execute given a context and return a result in a `@Flow.call` decorated `__call__` method -- **SiteContext**\*: Our class will take a specific flavor of context, defined above -- **GenericResult[str]**: Our class returns a `GenericResult`, which, as the name suggests, is just a generic container of a `value` (in this case, a `str`). We could've made this a custom result type as well, allowing greater type safety around our pipeline. - -When we execute our `RestModel`, it will take in a `SiteContext` instance, make an `HTTP` request to the site, and return the string `html` result. - -Before we move on to our other models, let's get this stage working with a CLI. - -## CLI - -`ccflow` provides some helper functions for linking together [hydra](https://hydra.cc) with our Callable models framework. - -```python -import hydra -from ccflow.utils.hydra import cfg_run, cfg_explain_cli - - -@hydra.main(config_path="config", config_name="base", version_base=None) -def main(cfg): - cfg_run(cfg) - -def explain(): - cfg_explain_cli(config_path="config", config_name="base", hydra_main=main) -``` - -The first function here takes a `hydra` configuration hierarchy, and tries to execute the top level `callable` attribute. -When we write the configuration files for our `RestModel`, we'll see how this links together. - -The second function here launches a helpful UI to browse the `hydra` configuration hierarchy we've constructed. - -> [!NOTE] -> These can be run directly from `ccflow`: -> `python -m ccflow.examples.etl` and `python -m ccflow.examples.etl.explain` - -## Configuration - -`hydra` is driven by `yaml` files, so let's write one of these for our ETL examples. - -**ccflow/examples/etl/config/base.yaml**. - -```yaml -extract: - _target_: ccflow.PublisherModel - model: - _target_: ccflow.examples.etl.models.RestModel - publisher: - _target_: ccflow.publishers.GenericFilePublisher - name: raw - suffix: .html - field: value -``` - -Our config here defines a key `extract` as a `PublisherModel`, which is itself a `CallableModel` that will execute the given `model` subkey and generate results using the corresponding `publisher` subkey. - -This isn't strictly necessary for our example, as we could've just written the file ourselves from the `RestModel`, but it's a good example of leveraging the same callable models in multiple places (in this case, a `GenericFilePublisher`). - -We can now execute this with the CLI we created above: - -```bash -python -m ccflow.examples.etl +callable=extract +context=[] -``` - -This will load our configuration and call `/extract` (a `PublisherModel`). -`PublisherModel` will itself call `RestModel`, and then feed the results to `GenericFilePublisher`. -The end result should be the extracted result as a file called `raw.html` containing the `HTML` content (as configured on the \`GenericFilePublisher). - -We can run the same CLI with a different context to extract a different website: - -```bash - python -m ccflow.examples.etl +callable=extract +context=["http://lobste.rs"] -``` - -We can even leverage hydra's [override grammar](https://hydra.cc/docs/advanced/override_grammar/basic/) to make tweaks at any layer of our configuration hierarchy: - -```bash -# Change the GenericFilePublisher's output file name to lobsters, generating lobsters.html instead of raw.html -python -m ccflow.examples.etl +callable=extract +context=["http://lobste.rs"] ++extract.publisher.name=lobsters -``` - -Let's implement the remaining steps of our pipeline as their own models - -## Remaining Models - -### Transform - -Our `LinksModel` will read an `HTML` file, extract all the links therein, and generate a `csv` string. - -```python -from csv import DictWriter -from io import StringIO -from bs4 import BeautifulSoup -from ccflow import CallableModel, Flow, GenericResult, NullContext - -class LinksModel(CallableModel): - file: str - - @Flow.call - def __call__(self, context: NullContext) -> GenericResult[str]: - with open(self.file, "r") as f: - html = f.read() - - # Use beautifulsoup to convert links into csv of name, url - soup = BeautifulSoup(html, "html.parser") - links = [{"name": a.text, "url": href} for a in soup.find_all("a", href=True) if (href := a["href"]).startswith("http")] - - io = StringIO() - writer = DictWriter(io, fieldnames=["name", "url"]) - writer.writeheader() - writer.writerows(links) - output = io.getvalue() - return GenericResult[str](value=output) -``` - -### Load - -Our `DBModel` will read a `csv` file and load it into a `SQLite` database. - -```python -import sqlite3 -from csv import DictReader -from pydantic import Field -from ccflow import CallableModel, Flow, GenericResult, NullContext - - -class DBModel(CallableModel): - file: str - db_file: str = Field(default="etl.db") - table: str = Field(default="links") - - @Flow.call - def __call__(self, context: NullContext) -> GenericResult[str]: - conn = sqlite3.connect(self.db_file) - cursor = conn.cursor() - cursor.execute(f"CREATE TABLE IF NOT EXISTS {self.table} (name TEXT, url TEXT)") - with open(self.file, "r") as f: - reader = DictReader(f) - for row in reader: - cursor.execute(f"INSERT INTO {self.table} (name, url) VALUES (?, ?)", (row["name"], row["url"])) - conn.commit() - return GenericResult[str](value="Data loaded into database") -``` - -## Full Configuration - -Let's register all of our models in the same configuration yaml: - -```yaml -extract: - _target_: ccflow.PublisherModel - model: - _target_: ccflow.examples.etl.models.RestModel - publisher: - _target_: ccflow.publishers.GenericFilePublisher - name: raw - suffix: .html - field: value - -transform: - _target_: ccflow.PublisherModel - model: - _target_: ccflow.examples.etl.models.LinksModel - file: ${extract.publisher.name}${extract.publisher.suffix} - publisher: - _target_: ccflow.publishers.GenericFilePublisher - name: extracted - suffix: .csv - field: value - -load: - _target_: ccflow.examples.etl.models.DBModel - file: ${transform.publisher.name}${transform.publisher.suffix} - db_file: etl.db - table: links -``` - -Our `transform` step will also use a `PublisherModel` to generate a `csv` file. -It will uses's `hydra`'s native support for [OmegaConf](https://omegaconf.readthedocs.io/en/2.3_branch/) interpolation to reference fields in other configuration blocks, e.g. `${extract.publisher.name}`. - -Our `load` does not require a publisher as it will optionally produce a `.db` file directly. - -> [!TIP] -> `hydra` provides a lot of utilities for breaking this config across multiple files and folders, -> providing defaults and overrides, etc. We will see an example of this below. - -## Running - -We've seen how to run our previous step with `+callable=extract`. -Our subsequent steps can be run similarly: - -```bash - -# Transform step: -python -m ccflow.examples.etl +callable=transform +context=[] - -# Transform step with overrides, read input lobsters.html and generate lobsters.csv -python -m ccflow.examples.etl +callable=transform +context=[] ++transform.model.file=lobsters.html ++transform.publisher.name=lobsters - -# Load step: -python -m ccflow.examples.etl +callable=load +context=[] - -# Load step with overrides, read input lobsters.csv and generate in-memory sqlite db -python -m ccflow.examples.etl +callable=load +context=[] ++load.file=lobsters.csv ++load.db_file=":memory:" -``` - -Similarly, we can tweak configurations as well - -## Visualizing - -`hydra` is reading `yaml` files and loading up the `ccflow` models into a single registry, which we can visualize in `JSON` form using the other CLI we wrote above: - -```bash -python -m ccflow.examples.etl.explain -``` - -This can be super helpful to see what fields are set to what values. - - - -We can also combine it with `hydra` overrides to ensure we're configuring everything correctly! - -```bash -python -m ccflow.examples.etl.explain ++extract.publisher.name=test -``` - - - -## Appendix - Multiple Files - -In `hydra`, its convenient to have things broken up across multiple files. -In our example above, we can do this as follows: - -**etl/config/base.yaml** - -```yaml -defaults: - - extract: rest - - transform: links - - load: db -``` - -**etl/config/extract/rest.yaml** - -```yaml -_target_: ccflow.PublisherModel -model: - _target_: ccflow.examples.etl.models.RestModel -publisher: - _target_: ccflow.publishers.GenericFilePublisher - name: raw - suffix: .html -field: value -``` - -**etl/config/transform/links.yaml** - -```yaml -_target_: ccflow.PublisherModel -model: - _target_: ccflow.examples.etl.models.LinksModel - file: ${extract.publisher.name}${extract.publisher.suffix} -publisher: - _target_: ccflow.publishers.GenericFilePublisher - name: extracted - suffix: .csv -field: value -``` - -**etl/config/load/db.yaml** - -```yaml -_target_: ccflow.examples.etl.models.DBModel -file: ${transform.publisher.name}${transform.publisher.suffix} -db_file: etl.db -table: links -``` - -In this organizational scheme, its easy to leveage `hydra` to provide a wide array of separate but interoperable functionality. -`hydra` has [much more documentation on this topic](https://hydra.cc/docs/tutorials/basic/your_first_app/config_groups/). diff --git a/docs/wiki/Key-Features.md b/docs/wiki/Key-Features.md deleted file mode 100644 index b8da5af0..00000000 --- a/docs/wiki/Key-Features.md +++ /dev/null @@ -1,220 +0,0 @@ -- [Base Model](#base-model) -- [Callable Model](#callable-model) -- [Model Registry](#model-registry) -- [Models](#models) -- [Publishers](#publishers) -- [Evaluators](#evaluators) -- [Results](#results) - -`ccflow` (Composable Configuration Flow) is a collection of tools for workflow configuration, orchestration, and dependency injection. -It is intended to be flexible enough to handle diverse use cases, including data retrieval, validation, transformation, and loading (i.e. ETL workflows), model training, microservice configuration, and automated report generation. - -## Base Model - -Central to `ccflow` is the `BaseModel` class. -`BaseModel` is the base class for models in the `ccflow` framework. -A model is basically a data class (class with attributes). -The naming was inspired by the open source library [Pydantic](https://docs.pydantic.dev/latest/) (`BaseModel` actually inherits from the Pydantic base model class). - -## Callable Model - -`CallableModel` is the base class for a special type of `BaseModel` which can be called. -`CallableModel`'s are called with a context (something that derives from `ContextBase`) and returns a result (something that derives from `ResultBase`). -As an example, you may have a `SQLReader` callable model that when called with a `DateRangeContext` returns a `ArrowResult` (wrapper around a Arrow table) with data in the date range defined by the context by querying some SQL database. - -`@Flow.model` is a plain-function API for defining `CallableModel`s with normal Python function signatures. -It keeps the same evaluator, registry, cache, and result machinery while making contextual execution more ergonomic via helpers like `.flow.compute(...)` and `.flow.with_context(...)`. -See [Flow Model](Flow-Model) for the full guide. - -## Model Registry - -A `ModelRegistry` is a named collection of models. -A `ModelRegistry` can be loaded from YAML configuration, which means you can define a collection of models using YAML. -This is really powerful because this gives you a easy way to define a collection of Python objects via configuration. - -## Models - -Although you are free to define your own models (`BaseModel` implementations) to use in your flow graph, -`ccflow` comes with some models that you can use off the shelf to solve common problems. `ccflow` comes with a range of models for reading data. - -The following table summarizes the available models. - -> [!NOTE] -> -> Some models are still in the process of being open sourced. - -## Publishers - -`ccflow` also comes with a range of models for writing data. -These are referred to as publishers. -You can "chain" publishers and callable models using `PublisherModel` to call a `CallableModel` and publish the results in one step. -In fact, `ccflow` comes with several implementations of `PublisherModel` for common publishing use cases. - -The following table summarizes the "publisher" models. - -> [!NOTE] -> -> Some models are still in the process of being open sourced. - -| Name | Path | Description | -| :--------------------------- | :------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `DictTemplateFilePublisher` | `ccflow.publishers` | Publish data to a file after populating a Jinja template. | -| `GenericFilePublisher` | `ccflow.publishers` | Publish data using a generic "dump" Callable. Uses `smart_open` under the hood so that local and cloud paths are supported. | -| `JSONPublisher` | `ccflow.publishers` | Publish data to file in JSON format. | -| `PandasFilePublisher` | `ccflow.publishers` | Publish a pandas data frame to a file using an appropriate method on pd.DataFrame. For large-scale exporting (using parquet), see `PandasParquetPublisher`. | -| `NarwhalsFilePublisher` | `ccflow.publishers` | Publish a narwhals data frame to a file using an appropriate method on nw.DataFrame. | -| `PicklePublisher` | `ccflow.publishers` | Publish data to a pickle file. | -| `PydanticJSONPublisher` | `ccflow.publishers` | Publish a pydantic model to a json file. See [Pydantic modeljson](https://docs.pydantic.dev/latest/concepts/serialization/#modelmodel_dump) | -| `YAMLPublisher` | `ccflow.publishers` | Publish data to file in YAML format. | -| `CompositePublisher` | `ccflow.publishers` | Highly configurable, publisher that decomposes a pydantic BaseModel or a dictionary into pieces and publishes each piece separately. | -| `PrintPublisher` | `ccflow.publishers` | Print data using python standard print. | -| `LogPublisher` | `ccflow.publishers` | Print data using python standard logging. | -| `PrintJSONPublisher` | `ccflow.publishers` | Print data in JSON format. | -| `PrintYAMLPublisher` | `ccflow.publishers` | Print data in YAML format. | -| `PrintPydanticJSONPublisher` | `ccflow.publishers` | Print pydantic model as json. See https://docs.pydantic.dev/latest/concepts/serialization/#modelmodel_dump_json | -| `ArrowDatasetPublisher` | *Coming Soon!* | | -| `PandasDeltaPublisher` | *Coming Soon!* | | -| `EmailPublisher` | *Coming Soon!* | | -| `MatplotlibFilePublisher` | *Coming Soon!* | | -| `MLFlowArtifactPublisher` | *Coming Soon!* | | -| `MLFlowPublisher` | *Coming Soon!* | | -| `PandasParquetPublisher` | *Coming Soon!* | | -| `PlotlyFilePublisher` | *Coming Soon!* | | -| `XArrayPublisher` | *Coming Soon!* | | - -## Evaluators - -`ccflow` comes with "evaluators" that allows you to evaluate (i.e. run) `CallableModel` s in different ways. - -The following table summarizes the "evaluator" models. - -> [!NOTE] -> -> Some models are still in the process of being open sourced. - -| Name | Path | Description | -| :---------------------------------- | :------------------ | :----------------------------------------------------------------------------------------------------------------------------- | -| `LazyEvaluator` | `ccflow.evaluators` | Evaluator that only actually runs the callable once an attribute of the result is queried (by hooking into `__getattribute__`) | -| `LoggingEvaluator` | `ccflow.evaluators` | Evaluator that logs information about evaluating the callable. | -| `MemoryCacheEvaluator` | `ccflow.evaluators` | Evaluator that caches results in memory. | -| `MultiEvaluator` | `ccflow.evaluators` | An evaluator that combines multiple evaluators. | -| `GraphEvaluator` | `ccflow.evaluators` | Evaluator that evaluates the dependency graph of callable models in topologically sorted order. | -| `RetryEvaluator` | `ccflow.evaluators` | Evaluator that retries the evaluation of a callable model on failure, with exponential backoff and jitter. | -| `ChunkedDateRangeEvaluator` | *Coming Soon!* | | -| `ChunkedDateRangeResultsAggregator` | *Coming Soon!* | | -| `RayChunkedDateRangeEvaluator` | *Coming Soon!* | | -| `DependencyTrackingEvaluator` | *Coming Soon!* | | -| `DiskCacheEvaluator` | *Coming Soon!* | | -| `ParquetCacheEvaluator` | *Coming Soon!* | | -| `RayCacheEvaluator` | *Coming Soon!* | | -| `RayGraphEvaluator` | *Coming Soon!* | | -| `RayDelayedDistributedEvaluator` | *Coming Soon!* | | -| `ParquetCacheEvaluator` | *Coming Soon!* | | - -### Retrying on failure - -The `RetryEvaluator` retries a `CallableModel` when evaluation raises an exception. It supports -exponential backoff (`wait_initial`, `wait_multiplier`, `wait_max`), random `wait_jitter` to avoid -thundering-herd retries, and stop conditions via `max_attempts` and `max_delay`. `max_delay` caps -the *cumulative* time spent sleeping between retries (not the total wall-clock runtime of the -evaluation); once the next backoff would push the accumulated wait over the budget, no further -retries are attempted. Use `retry_exceptions` / `no_retry_exceptions` to control which exceptions -are retried. - -```python -from ccflow import FlowOptionsOverride -from ccflow.evaluators import RetryEvaluator - -retry = RetryEvaluator( - max_attempts=5, - wait_initial=0.5, # first retry waits ~0.5s - wait_multiplier=2.0, # then 1s, 2s, 4s, ... - wait_max=30.0, # cap any single wait at 30s - wait_jitter=0.25, # add up to 0.25s of random jitter - max_delay=120.0, # stop retrying once cumulative waiting would exceed 2 minutes - retry_exceptions=[TimeoutError, ConnectionError], -) - -with FlowOptionsOverride(options={"evaluator": retry}): - result = my_model(my_context) -``` - -> [!NOTE] -> -> `retry_exceptions` and `no_retry_exceptions` accept either a bare exception class (as shown above, -> the natural form in Python) or any importable exception path as a string (e.g. -> `"httpx.ConnectError"`, the form to use in YAML/Hydra config files). The paths are validated -> (imported) eagerly, so a third-party exception requires that package to be installed. - -The evaluator is *transparent* (a successful result is identical to evaluating the model directly, -so caching and dependency graphs are unaffected) and holds no mutable per-call state, so a single -instance can be shared safely across threads and combined with parallel evaluators (e.g. a Ray or -Celery evaluator). Place it *inside* a parallel evaluator to retry each task independently, or -*outside* to retry the whole parallel dispatch as a unit. - -#### Choosing which models are retried - -An evaluator wraps *every* model evaluated in its scope (including dependencies). There are a few -ways to control which tasks are retried and which are not, from coarse to fine: - -- **Scope the override to model types or instances.** `FlowOptionsOverride` can target specific - models, so only those get the retry evaluator while everything else uses the default: - - ```python - with FlowOptionsOverride(options={"evaluator": retry}, model_types=(FetchFromApi,)): - result = pipeline(context) # only FetchFromApi instances are retried - ``` - - Use `models=(instance,)` to target a single instance instead of a type. - -- **Override per call.** Pass `_options` for a one-off retry without any surrounding context: - - ```python - result = my_model(context, _options={"evaluator": retry}) - ``` - -- **Pin it on the model.** Set `model.meta.options` so a model always carries its own retry policy. - -#### `RetryModel`: retrying a single model declaratively - -`RetryEvaluator` is the cross-cutting ("how to run") way to add retries: it is configured at -runtime and wraps whatever models fall in its scope. When you instead want a retry policy attached -to one specific model *as part of the graph itself*, use `RetryModel`. It wraps another -`CallableModel` and becomes a first-class node, so the policy can be declared statically in -config/registries and shows up explicitly in serialization and the dependency graph. It shares all -its configuration and retry mechanics with `RetryEvaluator` (both build on the same `RetryPolicy`), -and preserves the wrapped model's `context_type` / `result_type`. - -```python -from ccflow.models import RetryModel - -flaky = RetryModel( - model=fetch_from_api, # any CallableModel - max_attempts=5, - wait_initial=0.5, - retry_exceptions=[TimeoutError, ConnectionError], -) - -result = flaky(my_context) # same context/result types as fetch_from_api -``` - -Use `RetryEvaluator` for runtime, cross-cutting retries (including across parallel evaluators), and -`RetryModel` when the retry policy is a declarative, visible part of the model graph. - -## Results - -A Result is an object that holds the results from a callable model. It provides the equivalent of a strongly typed dictionary where the keys and schema are known upfront. - -The following table summarizes the "result" models. - -| Name | Path | Description | -| :------------------------ | :----------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `GenericResult` | `ccflow.result` | A generic result (holds anything). | -| `DictResult` | `ccflow.result` | A generic dict (key/value) result. | -| `ArrowResult` | `ccflow.result.pyarrow` | Holds an arrow table. | -| `ArrowDateRangeResult` | `ccflow.result.pyarrow` | Extension of `ArrowResult` for representing a table over a date range that can be divided by date, such that generation of any sub-range of dates gives the same results as the original table filtered for those dates. | -| `NarwhalsResult` | `ccflow.result.narwhals` | Holds a narwhals `DataFrame` or `LazyFrame`. | -| `NarwhalsDataFrameResult` | `ccflow.result.narwhals` | Holds a narwhals eager `DataFrame`. | -| `NumpyResult` | `ccflow.result.numpy` | Holds a numpy array. | -| `PandasResult` | `ccflow.result.pandas` | Holds a pandas dataframe. | -| `XArrayResult` | `ccflow.result.xarray` | Holds an xarray. | diff --git a/docs/wiki/Workflows.md b/docs/wiki/Workflows.md deleted file mode 100644 index 6ef92b12..00000000 --- a/docs/wiki/Workflows.md +++ /dev/null @@ -1,670 +0,0 @@ -- [Workflows with `ccflow`](#workflows-with-ccflow) - - [Result Type](#result-type) - - [`GenericResult`](#genericresult) - - [Common Result Types](#common-result-types) - - [Custom Results](#custom-results) - - [Context](#context) - - [`NullContext`](#nullcontext) - - [`GenericContext`](#genericcontext) - - [Common Context Types](#common-context-types) - - [Custom Contexts](#custom-contexts) - - [Properties](#properties) - - [Callable Models](#callable-models) - - [Type Checking](#type-checking) - - [Flow Decorator](#flow-decorator) - - [Metadata](#metadata) - - [Evaluators](#evaluators) - - [In-memory Caching](#in-memory-caching) - - [Graph Evaluation](#graph-evaluation) - - [User Defined Evaluators](#user-defined-evaluators) - -# Workflows with `ccflow` - -As described in the [Workflow Design Goals](Design-Goals#workflow-design-goals), `ccflow` wishes to make it easy to define (via configuration) the tasks/steps that make up a workflow. In the [Using Configuration](Configuration#using-configuration) section and in some of the examples on that page, we described how to bind functionality to the configuration by adding `__call__` methods to the models. - -In `ccflow` we formalize this pattern, by introducing the following - -- a "Result" type to hold the results of workflow steps -- a "Context" type to parameterize workflows, allowing them to be templatized and dynamically configured at runtime -- a "Flow" decorator to allow the framework to inject additional functionality (type checking, logging, caching, etc.) behind the scenes - -## Result Type - -In order to generically operate on the data that communicated between steps of the workflow, `ccflow` defines a base class for the results that a step returns. -These are also child classes of `BaseModel` so one gets all the type validation and serialization features of pydantic models for free. Insisting on a base class for all result types also allows -the framework to perform additional magic (such as delayed evaluation). - -Some result types are provided with `ccflow`, but it is straightforward to define your own. - -### `GenericResult` - -The `GenericResult` can hold anything in the value field. It is useful if the result type is not known upfront, not important, or for ad-hoc experimentation: - -```python -from ccflow import GenericResult -result = GenericResult(value="Anything goes here") -print(result) -#> GenericResult(value='Anything goes here') -``` - -However, it could also hold a dictionary: - -```python -result = GenericResult(value={"x": "foo", "y": 5.0}) -print(result) -#> GenericResult(value={'x': 'foo', 'y': 5.0}) -``` - -The `GenericResult` uses python Generics to provide some amount of type checking if desired on the single `value` field. For example, if you want to ensure that the value is always a string, you can specify this when creating the instance: - -```python -result = GenericResult[str](value="Any string") -print(result) -#> GenericResult(value='Any string') -``` - -Now, if you try to pass a dictionary, it will raise an exception - -```python -try: - result = GenericResult[str](value={"x": "foo", "y": 5.0}) -except ValueError as e: - print(e) -#> 1 validation error for GenericResult[str] -#> value -#> Input should be a valid string [type=string_type, input_value={'x': 'foo', 'y': 5.0}, input_type=dict] -#> For further information visit https://errors.pydantic.dev/2.11/v/string_type -``` - -We also leverage pydantic to reduce the amount of boilerplate needed to instantiate the `GenericResult`. It will automatically validate arbitrary values as needed: - -```python -print(GenericResult.model_validate("Any string")) -#> GenericResult(value='Any string') -``` - -### Common Result Types - -`ccflow` provides some other result types to make it easy to work with common data structures. -These include `PandasResult`, `NDArrayResult`, `ArrowResult`, `XArrayResult` and `NarwhalsFrameResult` (for generic compatibility across dataframe libraries using the excellent [Narwhals](https://narwhals-dev.github.io/narwhals/) library). -Each of these types is designed to hold a specific type of data structure (which will be validated via pydantic), and they all inherit from `ResultBase` to ensure compatibility with the workflow system. - -### Custom Results - -If you have any information about the return types, it makes the code more readable and more robust to define a proper schema by defining your own subclass of `ResultBase`: - -```python -class MyResult(ResultBase): - x: str - y: float - -result = MyResult(x="foo", y=5.0) -print(result) -#> MyResult(x='foo', y=5.0) -``` - -## Context - -The context is a collection of parameters that templetize a workflow, given all the other configuration parameters. `ccflow` provides a base class for all contexts, `ContextBase`. -Since we also wish to integrate with the configuration framework described above, `ContextBase` is a child class of `BaseModel` so that it is configurable and serializable, and furthermore it is -a child class of `ResultBase` so that contexts can be used as the return value from a step in a workflow. This provides a convenient way to introduce dynamism in the workflows. - -Contexts are helpful when a typical usage pattern might be to run many similar workflows where only one or two configuration parameters are different between them (i.e. date, region, product/user id, etc.). -Thus, the writer of a complex workflow with many configuration option can choose which parameters to expose to users of the workflow via the context, while keeping the rest of the parameters hidden in the configuration. -This allows for a more streamlined interface for the more common use cases, while all configuration options can still be modified via the `ccflow` configuration framework. - -### `NullContext` - -Some workflows may not require a context at all, in which case they can use the `NullContext`, which takes no parameters; these workflows are completely defined by their configuration. - -```python -from ccflow import NullContext -print(NullContext()) -#> NullContext() -``` - -Custom pydantic validation can create such a context directly from `None` or from an empty container: - -```python -from ccflow import NullContext -print(NullContext.model_validate(None)) -#> NullContext() -print(NullContext.model_validate([])) -#> NullContext() -``` - -### `GenericContext` - -Similar to the `GenericResult` described above, the `GenericContext` is a flexible context type that can hold any type of value. This is useful when the context is not known upfront or when you want to pass arbitrary data to a workflow step. - -```python -from ccflow import GenericContext -result = GenericContext[str](value="Any string") -print(result) -#> GenericContext(value='Any string') -``` - -Now, if you try to pass a dictionary, it will raise an exception - -```python -try: - result = GenericContext[str](value={"x": "foo", "y": 5.0}) -except ValueError as e: - print(e) -#> 1 validation error for GenericContext[str] -#> value -#> Input should be a valid string [type=string_type, input_value={'x': 'foo', 'y': 5.0}, input_type=dict] -#> For further information visit https://errors.pydantic.dev/2.11/v/string_type -``` - -Once again, we pydantic validation to cut down on boilerplate while also providing type safety: - -```python -print(GenericContext[str].model_validate(100)) -#> GenericContext[str](value='100') -``` - -### Common Context Types - -`ccflow` provides a number of other commonly used contexts for standardization. Below are some examples - -```python -from datetime import date, datetime -from ccflow import ( - DateContext, - DatetimeContext, - DateRangeContext, - UniverseContext, - UniverseDateContext, - VersionedDateContext, - ModelDateContext, -) -print(DateContext(date=date.today())) -print(DatetimeContext(dt=datetime.now())) -print(DateRangeContext(start_date=date(2022, 1, 1), end_date=date(2023, 2, 2))) -print(VersionedDateContext(date=date.today(), entry_time_cutoff=datetime.now(datetime.timezone.utc))) -print(UniverseContext(universe="US")) -print(UniverseDateContext(universe="US", date=date.today())) -print(ModelDateContext(mode="MyModel", date=date.today())) -``` - -Since date-based contexts are particularly popular, `ccflow` also provides additional pydantic validation to make it as easy as possible to construct them from basic types like strings and tuples. This comes in handy when running workflows from the command line. - -```python -from ccflow import DateContext, DateRangeContext -print(DateContext.model_validate("2025-01-01")) -#> DateContext(date=datetime.date(2025, 1, 1)) -print(DateContext.model_validate("0d")) -#> DateContext(date=datetime.date(2025, 4, 4)) -print(DateRangeContext.model_validate(("-7d", "0d"))) -#> DateRangeContext(start_date=datetime.date(2025, 3, 28), end_date=datetime.date(2025, 4, 4)) -``` - -### Custom Contexts - -The author of a workflow will frequently want to define a context which is specific to that workflow, or to a collection of related workflows. It is very simple to define your own custom context: - -```python -from ccflow import ContextBase -from datetime import datetime - -class LocationTimestampContext(ContextBase): - latitude: float - longitude: float - timestamp: datetime - -print(LocationTimestampContext(latitude=40.7128, longitude=-74.0060, timestamp=datetime.now())) -#> LocationTimestampContext(latitude=40.7128, longitude=-74.006, timestamp=datetime.datetime(2025, 4, 5, 12, 47, 43, 303252)) -``` - -### Properties - -Contexts by default are declared as `frozen` in Pydantic; they are immutable, and ideally they should be hashable as well. This is so that more advanced tools (i.e. such as caching) can use contexts as dictionary keys. - -## Callable Models - -With the context and result types defined, we are now ready to define a workflow step/task. -Following the design objectives, and the exposition of how best to use configuration objects, we define a `CallableModel` abstract base class, which is essentially a configurable object where one must implement the `__call__` method as a function of the context. It has a few other features which we will explore below. -Note that since every CallableModel is a BaseModel, this class is configurable using all the configuration logic described in the Configuration Tutorial. In particular any workflow step can be named and added to the registry for later reference. - -First, we illustrate a trivial example of defining a `CallableModel` that implements the "[FizzBuzz](https://en.wikipedia.org/wiki/Fizz_buzz)" programming problem. operating on a `GenericContext` and returning a `GenericResult` - -```python -from ccflow import CallableModel, Flow, GenericResult, GenericContext - -class FizzBuzzModel(CallableModel): - fizz: str = "Fizz" - buzz: str = "Buzz" - - @Flow.call - def __call__(self, context: GenericContext[int]) -> GenericResult[list[int|str]]: - n = context.value - result = [] - for i in range(1, n + 1): - if i % 3 == 0 and i % 5 == 0: - result.append(f"{self.fizz}{self.buzz}") - elif i % 3 == 0: - result.append(self.fizz) - elif i % 5 == 0: - result.append(self.buzz) - else: - result.append(i) - return result - -model = FizzBuzzModel() -print(model(15)) -#> GenericResult[list[Union[int, str]]](value=[1, 2, 'Fizz', 4, 'Buzz', 'Fizz', 7, 8, 'Fizz', 'Buzz', 11, 'Fizz', 13, 14, 'FizzBuzz']) -``` - -One of the first things that jumps out about the implementation above, is the use of the decorator `@Flow.call` (which is required by `CallableModel`). -We will discuss this more later, but the idea is to hide any advanced functionality of the framework regarding evaluation of the steps behind this decorator. -This is what lets us control type validation, logging, caching and other advanced evaluation methods. -The decorator itself is also a configurable object, but by default, it only provides type checking on the input and output types. - -### Type Checking - -In the above example, we could simply pass the value `15` to the `__call__` method as the `context` argument: the decorator invokes pydantic type checking which validates the input and converts it to the appropriate type (`GenericContext[int]` in this case). -Similarly, we are able to return the result list directly, and the decorator will validate this as well, ensuring that the output conforms to the expected type (`GenericResult[list[int|str]]`). -Thus, the `@Flow.call` decorator combines with the pydantic type checking to reduce boilerplate for end users while preserving runtime type safety. - -To further illustrate this point, if we try to pass an invalid input to the model, it will result in an error: - -```python -model = FizzBuzzModel() -try: - model("not an integer") -except ValueError as e: - print(e) -#> 1 validation error for GenericContext[int] -#> value -#> Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='not an integer', input_type=str] -#> For further information visit https://errors.pydantic.dev/2.11/v/int_parsing -``` - -In some cases, leveraging type hints for type checking may be limiting (especially if some underlying logic is not entirely type safe), and so we provide other ways to specify the context and result types. -In particular, there is a `context_type` and `result_type` property on every `CallableModel` that will return the types used for validation. By default, it will infer these types from the type signature of the `__call__` method, but they can also be overridden. -For example: - -```python -model = FizzBuzzModel() -print(model.context_type) -print(model.result_type) -#> -#> -``` - -Suppose that either the context type or the result type depends on the configuration of the object. To make this type-safe from a static type checking point of view, one option would be to use Generics, but this option is overly complicated for many users (and still doesn't allow one to do everything due to python's lack of "higher kinded types"). -A simpler method is to re-implement `context_type` and/or `result_type`. - -```python -from ccflow import CallableModel, Flow, GenericResult, GenericContext -from typing import Type - -class DynamicTypedModel(CallableModel): - input_type: Type - output_type: Type - - @property - def context_type(self): - return GenericContext[self.input_type] - - @property - def result_type(self): - return GenericResult[self.output_type] - - @Flow.call - def __call__(self, context): - # do something with the context - return context.value -``` - -We can now configure this model to take an integer and return a string. Because of the runtime type checking, this conversion will be done automatically for us: - -```python -model = DynamicTypedModel(input_type=int, output_type=str) -print(model(5)) -#> GenericResult[str](value='5') -``` - -If no type information is provided on the model at all, an error will be thrown: - -```python -from ccflow import CallableModel, Flow - -class NoTypeModel(CallableModel): - @Flow.call - def __call__(self, context): - return {"foo": 5} - -model = NoTypeModel() -try: - model() -except TypeError as e: - print(e) -#> Must either define a type annotation for context on __call__ or implement 'context_type' -``` - -### Flow Decorator - -Behind the `@Flow.call` decorator is where all the magic lives, including advanced features which are not yet well tested or have not yet been implemented. However, we walk through some of the more basic functionality to give an idea of how it can be used. - -The behavior of the `@Flow.call` decorator can be controlled in several ways: - -- by passing arguments to it when defining the CallableModel to customize model-specific behavior -- by using the FlowOptionsOverride context (which allows you to scope changes to specific models, specific model types or all models) -- by setting `options` in the `meta` attribute of the CallableModel -- by passing `_options` directly to the `__call__` method - -For hand-written `CallableModel` classes, `@Flow.call(auto_context=True)` is -also available when the `__call__` method should declare context fields as -keyword-only parameters instead of accepting one explicit context object. This -is an opt-in `Flow.call` feature; it does not add the dependency wiring or -`FromContext[...]` semantics provided by `@Flow.model`. - -An example of the first one (model-specific options) is to disable validation of the result type on a particular model - -```python -from ccflow import CallableModel, Flow, GenericResult, GenericContext - -class NoValidationModel(CallableModel): - @Flow.call(validate_result=False) - def __call__(self, context: GenericContext[str]) -> GenericResult[float]: - return "foo" - -model = NoValidationModel() -print(model("foo")) -#> foo -``` - -An example of the latter three is to change the log level for all model evaluations (including sub-models): - -```python -import logging - -from ccflow import FlowOptionsOverride - -logger = logging.getLogger() -logger.setLevel(logging.WARN) - -model = FizzBuzzModel() -with FlowOptionsOverride(options={"log_level": logging.WARN}): - print(model(15)) - -#[FizzBuzzModel]: Start evaluation of __call__ on GenericContext[int](value=15). -#[FizzBuzzModel]: FizzBuzzModel(meta=MetaData(name=''), fizz='Fizz', buzz='Buzz') -#[FizzBuzzModel]: End evaluation of __call__ on GenericContext[int](value=15) (time elapsed: 0:00:00.000035). -#> GenericResult[list[Union[int, str]]](value=[1, 2, 'Fizz', 4, 'Buzz', 'Fizz', 7, 8, 'Fizz', 'Buzz', 11, 'Fizz', 13, 14, 'FizzBuzz']) -``` - -If there is only a need to set the options for a specific model (and not any sub-models that it may call), then the `meta` attribute or passing `_options` to the `__call__` method are better options. -Setting `meta.options` is particularly useful when the model is being loaded from a configuration file, though it can be done interactively as well to persist the options with the model instance. - -```python -model = FizzBuzzModel() -model.meta.options = {"log_level": logging.WARN} -_ = model(15) -#[FizzBuzzModel]: Start evaluation of __call__ on GenericContext[int](value=15). -#[FizzBuzzModel]: FizzBuzzModel(meta=MetaData(name=''), fizz='Fizz', buzz='Buzz') -#[FizzBuzzModel]: End evaluation of __call__ on GenericContext[int](value=15) (time elapsed: 0:00:00.000074). -``` - -Most convenient for interactive work is to pass the options to the `__call__` method directly. - -```python -model = FizzBuzzModel() -_ = model(15, _options={"log_level": logging.WARN}) -#[FizzBuzzModel]: Start evaluation of __call__ on GenericContext[int](value=15). -#[FizzBuzzModel]: FizzBuzzModel(meta=MetaData(name=''), fizz='Fizz', buzz='Buzz') -#[FizzBuzzModel]: End evaluation of __call__ on GenericContext[int](value=15) (time elapsed: 0:00:00.000072). -``` - -To see a list of all the available options, you can look at the schema definition of the FlowOptions class: - -```python -from ccflow import FlowOptions -print(FlowOptions.model_fields) -#> {'log_level': FieldInfo(annotation=int, required=False, default=10, description="If no 'evaluator' is set, will use a LoggingEvaluator with this log level"), -#> 'verbose': FieldInfo(annotation=bool, required=False, default=True, description='Whether to use verbose logging'), -#> 'validate_result': FieldInfo(annotation=bool, required=False, default=True, description="Whether to validate the result to the model's result_type before returning"), -#> 'volatile': FieldInfo(annotation=bool, required=False, default=False, description='Whether this function is volatile (i.e. always returns a different value), and hence should always be excluded from caching'), -#> 'cacheable': FieldInfo(annotation=bool, required=False, default=False, description='Whether the model results should be cached if possible. This is False by default so that caching is opt-in'), -#> 'evaluator': FieldInfo(annotation=Union[Annotated[EvaluatorBase, InstanceOf], NoneType], required=False, default=None, description='A hook to set a custom evaluator')} -``` - -### Metadata - -For convenience (and consistency), the `CallableModel` base class comes with a `meta` attribute that stores metadata about the step as represented by the `MetaData` class. -Usage of this is optional, and the `name` and `description` fields are mostly placeholders. At the moment - -- The `name` field is used by the logging evaluator to identify the model in the logs (and will be automatically set when models are loaded into the `ModelRegistry` from `hydra` configs). -- The `description` is helpful when building a catalog of callable models from the `ModelRegistry` - -As the number and size of workflows grows, it is useful to set these. especially for users that are trying to understand large workflows that other people have configured. - -The `MetaData` class also has a field called `options` which is another way that `FlowOptions` can be set (as an alternative to the `FlowOptionsOverride` context manager, or setting them directly in the `@Flow.call` decorator). -The advantage of using the `meta` attribute for this is that the `FlowOptions` can be specified in the configuration file. - -## Evaluators - -One of the fields on `FlowOptions` above was the evaluator. This is a hook that allows for customization of how the steps in the workflow are evaluated. -At a high level, an evaluator is a special type of `CallableModel` that operates on a combination of the `__call__` function and the context, and returns a new function to be evaluated. -In such a way, additional functionality can be implemented, including - -- Logging (i.e. `LoggingEvaluator`), which is the default -- Local caching (i.e. `MemoryCacheEvaluator`) -- Graph evaluator (i.e. `GraphEvaluator`) -- Dependency tracking (not open sourced yet) -- Disk based caching (not open sourced yet) -- Distributed evaluation (not open sourced yet) -- User defined evaluators - -### In-memory Caching - -Since we use standard python code to schedule the tasks within the workflow and pass data between tasks, diamond dependencies will cause the same task to be called more than once. -This may have performance implications as well as other unwanted side-effects if the tasks are not idempotent. Thus, `ccflow` provides an in-memory caching evaluator (`MemoryCacheEvaluator`) that will make sure that each task is evaluated only once for each context. - -As caching is not always suitable for all models (if they are not referentially transparent), caching is opt-in by default by setting `FlowOptions.cacheable=True`. -Since it is also possible to set the default for all models to be cacheable via `FlowOptionsOverride`, there is also the ability to opt-out of caching for a specific model by setting `FlowOptions.volatile=True` in the `@Flow.call` decorator. - -To illustrate, we start with a model that is neither `cacheable` nor `volatile`, and run it the standard way: - -```python -from ccflow import CallableModel, Flow, GenericResult, GenericContext - -class FibonacciModel(CallableModel): - salt: int = 0 - - @Flow.call - def __call__(self, context: GenericContext[int]) -> GenericResult[int]: - print(f"Calling model with {context}") - if context.value <= 1: - return context.value - else: - return self(context.value - 1).value + self(context.value - 2).value - -model = FibonacciModel() -print(model(4)) -#> Calling model with GenericContext[int](value=4) -#> Calling model with GenericContext[int](value=3) -#> Calling model with GenericContext[int](value=2) -#> Calling model with GenericContext[int](value=1) -#> Calling model with GenericContext[int](value=0) -#> Calling model with GenericContext[int](value=1) -#> Calling model with GenericContext[int](value=2) -#> Calling model with GenericContext[int](value=1) -#> Calling model with GenericContext[int](value=0) -#> GenericResult[int](value=3) -``` - -Now we run it with `FlowOptions.cacheable=True` and the `MemoryCacheEvaluator`: - -```python -from ccflow.evaluators import MemoryCacheEvaluator -from ccflow import FlowOptionsOverride - -evaluator = MemoryCacheEvaluator() -with FlowOptionsOverride(options={"cacheable":True, "evaluator": evaluator}): - print(model(4)) -#> Calling model with GenericContext[int](value=4) -#> Calling model with GenericContext[int](value=3) -#> Calling model with GenericContext[int](value=2) -#> Calling model with GenericContext[int](value=1) -#> Calling model with GenericContext[int](value=0) -#> GenericResult[int](value=3) -``` - -If we call it again with the same evaluator, results will still be pulled from the cache. This holds even if we construct a new model object with the same attributes: - -```python -model = FibonacciModel() -with FlowOptionsOverride(options={"cacheable":True, "evaluator": evaluator}): - print(model(2)) -#> GenericResult[int](value=1) -``` - -Since the model attributes are part of the cache key, changing them will cause the cache to be invalidated: - -```python -model = FibonacciModel(salt=1) -with FlowOptionsOverride(options={"cacheable":True, "evaluator": evaluator}): - print(model(2)) -#> Calling model with GenericContext[int](value=2) -#> Calling model with GenericContext[int](value=1) -#> Calling model with GenericContext[int](value=0) -#> GenericResult[int](value=1) -``` - -#### How cache keys are built - -`MemoryCacheEvaluator` uses `ccflow.evaluators.cache_key()`, which now delegates to `ccflow.utils.compute_cache_token()`. - -At a high level, the cache key combines: - -- a **data token** for the serialized model/context payload -- a **behavior token** for each callable/evaluator class that can affect the result - -For a direct `CallableModel` call, the cache key depends on: - -- `model.model_dump(mode="python")` -- `compute_behavior_token(type(model))` - -For a `ModelEvaluationContext`, the default structural cache key depends on: - -- the underlying context payload plus the function name being evaluated (`__call__` vs `__deps__`) -- the behavior token of the underlying model class -- the data and behavior tokens of any **non-transparent** evaluators in the chain - -`MemoryCacheEvaluator` and graph construction request an effective key for -generated `@Flow.model` nodes. That effective key can project a runtime -`FlowContext` down to the contextual fields a generated node actually reads, so -unused ambient fields do not necessarily split memory-cache or graph identity. - -Transparent evaluators are skipped, so wrapping a model with logging or other pass-through evaluators does not change its cache identity. - -`compute_behavior_token()` hashes the class's Python-defined methods and also consults `__ccflow_tokenizer_deps__` for behavior that lives outside the class body. This is useful when the callable depends on module-level helpers or shared helper classes that should also invalidate the cache key when they change. - -```python -def helper(x): - return x + 1 - - -class SharedLogic: - def transform(self, x): - return x * 2 - - -class MyModel(CallableModel): - __ccflow_tokenizer_deps__ = [helper, SharedLogic] -``` - -Entries in `__ccflow_tokenizer_deps__` may be functions or classes. Class dependencies are tokenized recursively via `compute_behavior_token()`. Recursive class dependency graphs are rejected with a `TypeError`. - -### Graph Evaluation - -Dependencies between tasks/steps in a workflow is one of the defining characteristics of a workflow orchestration framework. Earlier we covered how dependencies defined (via composition) between configuration objects. -However, since a `CallableModel` is a `BaseModel`, the same principle applies. Until now, we have been evaluating the `__call__` functions of models in the same order that python would as part of standard execution, -and we used caching to avoid redundant evaluations. - -In other cases, it may be desirable to inspect the dependencies between tasks and evaluate them in a more optimal order. However, to do this, we need to describe an addition feature of the `CallableModel` class. -We eschew fancy inspection tricks to discover dependencies automatically, and instead rely on the user to define them explicitly. - -We illustrate by subclassing the `FibonacciModel` from above to create a `FibonacciDepsModel` with dependencies defined: - -```python -class FibonacciDepsModel(FibonacciModel): - @Flow.deps - def __deps__(self, context: GenericContext[int]): - if context.value <= 1: - return [] - return [(self, [GenericContext[int](value=context.value - 2), GenericContext[int](value=context.value - 1)])] -``` - -The type of the return value is `ccflow.GraphDepList`, which is `List[Tuple[CallableModelType, List[ContextType]]]`. In other words, for each `CallableModel` the evaluation depends on, the user lists which contexts it depends on for that model. -Note that the `__deps__` method is not called directly by users, but rather it is used internally by evaluator implementations. Since `__deps__` are explicitly specified by the creator of the model, they can return dependencies on models whose outputs do not directly feed into the current model (i.e. perhaps a model that writes a file to a shared location). -Similarly, the `__deps__` method can explicitly leave out actual dependencies if they can always be evaluated alongside the current model. - -With the dependencies specified, we can now use this information to evaluate the workflow in a more optimal order. For example, the `GraphEvaluator` will topologically sort the dependency graph and evaluate the calls in that order. -However, because the later calls will still execute the standard code in the body of the `__call__` method, we need to use the `MemoryCacheEvaluator` to pull these from the cache. - -```python -from ccflow.evaluators import GraphEvaluator, MemoryCacheEvaluator, MultiEvaluator -model = FibonacciDepsModel() -evaluator = MultiEvaluator(evaluators=[GraphEvaluator(), MemoryCacheEvaluator()]) -with FlowOptionsOverride(options={"cacheable":True, "evaluator": evaluator}): - print(model(4)) -#> Calling model with GenericContext[int](value=0) -#> Calling model with GenericContext[int](value=1) -#> Calling model with GenericContext[int](value=2) -#> Calling model with GenericContext[int](value=3) -#> Calling model with GenericContext[int](value=4) -#> GenericResult[int](value=3) -``` - -This also opens up possibilities for distributed evaluation. - -### User Defined Evaluators - -Most importantly, it is straightforward to implement your own evaluators, as no single library would be able to provide evaluators to cover all use cases. -Custom evaluators can be used to - -- run models on other computing platforms (i.e. `dask`, `ray`, `spark`, etc.) -- to implement custom caching strategies (i.e. `redis`, `s3`, etc.) -- to implement custom logging strategies -- to implement custom evaluation strategies (i.e. `batching`) - -An evaluator is basically another form of callable model, with a few caveats - -- It doesn't use the `@Flow.call` decorator -- It uses the `ModelEvaluationContext` as the context type - -The `ModelEvaluationContext` has fields for the model, the context, the function to evaluate (i.e. `__call__`), and the `FlowOptions`. -It too, has a `__call__` method that will evaluate the function on the model with the provided context (but ignoring any options). - -Evaluators that do not modify the return value (e.g. logging, caching, timing) should override the `is_transparent` method to return `True`. -This allows `cache_key()` to skip these layers when computing cache keys, so that wrapping a model with different transparent evaluators does not change its cache identity. -Evaluators that transform the result should inherit from `EvaluatorBase` directly and leave `is_transparent` as the default (`False`). - -Below we illustrate how to write a really simple evaluator that just prints the options and delegates to the `ModelEvaluationContext` to get the normal result. -Since it does not modify the return value, it overrides `is_transparent` to return `True`. - -```python -from ccflow import EvaluatorBase, ModelEvaluationContext, ResultType - -class MyEvaluator(EvaluatorBase): - - def is_transparent(self, context: ModelEvaluationContext) -> bool: - return True - - def __call__(self, context: ModelEvaluationContext) -> ResultType: - print("Custom evaluator with options:", context.options) - return context() - -evaluator = MyEvaluator() -model = FibonacciModel() -with FlowOptionsOverride(options={"cacheable": True, "evaluator": evaluator}): - print(model(0)) - -#> Custom evaluator with options: {'cacheable': True, 'type_': 'ccflow.callable.FlowOptions'} -#> Calling model with GenericContext[int](value=0) -#> GenericResult[int](value=0) -``` diff --git a/docs/wiki/_Sidebar.md b/docs/wiki/_Sidebar.md index 75c31c09..ca06d8cc 100644 --- a/docs/wiki/_Sidebar.md +++ b/docs/wiki/_Sidebar.md @@ -8,19 +8,40 @@ Notes for editors: **[Home](Home)** -**Get Started** +**Tutorials** -- [Installation](Installation) -- [Design Goals](Design-Goals) -- [Key Features](Key-Features) -- [Flow Model](Flow-Model) +- [Overview](Tutorials) - [First Steps](First-Steps) +- [Configuring Models](Configuring-Models) +- [Defining Workflows](Defining-Workflows) +- [Building an ETL Pipeline](Building-an-ETL-Pipeline) +- [Composing an ETL Application](Composing-an-ETL-Application) +- [Building a Configurable Calculator](Building-a-Configurable-Calculator) -**Tutorials** +**How-to Guides** + +- [Overview](How-to-Guides) +- [Install ccflow](Installation) +- [Configure Complex Values](Configure-Complex-Values) +- [Bind Logic to Configs](Bind-Logic-to-Configs) +- [Run Workflows from the CLI](Run-Workflows-from-the-CLI) +- [Cache Results](Cache-Results) +- [Retry on Failure](Retry-on-Failure) -- [Configuration](Configuration) -- [Workflows](Workflows) -- [ETL](ETL) +**Reference** + +- [Overview](Reference) +- [Core Types](Core-Types) +- [Contexts and Results](Contexts-and-Results) +- [Built-in Models](Built-in-Models) +- [Flow Model](Flow-Model) + +**Explanation** + +- [Overview](Explanation) +- [Core Concepts](Core-Concepts) +- [Design Goals](Design-Goals) +- [Configuration and Hydra](Configuration-and-Hydra) **Developer Guide** diff --git a/docs/wiki/explanation/Configuration-and-Hydra.md b/docs/wiki/explanation/Configuration-and-Hydra.md new file mode 100644 index 00000000..a187f048 --- /dev/null +++ b/docs/wiki/explanation/Configuration-and-Hydra.md @@ -0,0 +1,96 @@ +# Configuration and Hydra + +`ccflow` can be used entirely from Python, but it is designed to work hand-in-hand with [Hydra](https://hydra.cc/). This page explains *why* file-based configuration and a command line matter, what Hydra brings to the table, and how the `ModelRegistry` complements it. If you want the steps rather than the reasoning, see the [Configuring Models](Configuring-Models) tutorial, the [Composing an ETL Application](Composing-an-ETL-Application) tutorial, and the [Run Workflows from the CLI](Run-Workflows-from-the-CLI) guide. + +## Two ways of working, one set of objects + +Real applications accumulate configuration. A data pipeline has sources, credentials, transforms, output locations, schedules, calendars, retry policies, and execution settings — and each of those has its own parameters. `ccflow` targets two quite different ways of interacting with all that configuration: + +- **Static and versioned.** In production you want the entire configuration pinned down in files, reviewed, version-controlled, and reproducible. You run it from the command line and occasionally override a value. +- **Dynamic and interactive.** In research you want to construct, modify, and re-run configurations from a notebook or script, without editing files or restarting anything. + +The important design goal is that these are *the same objects* seen two ways. A model you build interactively in Python is the same model Hydra instantiates from a YAML file. This is what lets a workflow move from a research notebook into production without being rewritten. The interactive path is served by the [`ModelRegistry`](Core-Concepts); the file-and-CLI path is served by Hydra. + +## Why not just dictionaries and `argparse`? + +It is tempting to reach for plain dictionaries, environment variables, or `argparse` for configuration. These break down as an application grows: + +- **No schema.** A misspelled key (writing `weight` where the schema expects `weights`) or a wrong type (the string `"1234"` where an integer is expected) fails silently or far from its cause. `ccflow`'s pydantic models catch these when the configuration loads. See [Validate and Coerce Configuration](Configure-Complex-Values). +- **No composition.** A flat namespace does not let you assemble configuration from reusable pieces, spread it across files, or reuse the same fragment in several places. +- **No reuse of *choices*.** The hard part of large configuration is not the individual values, it is expressing "use the S3 cache here, the SQLite cache there, and swap them without touching anything else." + +Hydra exists to solve the composition and command-line problems; pydantic (via `ccflow.BaseModel`) solves the schema problem. `ccflow` uses both. + +## What Hydra provides + +[Hydra](https://hydra.cc/), built on [OmegaConf](https://omegaconf.readthedocs.io/), is a framework for **composing hierarchical configuration from multiple files** and **overriding it from the command line**. A few of its ideas do most of the work. + +### Composition from files + +Configuration can be split across many small YAML files in a directory tree and recombined. A top-level file declares a `defaults` list that pulls in the others: + +```yaml +defaults: + - _self_ + - cache: memory + - execution: local +``` + +Each entry brings in another file, and `_self_` marks where the current file's own content merges relative to those pieces. This keeps each concern in its own small, readable, independently reviewable file instead of one sprawling document. + +### Config groups: the key idea + +A **config group** is a directory of interchangeable options for one slice of the application. If you have a folder `cache/` containing `memory.yaml`, `redis.yaml`, and `s3.yaml`, then `cache` is a config group and each file is an option. You select one in the defaults list (`- cache: memory`) or from the command line (`cache=redis`). + +This is the mechanism that makes an application genuinely *composable*. Each subsystem — the cache, the execution policy, the set of credentials, the output target — becomes a named, swappable slot. Changing which implementation an application uses is a matter of naming a different option, not editing code or wiring. An application assembled this way is really a small matrix of choices; a concrete run is one selection through that matrix. + +Config groups also compose *across packages*. By adding a search path, an application can pull config groups out of an installed library: + +```yaml +hydra: + searchpath: + - pkg://mytoolkit.config +``` + +This is what lets a reusable toolkit ship a library of config groups (calendars, credential shapes, cache backends, output writers) that any downstream application can select from — a connector package can contribute `cache=redis` simply by being installed. The [Composing an ETL Application](Composing-an-ETL-Application) tutorial builds an application in exactly this style. + +### Packages and where config lands + +The `@package` directive controls *where* a file's content is placed in the final configuration tree. A file that begins with `# @package _global_` merges its content at the root rather than under its group name, which is useful for base configurations that define top-level keys. Packages let a config group option write to whatever part of the tree it needs to. + +### Overrides and interpolation + +Once composed, any node can be overridden from the command line without touching a file — append a new key (`+context.date=2025-01-01`), change an existing one (`++cache.ttl=60`), or select a different group option (`cache=redis`). This is the fast, safe way to run one-off variations of a pinned configuration. + +Within the configuration, OmegaConf **interpolation** lets one node reference another (`${extract.output.name}`), so a value is written once and reused. Resolvers extend this — `${oc.select:key,default}` resolves a key if it exists and falls back otherwise, which is a natural way to make a configuration *dispatch* between alternatives (for example, "run the backfill wrapper if one is selected, otherwise run the task directly"). + +### Instantiation into objects + +Finally, Hydra turns configuration into objects. A mapping with a `_target_` key names a class to construct. Because `ccflow.BaseModel` serializes its type to `_target_`, Hydra-composed configuration instantiates `ccflow` models directly — the composed YAML *is* the object graph. + +## How the registry complements Hydra + +Hydra is excellent at composing files and instantiating objects, but on its own it stops at instantiation. `ccflow`'s `ModelRegistry` adds two things Hydra does not. + +**Shared instances instead of copies.** OmegaConf interpolation (`${...}`) copies configuration: two nodes that interpolate the same source end up configured identically but as *separate instances*. The registry instead lets a model reference another *by name*, resolving to the **same object**. When configuration forms a dependency graph — one data source feeding many signals feeding one portfolio — you usually want the shared-instance semantics so a change at the source propagates everywhere. This name-based dependency injection is a `ccflow` addition on top of Hydra. [Core Concepts](Core-Concepts) and the [Configuring Models](Configuring-Models) tutorial show it in action. + +**The same power without files.** The registry provides the interactive half of the story. `ModelRegistry.load_config(...)` loads a dictionary of configs (resolving name references as it goes), and `load_config_from_path(...)` loads the same Hydra files a CLI would use — so a researcher can pull the production configuration into a notebook, inspect it, tweak an object, and re-run, all as live Python objects. + +## The payoff + +Put together, the configuration story gives you: + +- **One application, many configurations** — a versioned base plus command-line overrides for day-to-day variation. +- **Swappable subsystems** — change caches, executors, outputs, or credentials by selecting a different config group option, including options contributed by installed packages. +- **CLI-based dispatch** — choose *which* callable to run, and with what context, from the command line, so one entry point serves an entire catalog of workflows. +- **Reproducibility and testability** — configuration is data: reviewable in a pull request, diffable, and testable independently of the logic it drives. +- **A research-to-production path** — the same objects, whether built in a notebook or composed from files. + +This composition style is not just a convenience for large projects; it is the foundation for building a reusable workflow toolkit that others assemble applications from. The [Composing an ETL Application](Composing-an-ETL-Application) tutorial walks through building exactly such a thing. + +## Where to go next + +- [Configuring Models](Configuring-Models) — build and register configurations interactively, then load them from files. +- [Composing an ETL Application](Composing-an-ETL-Application) — assemble a config-group-driven, CLI-dispatched application. +- [Run Workflows from the CLI](Run-Workflows-from-the-CLI) — the mechanics of running and overriding configured workflows. +- Hydra's own [documentation](https://hydra.cc/docs/intro/) — the full override grammar, config groups, and compose API. diff --git a/docs/wiki/explanation/Core-Concepts.md b/docs/wiki/explanation/Core-Concepts.md new file mode 100644 index 00000000..a14687da --- /dev/null +++ b/docs/wiki/explanation/Core-Concepts.md @@ -0,0 +1,63 @@ +# Core Concepts + +`ccflow` (Composable Configuration Flow) is really two frameworks that share a foundation: + +- a **configuration framework** for defining hierarchical, strongly typed configuration and the relationships between pieces of it, and +- a **workflow framework** for associating code with that configuration, so that a configuration graph becomes a runnable graph of tasks. + +The two halves fit together because everything in `ccflow` is a *model*. A workflow step is just a configuration object that also knows how to run. This page introduces the vocabulary you will meet throughout the rest of the documentation and explains how the pieces relate. It is a conceptual map, not an API listing — for the exact classes and their fields, see the [Reference](Reference). + +## Models are configuration + +The central abstraction is the **model**, `ccflow.BaseModel`. The naming follows [pydantic](https://docs.pydantic.dev/latest/), whose `BaseModel` it extends: think of a model as a *configurable class* — a set of typed, validated attributes. Because it is a pydantic model, you get schema validation, coercion, and serialization for free. + +`ccflow` extends pydantic's base class in a few deliberate ways: it serializes the model's own type alongside its fields (so a serialized config can be reconstructed as the right subclass), aliases that type field to `_target_` for [`hydra`](Configuration-and-Hydra) compatibility, and rejects unknown or misnamed fields so configuration mistakes surface immediately rather than silently. + +Using classes rather than raw dictionaries for configuration is a design choice, not an accident. A schema catches typos and type errors *when the configuration loads* instead of when it is used, lets you validate combinations of parameters, and makes configuration self-documenting. [Design Goals](Design-Goals) discusses this reasoning in depth. + +## The registry binds configuration together + +A **`ModelRegistry`** is a named collection of models — conceptually a catalog. Registries can contain other registries, forming a hierarchy, and a single **root** registry sits at the top as a singleton. + +The registry is what turns a pile of independent config objects into a connected graph. Instead of passing configuration around by hand, a model can refer to another model *by its name in the root registry*, and `ccflow` resolves that reference to the actual instance. This is dependency injection: a high-level config depends on lower-level configs by name, changes propagate through the shared instance, and any part of the hierarchy can be swapped without disturbing the rest. + +This name-based linking is subtly but importantly different from copying configuration around. Two configs that both reference `data/prices` point at the *same object*, so a change in one place is seen everywhere — a graph, not a tree of duplicated values. + +## Contexts parameterize workflows + +A **context** (`ccflow.ContextBase`) is the set of parameters that *templatize* a workflow at runtime. The classic example is a date: the same pipeline runs for many dates, so date is a context rather than fixed configuration. + +Separating context from configuration lets a workflow author expose only the few parameters that vary between runs (date, region, symbol, universe) while keeping everything else fixed in configuration. It also keeps configuration immutable during a run — contexts, not configs, carry the per-run variation, which is why contexts are frozen and hashable (so they can serve as cache keys). A context is itself a kind of result, so one step can compute the context another step consumes. + +## Results carry data between steps + +A **result** (`ccflow.ResultBase`) is the typed output of a workflow step. Insisting that every step return a `ResultBase` (again, a pydantic model) gives the framework a uniform thing to validate, serialize, and operate on generically — and it is what makes features like delayed and cached evaluation possible. Results range from the catch-all `GenericResult` to typed wrappers around common data structures. + +## Callable models are workflow steps + +A **`CallableModel`** is a `BaseModel` that can be *called*: given a context, it returns a result. It is the point where configuration meets code. You implement its `__call__` method and decorate it with `@Flow.call`; the decorator is where the framework injects its machinery. + +Because a `CallableModel` is still a `BaseModel`, a workflow step is configurable, serializable, registrable, and composable exactly like any other configuration. Steps depend on other steps through ordinary object composition — Python itself passes data between them and determines evaluation order — so wiring a workflow needs no new language or scheduler. `@Flow.model` offers an alternative, function-first way to define the same kind of model from a plain Python signature. + +## The Flow decorator hides the machinery + +`@Flow.call` (and its sibling `@Flow.model`) is where "the magic lives". By default it performs runtime type checking of the context and result, cutting boilerplate while preserving type safety. But it is also the seam through which the framework layers on logging, caching, and alternative evaluation strategies — without the model's author writing any of that. The behavior is controlled by **`FlowOptions`**, which can be set on the decorator, on the model's metadata, through a scoped override context manager, or per call. + +## Evaluators control *how* steps run + +An **evaluator** decides how a callable model is actually executed. The default logs each evaluation; others cache results in memory, evaluate an explicit dependency graph in topological order, retry on failure, or (in extensions) distribute work across a cluster. Evaluators are themselves models, so the execution strategy is just more configuration. This separation — *what* the workflow computes versus *how* it is run — is deliberate: you can add caching, retries, or distribution to an existing workflow by changing an evaluator, not by rewriting steps. + +## Publishers send data out + +A **publisher** is a model that knows how to take data and send it somewhere — a file, an object store, a database, an email, a report. Giving all publishers a common interface means one can be substituted for another purely through configuration, so switching from writing a local file to emailing a report is a config change rather than a code change. + +## How it fits together + +A complete `ccflow` application is a registry of models. Some are pure configuration; some are callable steps wired together by composition and registry references; contexts parameterize them; evaluators decide how they run; publishers move their outputs. Because every one of these is a `BaseModel`, the whole thing can be defined interactively in Python for research, or loaded from files with [`hydra`](Configuration-and-Hydra) for production — the same objects either way. + +From here: + +- [Design Goals](Design-Goals) — the requirements and reasoning behind these abstractions. +- [Configuration and Hydra](Configuration-and-Hydra) — why file-based composition and a CLI matter, and how config groups build on the registry. +- [Tutorials](Tutorials) — put the concepts into practice, starting with [First Steps](First-Steps). +- [Reference](Reference) — the exact contexts, results, models, publishers, and evaluators that ship with `ccflow`. diff --git a/docs/wiki/Design-Goals.md b/docs/wiki/explanation/Design-Goals.md similarity index 100% rename from docs/wiki/Design-Goals.md rename to docs/wiki/explanation/Design-Goals.md diff --git a/docs/wiki/explanation/Explanation.md b/docs/wiki/explanation/Explanation.md new file mode 100644 index 00000000..7615d1ee --- /dev/null +++ b/docs/wiki/explanation/Explanation.md @@ -0,0 +1,9 @@ +# Explanation + +These pages build understanding of `ccflow` — what it is for, the ideas behind its design, and why it leans on the tools it does. They are for reading and reflection rather than for following along at the keyboard; when you want to *do* something, reach for a [How-to Guide](How-to-Guides) or work through a [Tutorial](Tutorials) instead. + +- **[Core Concepts](Core-Concepts)** — the vocabulary of `ccflow` (models, the registry, contexts, results, callable models, evaluators, publishers) and how the configuration half and the workflow half fit together into one framework. +- **[Design Goals](Design-Goals)** — the problems `ccflow` set out to solve for configuration and workflow management, and the requirements that shaped it. +- **[Configuration and Hydra](Configuration-and-Hydra)** — why file-based configuration and a command line matter, what `hydra` and its config groups bring, and how the `ModelRegistry` complements them to enable composable, dispatchable applications. + +If you are new, the [Design Goals](Design-Goals) explain the "why does this exist at all", [Core Concepts](Core-Concepts) give you the shared language used everywhere else, and [Configuration and Hydra](Configuration-and-Hydra) motivates the composition style used in the [Composing an ETL Application](Composing-an-ETL-Application) tutorial. diff --git a/docs/wiki/how-to/Bind-Logic-to-Configs.md b/docs/wiki/how-to/Bind-Logic-to-Configs.md new file mode 100644 index 00000000..0db7da31 --- /dev/null +++ b/docs/wiki/how-to/Bind-Logic-to-Configs.md @@ -0,0 +1,158 @@ +# Bind Logic to Configs + +A configuration object becomes useful when code lives close to the configuration it needs. This guide shows how to attach business logic to configuration classes so that a config is also a runnable unit — a reader, a publisher, or a stage in a data pipeline. + +The pattern is to keep logic *close* to the configuration it uses, rather than reaching into the registry for values throughout your codebase (which recreates the problems of global variables — see [Core Concepts](Core-Concepts)). For full workflow steps with contexts, results, and evaluators, use a [`CallableModel`](Defining-Workflows); the plain-method pattern here is handy for simpler cases. + +## Add a `__call__` method + +To make a config runnable, give it a `__call__` method. Following the [Single Responsibility Principle](https://en.wikipedia.org/wiki/Single-responsibility_principle), keep one purpose per class: + +```python +import pandas as pd +from pathlib import Path +from ccflow import BaseModel + +class MyParquetConfig(BaseModel): + file: Path + description: str = "" + + def __call__(self): + """Read the file as a pandas data frame.""" + return pd.read_parquet(self.file) + +df = MyParquetConfig(file="data.parquet")() +``` + +There are no restrictions on what lives inside — validation conforms the *fields*, but the logic is ordinary Python. Branch on configuration as needed: + +```python +import pyarrow.parquet as pq +from typing import Literal + +class MyParquetConfig(BaseModel): + file: Path + library: Literal["pandas", "pyarrow"] = "pandas" + + def read_pandas(self): + return pd.read_parquet(self.file) + + def read_arrow(self): + return pq.read_table(self.file) + + def __call__(self): + return self.read_pandas() if self.library == "pandas" else self.read_arrow() +``` + +## Write a custom publisher + +To send data somewhere (files, an object store, email, a report), implement the publisher interface `ccflow.publishers.BasePublisher`. A common interface means publishers can be swapped purely through configuration. Here is one that renders a list of strings as HTML in a notebook, using a [Jinja template](Configure-Complex-Values#configure-a-jinja-template-eg-a-sql-query): + +```python +from typing import List +from IPython.display import display, HTML +from ccflow import JinjaTemplate, BasePublisher + +class MyPublisher(BasePublisher): + data: List[str] = None + html_template: JinjaTemplate + + def __call__(self): + display(HTML(self.get_name())) + display(HTML(self.html_template.template.render(data="
".join(self.data)))) + +p = MyPublisher( + name="My {{desc}} publisher:", + html_template="""

{{data}}

""", +) +p.name_params = dict(desc="test") +p.data = ["Blue text.", "More blue text."] +p() +``` + +Because `data` is a typed field, invalid data is rejected before publishing: + +```python +try: + p.data = {} +except ValueError as v: + print(v) +``` + +`ccflow` ships publishers for common cases — see [Built-in Models](Built-in-Models#publishers). + +## Build a data pipeline + +To wire several stages together, give each stage a config with a `__call__`, and connect them by registry reference. The models below read, augment, and summarize data with [Polars](https://docs.pola.rs/) (requires `polars`): + +```python +from typing import List, Optional +from pathlib import Path +import polars as pl +from ccflow import BaseModel, ModelRegistry +from ccflow.exttypes.polars import PolarsExpression + + +class PolarsCallable(BaseModel): + """Base class for models that return a polars LazyFrame.""" + def __call__(self) -> pl.LazyFrame: ... + +class ParquetDataReader(PolarsCallable): + file: Path + n_rows: Optional[int] = None + + def __call__(self): + return pl.scan_parquet(self.file, n_rows=self.n_rows) + +class FeatureAdder(PolarsCallable): + data_input: PolarsCallable + group_col: str + average_cols: List[str] + + def __call__(self): + df = self.data_input() + agg = [pl.col(c).mean().alias(f"{c}_mean") for c in self.average_cols] + avg = df.group_by(pl.col(self.group_col)).agg(agg) + joined = df.join(avg, on=self.group_col) + resid = [(pl.col(c) - pl.col(f"{c}_mean")).abs().alias(f"{c}_resid") for c in self.average_cols] + return joined.with_columns(resid) + +class TopKFinder(PolarsCallable): + data_input: PolarsCallable + by: str + k: int = 1 + + def __call__(self): + return self.data_input().top_k(k=self.k, by=self.by) + +class ColumnSelector(PolarsCallable): + data_input: PolarsCallable + exprs: List[PolarsExpression] + + def __call__(self): + return self.data_input().select(self.exprs) +``` + +Register configurations, letting several downstream models share one upstream stage: + +```python +root = ModelRegistry.root().clear() +root.add("Raw Data", ParquetDataReader(file="example.parquet")) +root.add("Augmented Data", FeatureAdder(data_input="Raw Data", group_col="State", average_cols=["Sales", "Profit"])) +root.add("TopK Profit residuals", TopKFinder(data_input="Augmented Data", by="Profit_resid", k=3)) +root.add("Mean residuals", ColumnSelector(data_input="Augmented Data", exprs=[pl.col("Sales_resid").mean(), pl.col("Profit_resid").mean()])) +``` + +Now load any model from the registry and call it — you get back a `polars.LazyFrame` (you built the Polars graph via `ccflow`), which you materialize with `collect()`: + +```python +print(root["TopK Profit residuals"]().collect()) +``` + +Because both downstream models reference the same `"Augmented Data"` instance, changing it once (for example, `root["Augmented Data"].group_col = "Region"`) flows through to every result that depends on it. + +## See also + +- [Defining Workflows](Defining-Workflows) — the full `CallableModel` pattern with contexts, results, and evaluators. +- [Configure Complex Values](Configure-Complex-Values) — richer field types for your configs. +- [Built-in Models](Built-in-Models) — publishers and models that ship with `ccflow`. diff --git a/docs/wiki/how-to/Cache-Results.md b/docs/wiki/how-to/Cache-Results.md new file mode 100644 index 00000000..4b52e3ed --- /dev/null +++ b/docs/wiki/how-to/Cache-Results.md @@ -0,0 +1,138 @@ +# Cache Results + +Because `ccflow` schedules tasks with ordinary Python, a diamond-shaped dependency graph will call the same task more than once. This guide shows how to cache results to avoid redundant work, how to evaluate an explicit dependency graph, and how to write your own evaluator. It builds on [Defining Workflows](Defining-Workflows); the full evaluator catalog is in [Built-in Models](Built-in-Models#evaluators). + +The examples use this model, which prints on each call so you can see when it actually runs: + +```python +from ccflow import CallableModel, Flow, GenericResult, GenericContext, FlowOptionsOverride +from ccflow.evaluators import MemoryCacheEvaluator + +class FibonacciModel(CallableModel): + salt: int = 0 + + @Flow.call + def __call__(self, context: GenericContext[int]) -> GenericResult[int]: + print(f"Calling model with {context}") + if context.value <= 1: + return context.value + return self(context.value - 1).value + self(context.value - 2).value +``` + +## Enable in-memory caching + +Caching is opt-in. Set `cacheable=True` and supply a `MemoryCacheEvaluator`, scoped with `FlowOptionsOverride`, so each `(model, context)` runs once: + +```python +model = FibonacciModel() +evaluator = MemoryCacheEvaluator() +with FlowOptionsOverride(options={"cacheable": True, "evaluator": evaluator}): + print(model(4)) +#> Calling model with GenericContext[int](value=4) +#> Calling model with GenericContext[int](value=3) +#> Calling model with GenericContext[int](value=2) +#> Calling model with GenericContext[int](value=1) +#> Calling model with GenericContext[int](value=0) +#> GenericResult[int](value=3) +``` + +The redundant calls that plain evaluation would make are gone. Reusing the same evaluator keeps serving from the cache, even for a freshly constructed model with the same fields: + +```python +model = FibonacciModel() +with FlowOptionsOverride(options={"cacheable": True, "evaluator": evaluator}): + print(model(2)) +#> GenericResult[int](value=1) +``` + +To keep a model out of the cache even when caching is on globally, mark it `volatile=True` in its `@Flow.call`. + +## Understand cache invalidation + +A model's fields are part of its cache key, so changing them invalidates the cache automatically: + +```python +model = FibonacciModel(salt=1) +with FlowOptionsOverride(options={"cacheable": True, "evaluator": evaluator}): + print(model(2)) +#> Calling model with GenericContext[int](value=2) +#> ... +``` + +If a model's behavior depends on code *outside* the class body (a module-level helper or shared class), list those in `__ccflow_tokenizer_deps__` so the cache key changes when they change: + +```python +def helper(x): + return x + 1 + +class SharedLogic: + def transform(self, x): + return x * 2 + +class MyModel(CallableModel): + __ccflow_tokenizer_deps__ = [helper, SharedLogic] +``` + +Wrapping a model in *transparent* evaluators (logging, timing) does not change its cache identity. For the full key-derivation rules, see [Built-in Models](Built-in-Models#how-cache-keys-are-built). + +## Evaluate a dependency graph + +To evaluate steps in an optimal order rather than Python's call order, declare dependencies explicitly with `@Flow.deps` and use the `GraphEvaluator` (together with the cache, since graph nodes still run their `__call__` bodies): + +```python +class FibonacciDepsModel(FibonacciModel): + @Flow.deps + def __deps__(self, context: GenericContext[int]): + if context.value <= 1: + return [] + return [(self, [GenericContext[int](value=context.value - 2), GenericContext[int](value=context.value - 1)])] +``` + +`__deps__` returns a `GraphDepList` — for each model an evaluation depends on, the list of contexts it needs. Then combine the evaluators: + +```python +from ccflow.evaluators import GraphEvaluator, MultiEvaluator + +model = FibonacciDepsModel() +evaluator = MultiEvaluator(evaluators=[GraphEvaluator(), MemoryCacheEvaluator()]) +with FlowOptionsOverride(options={"cacheable": True, "evaluator": evaluator}): + print(model(4)) +#> Calling model with GenericContext[int](value=0) +#> Calling model with GenericContext[int](value=1) +#> Calling model with GenericContext[int](value=2) +#> Calling model with GenericContext[int](value=3) +#> Calling model with GenericContext[int](value=4) +#> GenericResult[int](value=3) +``` + +Note the topological order (0, 1, 2, 3, 4), and that each node runs once. This is also the foundation for distributed evaluation. + +## Write a custom evaluator + +No library can provide every execution strategy, so evaluators are extensible. An evaluator is a model that takes a `ModelEvaluationContext` (which carries the model, context, function, and options) and returns a result. Override `is_transparent` to return `True` if it does not change the result (so caching ignores it): + +```python +from ccflow import EvaluatorBase, ModelEvaluationContext, ResultType + +class MyEvaluator(EvaluatorBase): + def is_transparent(self, context: ModelEvaluationContext) -> bool: + return True + + def __call__(self, context: ModelEvaluationContext) -> ResultType: + print("Custom evaluator with options:", context.options) + return context() + +with FlowOptionsOverride(options={"cacheable": True, "evaluator": MyEvaluator()}): + print(FibonacciModel()(0)) +#> Custom evaluator with options: {'cacheable': True, 'type_': 'ccflow.callable.FlowOptions'} +#> Calling model with GenericContext[int](value=0) +#> GenericResult[int](value=0) +``` + +Custom evaluators can run models on other platforms (Dask, Ray, Spark), implement other caching backends (Redis, S3), or add batching and custom logging. + +## See also + +- [Retry on Failure](Retry-on-Failure) — the retry evaluator and `RetryModel`. +- [Built-in Models](Built-in-Models#evaluators) — the full evaluator catalog and cache-key rules. +- [Defining Workflows](Defining-Workflows) — where `FlowOptions` and evaluators are introduced. diff --git a/docs/wiki/how-to/Configure-Complex-Values.md b/docs/wiki/how-to/Configure-Complex-Values.md new file mode 100644 index 00000000..92a71a0f --- /dev/null +++ b/docs/wiki/how-to/Configure-Complex-Values.md @@ -0,0 +1,182 @@ +# Configure Complex Values + +Configuration is rarely just strings and numbers. This guide collects recipes for putting richer values into `ccflow` configs — custom coercion, templates, expressions, arrays, arbitrary objects, and references to Python objects by path. Each section is independent; jump to the one you need. + +All examples assume: + +```python +from ccflow import BaseModel +from typing import List +``` + +## Coerce a non-dict value into a model + +To let a config accept a convenient shorthand (e.g. a plain string) instead of a full dictionary, add a wrap validator that reshapes the input: + +```python +from pydantic import model_validator + +class NameConfig(BaseModel): + first_name: str + last_name: str + + @model_validator(mode="wrap") + @classmethod + def coerce_from_string(cls, data, handler): + if isinstance(data, str): + names = data.split(" ") + if len(names) == 2: + data = dict(first_name=names[0], last_name=names[1]) + return handler(data) + +print(NameConfig.model_validate("John Doe")) +#> NameConfig(first_name='John', last_name='Doe') +``` + +Now anywhere `NameConfig` is expected, a string is accepted and coerced: + +```python +class MyConfig(BaseModel): + name: NameConfig + +print(MyConfig(name="John Doe")) +#> MyConfig(name=NameConfig(first_name='John', last_name='Doe')) +``` + +For the full range of validators, see the pydantic docs on [validators](https://docs.pydantic.dev/latest/concepts/validators/). + +## Configure a Jinja template (e.g. a SQL query) + +To parameterize a template safely from configuration, use the `JinjaTemplate` extension type. It converts a string into a renderable [Jinja](https://jinja.palletsprojects.com/) template: + +```python +from ccflow import JinjaTemplate + +class MyTemplateConfig(BaseModel): + greeting: JinjaTemplate + user: str + place: str + +config = MyTemplateConfig( + greeting="Hello {{user|upper}}, welcome to {{place}}!", + user="friend", + place="the tutorial", +) +print(config.greeting.template.render(config.dict())) +#> Hello FRIEND, welcome to the tutorial! +``` + +This is the safe way to build SQL queries from parameters rather than string-formatting them: + +```python +from datetime import date + +class MyQueryTemplate(BaseModel): + query: JinjaTemplate + columns: List[str] + where: str = "Test" + query_date: date = date(2024, 1, 1) + filters: List[str] = [] + +query = """select {{columns|join(",\n\t")}} +from MyDatabase +where WhereCol = '{{where}}' + and Date = '{{query_date}}' + {% for filter in filters %}and {{filter}} {% endfor %} +""" + +config = MyQueryTemplate(query=query, columns=["Col1", "Col2 as MyOtherCol", "SomeID"]) +config.query_date = date(2022, 1, 1) +config.filters = ["SomeID IS NOT NULL"] +print(config.query.template.render(config.dict())) +``` + +## Configure a Polars expression + +To define data transformations as configuration, use the `PolarsExpression` extension type, which parses a string into a [Polars](https://docs.pola.rs/) expression (requires `polars`): + +```python +from ccflow.exttypes.polars import PolarsExpression +import polars as pl + +class PolarsConfig(BaseModel): + columns: List[PolarsExpression] + where: PolarsExpression + +config = { + "columns": ["pl.col('Col1')", "pl.col('Col2').alias('MyOtherCol')"], + "where": "pl.col('WhereCol')=='Test'", +} +config_model = PolarsConfig.model_validate(config) +assert isinstance(config_model.columns[0], pl.Expr) +assert isinstance(config_model.where, pl.Expr) +``` + +## Configure a NumPy array + +To hold array data as a real `numpy.ndarray` (with the input conformed automatically), use `NDArray`: + +```python +from ccflow import NDArray +import numpy as np + +class MyNumpyConfig(BaseModel): + my_array: NDArray[np.float64] + my_list: List[float] + +config = MyNumpyConfig(my_array=[1, 2, 3], my_list=[1, 2, 3]) +assert isinstance(config.my_array, np.ndarray) +assert isinstance(config.my_list, list) +``` + +## Allow an arbitrary (non-built-in) type + +To store objects that are not standard types, opt in with `arbitrary_types_allowed` and let Hydra construct the value. Pydantic still validates that the result is an instance of the declared type: + +```python +from hydra.utils import instantiate + +class MyCustomType: + pass + +class MyConfigWithCustomType(BaseModel): + model_config = {"arbitrary_types_allowed": True} + custom: MyCustomType + +config = { + "_target_": "__main__.MyConfigWithCustomType", + "custom": {"_target_": "__main__.MyCustomType"}, +} +config_model = instantiate(config) +assert isinstance(config_model.custom, MyCustomType) +``` + +## Reference a Python object by import path + +To point at something already defined in code — a function, constant, enum, or class — use `PyObjectPath`: + +```python +from ccflow import PyObjectPath + +class MyConfigWithPaths(BaseModel): + builtin_func: PyObjectPath = PyObjectPath("builtins.len") + separator: PyObjectPath = PyObjectPath("ccflow.REGISTRY_SEPARATOR") + +config = MyConfigWithPaths() +assert config.builtin_func.object([1, 2, 3]) == 3 +print(config.separator.object) +#> / +``` + +## Reuse registry aliases and Python defaults + +`ccflow.compose` offers small helpers for interacting with the registry and Python-backed defaults from configuration: + +- `model_alias(model_name)` — resolve a model instance by string name from the active registry. +- `update_from_template(base, update=None, target_class=None)` — build a new instance or dict by shallow-copying `base` and applying updates. `base` may be a registry alias (pass the alias string, or use `_target_: ccflow.compose.model_alias`) or a dict; the shallow copy preserves nested `BaseModel` identity. +- `from_python(py_object_path, indexer=None)` — resolve any Python object by import path, optionally indexing into it with a list of keys. + +## See also + +- [Configuring Models](Configuring-Models) — the tutorial these recipes build on. +- [Bind Logic to Configs](Bind-Logic-to-Configs) — attach behavior to configuration classes. diff --git a/docs/wiki/how-to/How-to-Guides.md b/docs/wiki/how-to/How-to-Guides.md new file mode 100644 index 00000000..e6f73c9e --- /dev/null +++ b/docs/wiki/how-to/How-to-Guides.md @@ -0,0 +1,18 @@ +# How-to Guides + +These guides are recipes for getting a specific job done. They assume you already know the basics from the [Tutorials](Tutorials) and get straight to the task. If you want to understand *why* rather than *how*, see the [Explanation](Explanation) pages; for exact signatures and catalogs, see the [Reference](Reference). + +**Setup** + +- [Install ccflow](Installation) — pip, conda, or from source. + +**Configuration** + +- [Configure Complex Values](Configure-Complex-Values) — custom validation and coercion, Jinja/SQL templates, Polars expressions, NumPy arrays, arbitrary types, and object-by-path references. +- [Bind Logic to Configs](Bind-Logic-to-Configs) — attach business logic to configuration classes, including custom publishers and data pipelines. + +**Running workflows** + +- [Run Workflows from the CLI](Run-Workflows-from-the-CLI) — run configured callables, apply overrides, and inspect the composed configuration. +- [Cache Results](Cache-Results) — avoid redundant work with in-memory caching and graph evaluation, and write your own evaluator. +- [Retry on Failure](Retry-on-Failure) — make flaky steps resilient with retry evaluators and `RetryModel`. diff --git a/docs/wiki/Installation.md b/docs/wiki/how-to/Installation.md similarity index 100% rename from docs/wiki/Installation.md rename to docs/wiki/how-to/Installation.md diff --git a/docs/wiki/how-to/Retry-on-Failure.md b/docs/wiki/how-to/Retry-on-Failure.md new file mode 100644 index 00000000..eb1d2eb2 --- /dev/null +++ b/docs/wiki/how-to/Retry-on-Failure.md @@ -0,0 +1,77 @@ +# Retry on Failure + +To make a flaky step resilient — a network fetch, a rate-limited API — retry it on failure. `ccflow` offers two approaches: a `RetryEvaluator` that wraps whatever runs in its scope, and a `RetryModel` that bakes a retry policy into the graph. This guide covers both. They share the same `RetryPolicy` mechanics; the evaluator catalog is in [Built-in Models](Built-in-Models#evaluators). + +## Retry with an evaluator + +Configure a `RetryEvaluator` and apply it through `FlowOptionsOverride`. It supports exponential backoff, jitter, and stop conditions: + +```python +from ccflow import FlowOptionsOverride +from ccflow.evaluators import RetryEvaluator + +retry = RetryEvaluator( + max_attempts=5, + wait_initial=0.5, # first retry waits ~0.5s + wait_multiplier=2.0, # then 1s, 2s, 4s, ... + wait_max=30.0, # cap any single wait at 30s + wait_jitter=0.25, # add up to 0.25s of random jitter + max_delay=120.0, # stop once cumulative waiting would exceed 2 minutes + retry_exceptions=[TimeoutError, ConnectionError], +) + +with FlowOptionsOverride(options={"evaluator": retry}): + result = my_model(my_context) +``` + +`max_delay` caps the *cumulative* sleep time between retries (not total runtime): once the next backoff would push accumulated waiting over the budget, no further retries happen. Use `retry_exceptions` / `no_retry_exceptions` to choose which exceptions are retried. + +> [!NOTE] +> `retry_exceptions` and `no_retry_exceptions` accept either a bare exception class (as above) or an importable path string (e.g. `"httpx.ConnectError"`, the form for YAML/Hydra configs). Paths are imported eagerly, so a third-party exception requires that package installed. + +The evaluator is transparent (a successful result is identical to a direct call, so caching and dependency graphs are unaffected) and holds no per-call state, so one instance is safe to share across threads and combine with parallel evaluators. Place it *inside* a parallel evaluator to retry each task independently, or *outside* to retry the whole dispatch as a unit. + +## Choose which models are retried + +An evaluator wraps *every* model in its scope, including dependencies. To narrow that, from coarse to fine: + +**Scope the override to model types or instances.** + +```python +with FlowOptionsOverride(options={"evaluator": retry}, model_types=(FetchFromApi,)): + result = pipeline(context) # only FetchFromApi instances are retried +``` + +Use `models=(instance,)` to target a single instance. + +**Override per call.** + +```python +result = my_model(context, _options={"evaluator": retry}) +``` + +**Pin it on the model.** Set `model.meta.options` so a model always carries its retry policy. + +## Bake a policy into the graph with `RetryModel` + +When you want the retry policy to be part of the workflow itself — declared statically in config, visible in serialization and the dependency graph — wrap the model in a `RetryModel` instead. It becomes a first-class node and preserves the wrapped model's `context_type` / `result_type`: + +```python +from ccflow.models import RetryModel + +flaky = RetryModel( + model=fetch_from_api, # any CallableModel + max_attempts=5, + wait_initial=0.5, + retry_exceptions=[TimeoutError, ConnectionError], +) + +result = flaky(my_context) # same context/result types as fetch_from_api +``` + +Use `RetryEvaluator` for a cross-cutting "how to run" policy applied at runtime; use `RetryModel` when the policy belongs to one specific model as part of the graph. + +## See also + +- [Cache Results](Cache-Results) — combine retries with caching and graph evaluation. +- [Built-in Models](Built-in-Models#evaluators) — the full evaluator catalog. diff --git a/docs/wiki/how-to/Run-Workflows-from-the-CLI.md b/docs/wiki/how-to/Run-Workflows-from-the-CLI.md new file mode 100644 index 00000000..8eec978a --- /dev/null +++ b/docs/wiki/how-to/Run-Workflows-from-the-CLI.md @@ -0,0 +1,85 @@ +# Run Workflows from the CLI + +Once a workflow is configured, you run it from the command line through a small [Hydra](Configuration-and-Hydra) entry point. This guide covers running a configured callable, choosing which one to run, applying overrides, and inspecting the composed configuration. It assumes you have a config directory and an entry point as built in [Building an ETL Pipeline](Building-an-ETL-Pipeline) and [Composing an ETL Application](Composing-an-ETL-Application). + +## Set up the entry point + +An entry point wires Hydra to `ccflow`'s runner and explainer: + +```python +import hydra +from ccflow.utils.hydra import cfg_run, cfg_explain_cli + + +@hydra.main(config_path="config", config_name="base", version_base=None) +def main(cfg): + return cfg_run(cfg) + +def explain(): + cfg_explain_cli(config_path="config", config_name="base", hydra_main=main) +``` + +`cfg_run` executes the callable named by the configuration's top-level `callable` key. `cfg_explain_cli` opens a UI over the composed configuration without running anything. + +## Run a callable + +Select which callable to run with `+callable`, and pass a context with `+context`: + +```bash +# Run the 'extract' callable with an empty context: +python -m myapp +callable=extract +context=[] + +# Run it with a populated context: +python -m myapp +callable=extract +context=["http://lobste.rs"] +``` + +`+key=value` *adds* a key that is not already in the config; use it for `callable` and `context`, which are supplied at run time. + +## Override configuration values + +Use Hydra's [override grammar](https://hydra.cc/docs/advanced/override_grammar/basic/) to change any node of the composed configuration: + +```bash +# Change a nested value (++ forces an existing key): +python -m myapp +callable=extract +context=[] ++extract.publisher.name=lobsters + +# Point a stage at different inputs: +python -m myapp +callable=transform +context=[] ++transform.model.file=lobsters.html +``` + +- `++key=value` sets an existing key (force-override). +- `+key=value` appends a new key. +- `key=value` selects a [config group](Composing-an-ETL-Application#swapping-an-option) option. + +## Swap a config-group option + +If a subsystem is a config group, choose a different option by name — no `+`/`++`: + +```bash +python -m myapp +callable=extract +context=[] extract=file # use extract/file instead of the default +``` + +## Point at your own config directory + +Any Hydra app accepts a config directory and root config name on the command line, so a shared entry point can run different applications: + +```bash +python -m myapp --config-dir ./config --config-name pipeline +context=[] +``` + +## Inspect the composed configuration + +Before (or instead of) running, use the explain entry point to see exactly what Hydra composed — invaluable for confirming overrides landed where you expect: + +```bash +python -m myapp.explain +python -m myapp.explain ++extract.publisher.name=test +``` + + + +## See also + +- [Composing an ETL Application](Composing-an-ETL-Application) — build a config-group-driven app around this entry point. +- [Configuration and Hydra](Configuration-and-Hydra) — the composition and override model in depth. +- Hydra's [override grammar](https://hydra.cc/docs/advanced/override_grammar/basic/) — the full command-line syntax. diff --git a/docs/wiki/reference/Built-in-Models.md b/docs/wiki/reference/Built-in-Models.md new file mode 100644 index 00000000..a80a0f1b --- /dev/null +++ b/docs/wiki/reference/Built-in-Models.md @@ -0,0 +1,108 @@ +# Built-in Models + +The models, publishers, and evaluators that ship with `ccflow`. Use these off the shelf or as references for your own. For usage, see the [How-to Guides](How-to-Guides); for the base classes, see [Core Types](Core-Types). + +> [!NOTE] +> Some entries are marked *Coming Soon!* — they are part of the design and in the process of being open-sourced. + +## Models + +Beyond the `CallableModel` and `BaseModel` subclasses you write yourself, `ccflow` provides composition models used throughout the framework, including: + +- `PublisherModel` — runs a `CallableModel` and hands its result to a publisher in one step. +- `RetryModel` — wraps a `CallableModel` with a retry policy as a first-class graph node (see [Retry on Failure](Retry-on-Failure)). +- Narwhals-based models for cross-dataframe-library operations. + +Additional data-reading models are being open-sourced over time. + +## Publishers + +Publishers (`ccflow.publishers`) are models that write or send data. A common interface lets one be substituted for another purely through configuration. See [Bind Logic to Configs](Bind-Logic-to-Configs#write-a-custom-publisher) to write your own. + +| Name | Path | Description | +| :--------------------------- | :------------------ | :----------------------------------------------------------------------------------------- | +| `DictTemplateFilePublisher` | `ccflow.publishers` | Publish to a file after populating a Jinja template. | +| `GenericFilePublisher` | `ccflow.publishers` | Publish using a generic "dump" callable. Uses `smart_open`, so local and cloud paths work. | +| `JSONPublisher` | `ccflow.publishers` | Publish to a file in JSON format. | +| `PandasFilePublisher` | `ccflow.publishers` | Publish a pandas DataFrame using an appropriate `pd.DataFrame` method. | +| `NarwhalsFilePublisher` | `ccflow.publishers` | Publish a Narwhals DataFrame using an appropriate `nw.DataFrame` method. | +| `PicklePublisher` | `ccflow.publishers` | Publish data to a pickle file. | +| `PydanticJSONPublisher` | `ccflow.publishers` | Publish a pydantic model to a JSON file. | +| `YAMLPublisher` | `ccflow.publishers` | Publish to a file in YAML format. | +| `CompositePublisher` | `ccflow.publishers` | Decompose a model or dict into pieces and publish each separately. | +| `PrintPublisher` | `ccflow.publishers` | Print data using standard `print`. | +| `LogPublisher` | `ccflow.publishers` | Print data using standard logging. | +| `PrintJSONPublisher` | `ccflow.publishers` | Print data in JSON format. | +| `PrintYAMLPublisher` | `ccflow.publishers` | Print data in YAML format. | +| `PrintPydanticJSONPublisher` | `ccflow.publishers` | Print a pydantic model as JSON. | +| `ArrowDatasetPublisher` | *Coming Soon!* | | +| `PandasParquetPublisher` | *Coming Soon!* | | +| `PandasDeltaPublisher` | *Coming Soon!* | | +| `EmailPublisher` | *Coming Soon!* | | +| `MatplotlibFilePublisher` | *Coming Soon!* | | +| `PlotlyFilePublisher` | *Coming Soon!* | | +| `XArrayPublisher` | *Coming Soon!* | | +| `MLFlowArtifactPublisher` | *Coming Soon!* | | +| `MLFlowPublisher` | *Coming Soon!* | | + +## Evaluators + +Evaluators control *how* a `CallableModel` runs. Set one through `FlowOptions` (see [Core Types](Core-Types#flow-options)). Usage is in [Cache Results](Cache-Results) and [Retry on Failure](Retry-on-Failure). + +| Name | Path | Description | +| :---------------------------------- | :------------------ | :----------------------------------------------------------------- | +| `LazyEvaluator` | `ccflow.evaluators` | Runs the callable only once an attribute of the result is queried. | +| `LoggingEvaluator` | `ccflow.evaluators` | Logs information about evaluating the callable (the default). | +| `MemoryCacheEvaluator` | `ccflow.evaluators` | Caches results in memory. | +| `MultiEvaluator` | `ccflow.evaluators` | Combines multiple evaluators. | +| `GraphEvaluator` | `ccflow.evaluators` | Evaluates the dependency graph in topologically sorted order. | +| `RetryEvaluator` | `ccflow.evaluators` | Retries evaluation on failure with exponential backoff and jitter. | +| `ChunkedDateRangeEvaluator` | *Coming Soon!* | | +| `ChunkedDateRangeResultsAggregator` | *Coming Soon!* | | +| `DependencyTrackingEvaluator` | *Coming Soon!* | | +| `DiskCacheEvaluator` | *Coming Soon!* | | +| `ParquetCacheEvaluator` | *Coming Soon!* | | +| `RayChunkedDateRangeEvaluator` | *Coming Soon!* | | +| `RayCacheEvaluator` | *Coming Soon!* | | +| `RayGraphEvaluator` | *Coming Soon!* | | +| `RayDelayedDistributedEvaluator` | *Coming Soon!* | | + +### How cache keys are built + +`MemoryCacheEvaluator` uses `ccflow.evaluators.cache_key()`, which delegates to `ccflow.utils.compute_cache_token()`. A key combines a **data token** (for the serialized model/context payload) and a **behavior token** (for each callable/evaluator class that can affect the result). + +- For a direct `CallableModel` call, the key depends on `model.model_dump(mode="python")` and `compute_behavior_token(type(model))`. +- For a `ModelEvaluationContext`, the structural key depends on the context payload plus the function name (`__call__` vs `__deps__`), the behavior token of the model class, and the data/behavior tokens of any **non-transparent** evaluators in the chain. + +Transparent evaluators (those whose `is_transparent` returns `True`) are skipped, so wrapping a model in logging or timing does not change its cache identity. + +`compute_behavior_token()` hashes the class's Python-defined methods and also consults `__ccflow_tokenizer_deps__` for behavior defined outside the class body: + +```python +class MyModel(CallableModel): + __ccflow_tokenizer_deps__ = [helper, SharedLogic] +``` + +Entries may be functions or classes; class dependencies are tokenized recursively. Recursive class dependency graphs raise a `TypeError`. + +### `RetryEvaluator` parameters + +`RetryEvaluator` (and `RetryModel`, which share a `RetryPolicy`) support: + +| Parameter | Meaning | +| :-------------------- | :--------------------------------------------------------------------------- | +| `max_attempts` | Maximum number of attempts. | +| `wait_initial` | Wait before the first retry. | +| `wait_multiplier` | Backoff multiplier between retries. | +| `wait_max` | Cap on any single wait. | +| `wait_jitter` | Maximum random jitter added to a wait. | +| `max_delay` | Cap on *cumulative* waiting; no retry once the next backoff would exceed it. | +| `retry_exceptions` | Exceptions (classes or importable path strings) that trigger a retry. | +| `no_retry_exceptions` | Exceptions that must not be retried. | + +See [Retry on Failure](Retry-on-Failure) for usage. + +## See also + +- [Core Types](Core-Types) — the base classes these extend. +- [Flow Model](Flow-Model) — the `@Flow.model` API. diff --git a/docs/wiki/reference/Contexts-and-Results.md b/docs/wiki/reference/Contexts-and-Results.md new file mode 100644 index 00000000..75dc35ef --- /dev/null +++ b/docs/wiki/reference/Contexts-and-Results.md @@ -0,0 +1,82 @@ +# Contexts and Results + +The built-in context and result types. Contexts parameterize workflow steps; results carry their output. Both are `BaseModel` subclasses, so they validate and serialize like any config. See [Defining Workflows](Defining-Workflows) for usage. + +## Contexts + +All contexts derive from `ContextBase` and are frozen and hashable so they can serve as cache keys. + +| Type | Fields | Notes | +| :--------------------- | :-------------------------- | :------------------------------------------- | +| `NullContext` | none | For steps with no runtime parameters. | +| `GenericContext[T]` | `value: T` | Holds an arbitrary (optionally typed) value. | +| `DateContext` | `date` | Single date. | +| `DatetimeContext` | `dt` | Single datetime. | +| `DateRangeContext` | `start_date`, `end_date` | A date range. | +| `VersionedDateContext` | `date`, `entry_time_cutoff` | Date with an as-of cutoff. | +| `UniverseContext` | `universe` | A named universe. | +| `UniverseDateContext` | `universe`, `date` | Universe plus date. | +| `ModelDateContext` | `mode`, `date` | Model name plus date. | + +### Validation conveniences + +`NullContext` validates from `None` or an empty container: + +```python +NullContext.model_validate(None) # NullContext() +NullContext.model_validate([]) # NullContext() +``` + +`GenericContext[T]` coerces its value to `T`: + +```python +GenericContext[str].model_validate(100) # GenericContext[str](value='100') +``` + +Date-based contexts accept strings and tuples, including relative offsets (handy from the command line): + +```python +DateContext.model_validate("2025-01-01") # explicit date +DateContext.model_validate("0d") # today +DateRangeContext.model_validate(("-7d", "0d")) # last 7 days through today +``` + +## Results + +All results derive from `ResultBase`. + +| Type | Holds | +| :-------------------- | :----------------------------------------------------------------------------------------------------- | +| `GenericResult[T]` | An arbitrary (optionally typed) `value`. | +| `PandasResult` | A pandas `DataFrame`. | +| `NDArrayResult` | A NumPy array. | +| `ArrowResult` | A PyArrow table. | +| `XArrayResult` | An xarray structure. | +| `NarwhalsFrameResult` | A dataframe via [Narwhals](https://narwhals-dev.github.io/narwhals/), for cross-library compatibility. | + +### `GenericResult` + +Holds anything, with optional generic typing and boilerplate-reducing validation: + +```python +GenericResult(value={"x": "foo", "y": 5.0}) # any value +GenericResult[str](value="Any string") # typed +GenericResult.model_validate("Any string") # bare value validated into the wrapper +``` + +### Custom results + +Define a schema by subclassing `ResultBase`: + +```python +from ccflow import ResultBase + +class MyResult(ResultBase): + x: str + y: float +``` + +## See also + +- [Core Types](Core-Types) — `ContextBase`, `ResultBase`, and the surrounding machinery. +- [Defining Workflows](Defining-Workflows) — contexts and results in a running workflow. diff --git a/docs/wiki/reference/Core-Types.md b/docs/wiki/reference/Core-Types.md new file mode 100644 index 00000000..60186ff1 --- /dev/null +++ b/docs/wiki/reference/Core-Types.md @@ -0,0 +1,101 @@ +# Core Types + +Reference for the base classes and options at the center of `ccflow`. For conceptual orientation see [Core Concepts](Core-Concepts); for usage see the [Tutorials](Tutorials). + +## `BaseModel` + +The base class for all configuration in `ccflow`. Inherits from [pydantic](https://docs.pydantic.dev/latest/)'s `BaseModel` and adds: + +- serialization of the model's own type as `type_`, aliased to `_target_`, so a serialized config reconstructs as the correct subclass and is instantiable by Hydra; +- rejection of unknown/misnamed fields (extra inputs forbidden) by default; +- compatibility with Hydra instantiation and the `ModelRegistry`. + +All pydantic model features (validation, coercion, `model_dump`, `model_validate`, JSON schema) apply. + +## `ModelRegistry` + +A named, mutable collection of `BaseModel` instances that behaves like a `collections.abc.Mapping`. + +- `add(name, model, overwrite=False)` — register a model; raises if `name` exists unless `overwrite=True`; validates that `model` is a `BaseModel`. +- `__getitem__` / `get(name, default=...)` — retrieve by name; supports path syntax (`"a/b"`) across nested registries. +- `models` — a mapping of the top-level entries only. +- iteration / `len` — over all combined keys, including those in sub-registries. +- `clear()` — remove all entries (and reset registrations/dependencies of previously registered models). +- `load_config(dict)` — load a nested dictionary of configs, interpreting nested dicts (without `_target_`) as sub-registries and resolving string references against the root registry. +- `load_config_from_path(path, config_key=...)` — load configuration from Hydra files; `config_key` selects the subtree to register. + +Registries may contain other registries, forming a hierarchy. + +### `ModelRegistry.root()` + +Returns the singleton root registry. Name references in configs (e.g. `"data/source1"`) are resolved against this root, which is how `ccflow` performs dependency injection over shared instances. + +Models expose their placement in the registry: + +- `get_registrations()` — list of `(registry, name)` tuples. +- `get_registered_names()` — list of path strings. +- `get_registry_dependencies()` — the registered models this model depends on, found recursively. + +## `ContextBase` + +Base class for contexts — the runtime parameters that templatize a workflow. A `ContextBase` is a `BaseModel` and also a `ResultBase` (so a context can be produced as a step's result). Contexts are **frozen** (immutable) and hashable by default, so they can serve as cache keys. See [Contexts and Results](Contexts-and-Results) for the built-in context types. + +## `ResultBase` + +Base class for the results returned by workflow steps. A `ResultBase` is a `BaseModel`, giving uniform validation, serialization, and support for delayed/cached evaluation. See [Contexts and Results](Contexts-and-Results) for the built-in result types. + +## `CallableModel` + +A `BaseModel` that is called with a context and returns a result. Subclasses implement `__call__`, decorated with `@Flow.call`. + +- `context_type` — the type used to validate the incoming context; inferred from the `__call__` annotation by default, overridable as a property. +- `result_type` — the type used to validate the returned result; inferred by default, overridable as a property. +- `meta` — a `MetaData` instance (see below). + +Raises `TypeError` on call if no context type can be determined (neither an annotation nor a `context_type` property is provided). + +## The `Flow` decorators + +- `@Flow.call` — required on a `CallableModel.__call__`. Applies input/output type validation and is the seam through which the framework injects logging, caching, and evaluation. Accepts options such as `validate_result=False`. `@Flow.call(auto_context=True)` lets `__call__` declare context fields as keyword-only parameters instead of one context object. +- `@Flow.deps` — decorates a `__deps__` method that returns a `GraphDepList` (`List[Tuple[CallableModelType, List[ContextType]]]`), declaring explicit dependencies for graph evaluation. +- `@Flow.model` — turns a plain Python function into a `CallableModel`. See [Flow Model](Flow-Model). +- `@Flow.context_transform` — defines reusable contextual rewrites for `@Flow.model`. See [Flow Model](Flow-Model). + +## `MetaData` + +Metadata carried on every `CallableModel` via its `meta` attribute: + +- `name` — used by the logging evaluator to identify the model; set automatically when models are loaded from Hydra configs. +- `description` — free text, useful for cataloging models in a registry. +- `options` — a `FlowOptions` payload, allowing flow options to be specified in a config file and travel with the model instance. + +## Flow Options + +`FlowOptions` controls the behavior injected by `@Flow.call`. It can be set as decorator arguments, via `FlowOptionsOverride`, on `meta.options`, or per call with `_options`. + +| Field | Type | Default | Description | +| :---------------- | :---------------------- | :------ | :----------------------------------------------------------------------------------------------- | +| `log_level` | `int` | `10` | If no `evaluator` is set, uses a `LoggingEvaluator` at this level. | +| `verbose` | `bool` | `True` | Whether to use verbose logging. | +| `validate_result` | `bool` | `True` | Whether to validate the result against the model's `result_type` before returning. | +| `volatile` | `bool` | `False` | Whether the function is volatile (always returns a new value) and must be excluded from caching. | +| `cacheable` | `bool` | `False` | Whether results may be cached; caching is opt-in. | +| `evaluator` | `EvaluatorBase \| None` | `None` | A hook to set a custom evaluator. | + +### `FlowOptionsOverride` + +A context manager that applies `FlowOptions` within its scope. Accepts `options=...` plus optional `model_types=(...)` or `models=(...)` to restrict the override to specific model classes or instances. + +## Evaluator types + +- `EvaluatorBase` — base class for evaluators. An evaluator uses `ModelEvaluationContext` as its context type and does not use `@Flow.call`. Override `is_transparent(context) -> bool` to return `True` for evaluators that do not change the result (logging, timing), so cache-key computation skips them. +- `ModelEvaluationContext` — carries the model, context, function to evaluate (`__call__` or `__deps__`), and `FlowOptions`; calling it evaluates the function on the model with the context (ignoring options). + +See [Built-in Models](Built-in-Models#evaluators) for the concrete evaluators. + +## Publisher types + +- `BasePublisher` (`ccflow.publishers`) — base interface for components that send data somewhere; a common interface lets publishers be swapped through configuration. +- `PublisherModel` — a `CallableModel` that runs a `model` and hands its result to a `publisher` in one step. + +See [Built-in Models](Built-in-Models#publishers) for the concrete publishers. diff --git a/docs/wiki/Flow-Model.md b/docs/wiki/reference/Flow-Model.md similarity index 99% rename from docs/wiki/Flow-Model.md rename to docs/wiki/reference/Flow-Model.md index bd74f96b..2a5043a7 100644 --- a/docs/wiki/Flow-Model.md +++ b/docs/wiki/reference/Flow-Model.md @@ -44,7 +44,8 @@ result = prefilled.flow.compute() assert result.value == 23 ``` -> **Note:** When the function returns a plain value (like `int` above) instead +> [!NOTE] +> When the function returns a plain value (like `int` above) instead > of a `ResultBase` subclass, `@Flow.model` automatically wraps it in > `GenericResult`. Access the inner value with `.value`. @@ -481,7 +482,7 @@ function body on each call, use a normal `CallableModel` subclass instead of For class-based `CallableModel` methods that want to declare context fields as keyword-only parameters, see `Flow.call(auto_context=...)` in -[Workflows](Workflows#flow-decorator). +[Defining Workflows](Defining-Workflows#controlling-the-flow-decorator). ## Introspection diff --git a/docs/wiki/reference/Reference.md b/docs/wiki/reference/Reference.md new file mode 100644 index 00000000..d90ff67d --- /dev/null +++ b/docs/wiki/reference/Reference.md @@ -0,0 +1,11 @@ +# Reference + +Technical description of the pieces `ccflow` provides. These pages are for looking things up — neutral facts, not lessons. To learn the framework, start with the [Tutorials](Tutorials); to accomplish a task, see the [How-to Guides](How-to-Guides). + +- **[Core Types](Core-Types)** — the base classes and options at the heart of `ccflow`: `BaseModel`, `ModelRegistry`, `ContextBase`, `ResultBase`, `CallableModel`, the `Flow` decorators, `FlowOptions`, and metadata. +- **[Contexts and Results](Contexts-and-Results)** — the built-in context and result types. +- **[Built-in Models](Built-in-Models)** — the models, publishers, and evaluators that ship with `ccflow`, plus cache-key derivation. +- **[Flow Model](Flow-Model)** — the `@Flow.model` API for defining callable models from plain Python functions. + +> [!NOTE] +> Some items are marked *Coming Soon!* — they are part of the design and are in the process of being open-sourced. diff --git a/docs/wiki/tutorials/Building-a-Configurable-Calculator.md b/docs/wiki/tutorials/Building-a-Configurable-Calculator.md new file mode 100644 index 00000000..1bfbd197 --- /dev/null +++ b/docs/wiki/tutorials/Building-a-Configurable-Calculator.md @@ -0,0 +1,420 @@ +# Building a Configurable Calculator + +This is the capstone. You have configured models, defined workflows, built an ETL pipeline, and composed it into a config-group application. Now you will combine the **functional `@Flow.model` API** with everything you know about Hydra to build a small "calculator" you drive entirely from the command line: read input operands, apply a calculation chosen from a registry of functions, and emit the result through a choice of outputs — every part swappable and configurable from YAML or the CLI. Along the way the functional dependency graph will do real work. + +After this tutorial you should feel comfortable assembling your own system from these parts. Keep the [Flow Model](Flow-Model) reference open alongside this page for `@Flow.model` API details. + +> [!NOTE] +> The full source is in-tree at [`ccflow/examples/calculator`](https://github.com/Point72/ccflow/tree/main/ccflow/examples/calculator), and runs with `python -m ccflow.examples.calculator`. + +## Input vs. configuration + +A calculator reads *input* (the operands) and applies a *configured* operation (add an offset, scale by a factor, raise to an exponent). The functional API makes that split explicit, and the split is what makes calculations interchangeable: + +- **`FromContext[...]` parameters** are the runtime *input* — the operands, supplied per run. +- **Ordinary parameters** are the *configuration* — bound from YAML or the CLI, fixed for a run. + +Get that split right and each calculation becomes a drop-in replacement for the others, selectable and configurable purely from Hydra. + +## Writing the calculations + +Instead of subclassing `CallableModel`, decorate plain functions with `@Flow.model`. Each declares a context type for its input and exposes its configuration as ordinary parameters: + +```python +# ccflow/examples/calculator/functions.py +from ccflow import ContextBase, Flow, FromContext + + +class Numbers(ContextBase): + """Runtime input for the calculators: the operands to compute on.""" + values: list[float] = [] + + +@Flow.model(context_type=Numbers) +def add(values: FromContext[list[float]], offset: float = 0.0) -> float: + """Sum the input values, then add a configurable offset.""" + return sum(values) + offset + + +@Flow.model(context_type=Numbers) +def scale(values: FromContext[list[float]], factor: float = 1.0) -> float: + """Sum the input values, then multiply by a configurable factor.""" + return sum(values) * factor + + +@Flow.model(context_type=Numbers) +def power(values: FromContext[list[float]], exponent: float = 2.0) -> float: + """Raise each input value to a configurable exponent and sum the results.""" + return sum(value**exponent for value in values) +``` + +Read one signature and the contract is obvious: `values` comes from the runtime context (marked `FromContext`), while `offset` / `factor` / `exponent` are bound configuration. `@Flow.model` turns each function into a real `CallableModel` — the same registry, evaluator, caching, and Hydra machinery you have used all along — and wraps the plain `float` return in a `GenericResult`. + +> [!NOTE] +> Every `@Flow.model` with a `context_type` must have at least one `FromContext[...]` parameter; that is how the framework knows which inputs come from the context. + +## Selecting a calculation from a config group + +Make the calculations interchangeable from configuration. Each becomes an option in a `function` config group — a small YAML file that targets the function and sets its default configuration: + +**config/function/add.yaml** + +```yaml +_target_: ccflow.examples.calculator.functions.add +offset: 0.0 +``` + +**config/function/scale.yaml** + +```yaml +_target_: ccflow.examples.calculator.functions.scale +factor: 1.0 +``` + +**config/function/power.yaml** + +```yaml +_target_: ccflow.examples.calculator.functions.power +exponent: 2.0 +``` + +Using a `@Flow.model` function as a Hydra `_target_` calls it with the given fields (`add(offset=0.0)`), building a configured model instance. The base config selects a default and dispatches to whatever ends up under the `function` key. We start small and grow this file as the tutorial goes: + +**config/base.yaml** + +```yaml +# @package _global_ + +defaults: + - _self_ + - function: add + +callable: /function +``` + +Selecting `function: add` places `config/function/add.yaml` under the `function` key; loading the config registers that model at `/function`; and `callable: /function` tells the runner what to execute — the same [dispatch pattern](Composing-an-ETL-Application#dispatching-which-callable-to-run) from the ETL application, now dispatching a functional model. + +## Running it + +The entry point is the familiar Hydra runner: + +```python +# ccflow/examples/calculator/__main__.py +import hydra +from ccflow.utils.hydra import cfg_run + + +@hydra.main(config_path="config", config_name="base", version_base=None) +def main(cfg): + cfg_run(cfg) +``` + +Run the default calculation, passing the operands as the context: + +```bash +python -m ccflow.examples.calculator +context.values=[1,2,3] +#> [...][ccflow.utils.hydra][INFO] - {'value': 6.0, '_target_': 'ccflow.result.generic.GenericResult'} +``` + +`+context.values=[1,2,3]` provides the runtime input (`+` adds the `context` key); `add` sums it to `6` and applies its `offset` of `0`. + +### Swap the calculation + +Because `function` is a config group, name a different option by referencing the file using +the override grammar: + +```bash +python -m ccflow.examples.calculator function=scale +context.values=[1,2,3] +#> {'value': 6.0, ...} +``` + +> [!NOTE] +> **`+` or no `+`?** `function` is already in the defaults list, so you *override* the selection with `function=scale` (no `+`). If the base did **not** default-select a function, you would *append* with `+function=scale`. Same idea for any config group. + +### Configure the calculation + +Override its fields with dotted paths — the input stays the same, the calculation changes: + +```bash +python -m ccflow.examples.calculator function=scale function.factor=10 +context.values=[1,2,3] +#> {'value': 60.0, ...} + +python -m ccflow.examples.calculator function=power function.exponent=3 +context.values=[1,2,3] +#> {'value': 36.0, ...} +``` + +`function.factor=10` reaches into the composed `function` subtree. That is the input/configuration split at work: `+context.values=...` changes *what* you compute on, `function.=...` changes *how*. + +## Composing calculations in YAML + +A regular (non-`FromContext`) parameter can be fed by *another model*, letting you wire a graph in configuration. Add a `rounded` stage whose input is produced by an upstream calculation: + +```python +@Flow.model +def rounded(value: float, digits: int = 2) -> float: + """Round an upstream result to a configurable precision.""" + return round(value, digits) +``` + +**config/function/rounded.yaml** + +```yaml +_target_: ccflow.examples.calculator.functions.rounded +digits: 2 +value: + _target_: ccflow.examples.calculator.functions.power + exponent: 2.0 +``` + +`value` is a regular parameter bound to a `power` model. When `rounded` runs, `@Flow.model` first evaluates `power` with the ambient context, then passes its result in: + +```bash +python -m ccflow.examples.calculator function=rounded function.digits=1 +context.values=[1.5,2.5] +#> {'value': 8.5, ...} +``` + +`power` squares and sums `1.5` and `2.5` to `8.5`, and `rounded` rounds it — two functions composed entirely through YAML. + +## Making the dependency graph do real work + +That `rounded ← power` chain is linear, so a graph has nothing to optimize. The dependency graph earns its keep when a node is *reused*. Consider a `tail_ratio` — how far the largest value sits above the mean, relative to how far the smallest sits below. It needs `upper_gap` and `lower_gap`, and *both* need the `mean`: + +```python +@Flow.model(context_type=Numbers) +def mean(values: FromContext[list[float]]) -> float: + """Average of the input values.""" + return sum(values) / len(values) + + +@Flow.model(context_type=Numbers) +def upper_gap(values: FromContext[list[float]], center: float) -> float: + """How far the largest value sits above `center` (typically the mean).""" + return max(values) - center + + +@Flow.model(context_type=Numbers) +def lower_gap(values: FromContext[list[float]], center: float) -> float: + """How far the smallest value sits below `center` (typically the mean).""" + return center - min(values) + + +@Flow.model +def tail_ratio(upper: float, lower: float) -> float: + """Ratio of the upper gap to the lower gap.""" + return upper / lower +``` + +```mermaid +graph TD + tail_ratio --> upper_gap + tail_ratio --> lower_gap + upper_gap --> mean + lower_gap --> mean +``` + +That shared `mean` is a diamond. In YAML it is just nesting — `center` on each gap is fed by a `mean`: + +**config/function/tail_ratio.yaml** + +```yaml +_target_: ccflow.examples.calculator.functions.tail_ratio +upper: + _target_: ccflow.examples.calculator.functions.upper_gap + center: + _target_: ccflow.examples.calculator.functions.mean +lower: + _target_: ccflow.examples.calculator.functions.lower_gap + center: + _target_: ccflow.examples.calculator.functions.mean +``` + +You never wrote a `__deps__` method: with `@Flow.model`, wiring `mean` into each gap's `center` *is* the dependency declaration, and the decorator generates the `__deps__` that a class-based `CallableModel` declares by hand with `@Flow.deps` (as in [Defining Workflows](Defining-Workflows#evaluators-run-your-steps)). Because the graph is known, a [`GraphEvaluator`](Cache-Results#evaluate-a-dependency-graph) plus cache computes the shared `mean` **once** instead of once per branch. Instrument `mean` to watch it happen: + +```python +from ccflow import Flow, FlowOptionsOverride, FromContext +from ccflow.evaluators import GraphEvaluator, MemoryCacheEvaluator, MultiEvaluator +from ccflow.examples.calculator.functions import Numbers, lower_gap, tail_ratio, upper_gap + +calls = {"mean": 0} + + +@Flow.model(context_type=Numbers) +def counting_mean(values: FromContext[list[float]]) -> float: + calls["mean"] += 1 + return sum(values) / len(values) + + +m = counting_mean() +model = tail_ratio(upper=upper_gap(center=m), lower=lower_gap(center=m)) + +model({"values": [1, 2, 3, 10]}) +print("plain:", calls["mean"]) +#> plain: 2 + +calls["mean"] = 0 +evaluator = MultiEvaluator(evaluators=[GraphEvaluator(), MemoryCacheEvaluator()]) +with FlowOptionsOverride(options={"cacheable": True, "evaluator": evaluator}): + model({"values": [1, 2, 3, 10]}) +print("graph + cache:", calls["mean"]) +#> graph + cache: 1 +``` + +Plain evaluation runs `mean` twice, once per branch; the graph evaluator runs it once. Turn this on for *every* run by adding a `cli.model` block to `base.yaml` — the [shared execution policy](Composing-an-ETL-Application#sharing-an-execution-policy) that `cfg_run` applies to whatever it dispatches: + +```yaml +# added to config/base.yaml +cli: + model: + _target_: ccflow.FlowOptions + evaluator: + _target_: ccflow.evaluators.MultiEvaluator + evaluators: + - _target_: ccflow.evaluators.GraphEvaluator + - _target_: ccflow.evaluators.MemoryCacheEvaluator + cacheable: true +``` + +Now the diamond is deduplicated automatically: + +```bash +python -m ccflow.examples.calculator function=tail_ratio +context.values=[1,2,3,10] +#> {'value': 2.0, ...} +``` + +For `[1, 2, 3, 10]` the mean is `4`: the top sits `6` above it and the bottom `3` below, so the ratio is `2.0`. This is where the functional API pays off — compose functions, and the wiring becomes a real dependency graph that a graph evaluator runs efficiently. + +## Choosing how to emit the result + +So far the runner just logs the result. Let's make *emission* swappable too, with a second config group, `output`, whose options decide how the result leaves the program: print it, log it, or write it to a file. Output is independent of the calculation, so it composes with any `function`. + +Each option is a [`PublisherModel`](Bind-Logic-to-Configs#write-a-custom-publisher) — a `CallableModel` that runs an inner model and hands its result to a publisher. The inner model is the calculation registered at `/function`, referenced by name: + +**config/output/print.yaml** + +```yaml +_target_: ccflow.PublisherModel +model: /function +publisher: + _target_: ccflow.publishers.PrintPublisher +field: value +``` + +**config/output/log.yaml** + +```yaml +_target_: ccflow.PublisherModel +model: /function +publisher: + _target_: ccflow.publishers.LogPublisher + level: info +field: value +``` + +**config/output/write.yaml** + +```yaml +_target_: ccflow.PublisherModel +model: /function +publisher: + _target_: ccflow.publishers.GenericFilePublisher + name: result + suffix: .txt +field: value +``` + +`model: /function` is a registry reference — the [dependency injection](Configuring-Models#dependencies-and-dependency-injection) from earlier — so the output stage wraps whichever calculation is selected. `field: value` publishes the `value` field of the calculation's `GenericResult` (the bare number). + +Add the `output` group to the defaults and dispatch through it. Here is the base config in full, exactly as the example ships: + +**config/base.yaml** + +```yaml +# @package _global_ + +defaults: + - _self_ + - function: add + - output: print + +callable: /output + +cli: + model: + _target_: ccflow.FlowOptions + evaluator: + _target_: ccflow.evaluators.MultiEvaluator + evaluators: + - _target_: ccflow.evaluators.GraphEvaluator + - _target_: ccflow.evaluators.MemoryCacheEvaluator + cacheable: true +``` + +`callable: /output` now runs the publisher, which calls the selected `/function` and emits its result. With `output: print` as the default, a run both prints the bare value and logs the result object. Vary the output while holding the calculation at the default `add`: + +```bash +# Print (the default): the bare value on stdout, plus the runner's log line +python -m ccflow.examples.calculator +context.values=[1,2,3] +#> 6.0 +#> [...][ccflow.utils.hydra][INFO] - {'value': 6.0, ...} + +# Log through the ccflow logger instead: +python -m ccflow.examples.calculator output=log +context.values=[1,2,3] +#> [...][ccflow][INFO] - 6.0 + +# Write to a file (result.txt in the working directory): +python -m ccflow.examples.calculator output=write function=scale function.factor=10 +context.values=[1,2,3] +#> [...][ccflow.utils.hydra][INFO] - {'value': PosixPath('result.txt'), ...} +``` + +```bash +$ cat result.txt +60.0 +``` + +The calculator now has two independent dials: `function=` picks *what* to compute and `output=` picks *how to emit it*, and either is reconfigurable (`function.factor=10`, or a file publisher's `output.publisher.suffix=.csv`) without touching the other — a small matrix of choices, all from YAML and the command line. + +## Using it interactively + +The same configuration is available as live objects for research. Load it into the registry and run it from Python, applying the overrides you would pass on the CLI: + +```python +from ccflow import ModelRegistry +from ccflow.examples.calculator import load_config + +registry = load_config(overrides=["function=power", "function.exponent=3"]) +model = registry["/function"] +print(model.exponent) +#> 3.0 +print(model({"values": [1, 2, 3]}).value) +#> 36.0 +ModelRegistry.root().clear() +``` + +The selected calculation is a normal model at `/function`; its configuration is just attributes (`model.exponent`), and you run it by calling it with the input context. + +## Inspecting the configuration + +The explain entry point shows exactly what Hydra composed, including which config-group options exist and which was selected: + +```bash +python -m ccflow.examples.calculator.explain function=power --no-gui +``` + +## What you built — and what's next + +You built a command-line calculator where: + +- calculations are plain functions promoted to models with `@Flow.model`, +- the input/configuration split falls out of `FromContext[...]` vs. ordinary parameters, +- calculations are an interchangeable config group, selected with `function=` and configured with `function.=`, +- calculations compose into a dependency graph in YAML, and reused nodes (the shared `mean` in `tail_ratio`) are evaluated once under the graph evaluator, +- emission is a second, independent config group (`output=print` / `log` / `write`) built from publishers, +- and the whole thing runs from the CLI or interactively from Python. + +That is the combined flexibility of `ccflow` + Hydra in one small program: a registry of functions, per-function configuration, a real dependency graph, and CLI dispatch, all driven from flexible YAML. + +You now have every piece to build your own system — configuration ([Configuring Models](Configuring-Models)), workflows ([Defining Workflows](Defining-Workflows)), pipelines ([Building an ETL Pipeline](Building-an-ETL-Pipeline)), config-group composition ([Composing an ETL Application](Composing-an-ETL-Application)), and functional dispatch (this tutorial). Go build one. + +- [Flow Model](Flow-Model) — the complete `@Flow.model` reference, including `Dependency[...]`, `.flow.with_context(...)`, and context transforms. +- [Configuration and Hydra](Configuration-and-Hydra) — the reasoning behind config groups and dispatch. +- [How-to Guides](How-to-Guides) — recipes for caching, retries, and richer configuration as your system grows. diff --git a/docs/wiki/tutorials/Building-an-ETL-Pipeline.md b/docs/wiki/tutorials/Building-an-ETL-Pipeline.md new file mode 100644 index 00000000..662c2a4a --- /dev/null +++ b/docs/wiki/tutorials/Building-an-ETL-Pipeline.md @@ -0,0 +1,258 @@ +# Building an ETL Pipeline + +Now let's put [Defining Workflows](Defining-Workflows) to work and build an end-to-end [ETL](https://en.wikipedia.org/wiki/Extract,_transform,_load) pipeline. You will write three callable models — extract, transform, and load — and drive them from a single [Hydra](Configuration-and-Hydra) configuration file with a command-line entry point. + +The goal is to construct a set of callable models into which we pass contexts and get back results, and to define the pipeline via a static configuration file. This is a toy example, but it exercises every core idea end to end. + +> [!NOTE] +> The full source is in-tree at [`ccflow/examples/etl`](https://github.com/Point72/ccflow/tree/main/ccflow/examples/etl), and can be run directly with `python -m ccflow.examples.etl`. + +The pipeline does three things: + +- **Extract** a website's HTML and save it. +- **Transform** that HTML into a CSV of link names and URLs. +- **Load** that CSV into a queryable SQLite database. + +## A context for the pipeline + +Our first step takes a context, so we can point the pipeline at different sites at runtime. Contexts are ideal for the small handful of parameters that vary between runs. + +```python +from ccflow import ContextBase +from pydantic import Field + +class SiteContext(ContextBase): + site: str = Field(default="https://news.ycombinator.com") +``` + +Once the CLI is wired up, we will be able to select a context at runtime: + +```bash +etl-cli +context=[] # default: hacker news +etl-cli +context=["http://lobste.rs"] # a different site +``` + +## Extract + +The extract step queries a site over HTTP and returns its HTML: + +```python +from typing import Optional +from httpx import Client +from ccflow import CallableModel, Flow, GenericResult + + +class RestModel(CallableModel): + @Flow.call + def __call__(self, context: Optional[SiteContext] = None) -> GenericResult[str]: + context = context or SiteContext() + resp = Client().get(context.site, follow_redirects=True) + return GenericResult[str](value=resp.text) +``` + +The key elements: + +- It inherits from `CallableModel`, so it runs given a context and returns a result from a `@Flow.call`-decorated `__call__`. +- It takes a `SiteContext`. +- It returns a `GenericResult[str]`. (You could define a custom result type for stronger typing.) + +Before adding the other steps, let's get this one running from a CLI. + +## Wiring up a CLI + +`ccflow` provides helpers that connect [Hydra](Configuration-and-Hydra) to the callable-models framework: + +```python +import hydra +from ccflow.utils.hydra import cfg_run, cfg_explain_cli + + +@hydra.main(config_path="config", config_name="base", version_base=None) +def main(cfg): + cfg_run(cfg) + +def explain(): + cfg_explain_cli(config_path="config", config_name="base", hydra_main=main) +``` + +`cfg_run` takes a Hydra configuration hierarchy and executes its top-level `callable`. `cfg_explain_cli` launches a UI for browsing the composed configuration. Both are runnable directly from the shipped example: `python -m ccflow.examples.etl` and `python -m ccflow.examples.etl.explain`. + +## Configuring the extract step + +Hydra is driven by YAML. Here is a configuration for the extract step: + +```yaml +# ccflow/examples/etl/config/base.yaml +extract: + _target_: ccflow.PublisherModel + model: + _target_: ccflow.examples.etl.models.RestModel + publisher: + _target_: ccflow.publishers.GenericFilePublisher + name: raw + suffix: .html + field: value +``` + +The `extract` key is a `PublisherModel` — a `CallableModel` that runs the given `model` and hands its result to the given `publisher`. Here `RestModel` produces the HTML and `GenericFilePublisher` writes it to `raw.html`. (We could have written the file directly, but this shows how to reuse the same publisher anywhere.) + +Run it: + +```bash +python -m ccflow.examples.etl +callable=extract +context=[] +``` + +This calls `extract`, which calls `RestModel` and feeds the result to `GenericFilePublisher`, producing `raw.html`. Point it at a different site: + +```bash +python -m ccflow.examples.etl +callable=extract +context=["http://lobste.rs"] +``` + +Hydra's [override grammar](https://hydra.cc/docs/advanced/override_grammar/basic/) lets you tweak any node — for example, change the output file name: + +```bash +python -m ccflow.examples.etl +callable=extract +context=["http://lobste.rs"] ++extract.publisher.name=lobsters +``` + +## Transform + +The transform step reads an HTML file, extracts its links, and produces CSV: + +```python +from csv import DictWriter +from io import StringIO +from bs4 import BeautifulSoup +from ccflow import CallableModel, Flow, GenericResult, NullContext + +class LinksModel(CallableModel): + file: str + + @Flow.call + def __call__(self, context: NullContext) -> GenericResult[str]: + with open(self.file, "r") as f: + html = f.read() + + soup = BeautifulSoup(html, "html.parser") + links = [{"name": a.text, "url": href} for a in soup.find_all("a", href=True) if (href := a["href"]).startswith("http")] + + io = StringIO() + writer = DictWriter(io, fieldnames=["name", "url"]) + writer.writeheader() + writer.writerows(links) + return GenericResult[str](value=io.getvalue()) +``` + +## Load + +The load step reads a CSV file and loads it into SQLite: + +```python +import sqlite3 +from csv import DictReader +from pydantic import Field +from ccflow import CallableModel, Flow, GenericResult, NullContext + + +class DBModel(CallableModel): + file: str + db_file: str = Field(default="etl.db") + table: str = Field(default="links") + + @Flow.call + def __call__(self, context: NullContext) -> GenericResult[str]: + conn = sqlite3.connect(self.db_file) + cursor = conn.cursor() + cursor.execute(f"CREATE TABLE IF NOT EXISTS {self.table} (name TEXT, url TEXT)") + with open(self.file, "r") as f: + reader = DictReader(f) + for row in reader: + cursor.execute(f"INSERT INTO {self.table} (name, url) VALUES (?, ?)", (row["name"], row["url"])) + conn.commit() + return GenericResult[str](value="Data loaded into database") +``` + +## The full pipeline in one config + +Register all three steps in the same YAML file: + +```yaml +extract: + _target_: ccflow.PublisherModel + model: + _target_: ccflow.examples.etl.models.RestModel + publisher: + _target_: ccflow.publishers.GenericFilePublisher + name: raw + suffix: .html + field: value + +transform: + _target_: ccflow.PublisherModel + model: + _target_: ccflow.examples.etl.models.LinksModel + file: ${extract.publisher.name}${extract.publisher.suffix} + publisher: + _target_: ccflow.publishers.GenericFilePublisher + name: extracted + suffix: .csv + field: value + +load: + _target_: ccflow.examples.etl.models.DBModel + file: ${transform.publisher.name}${transform.publisher.suffix} + db_file: etl.db + table: links +``` + +Notice the `transform` step references the extract step's output with Hydra/OmegaConf [interpolation](https://omegaconf.readthedocs.io/en/2.3_branch/usage.html#variable-interpolation): `${extract.publisher.name}${extract.publisher.suffix}` resolves to `raw.html`. The `load` step needs no publisher — it writes the database directly. + +## Running the pipeline + +Run each step the same way, with overrides as needed: + +```bash +# Transform +python -m ccflow.examples.etl +callable=transform +context=[] + +# Transform with overrides: read lobsters.html, write lobsters.csv +python -m ccflow.examples.etl +callable=transform +context=[] ++transform.model.file=lobsters.html ++transform.publisher.name=lobsters + +# Load +python -m ccflow.examples.etl +callable=load +context=[] + +# Load with overrides: read lobsters.csv into an in-memory database +python -m ccflow.examples.etl +callable=load +context=[] ++load.file=lobsters.csv ++load.db_file=":memory:" +``` + +## Visualizing the configuration + +Because Hydra loads all this into a single registry, you can inspect the resolved configuration with the explain entry point: + +```bash +python -m ccflow.examples.etl.explain +``` + + + +Combine it with overrides to confirm everything resolves as intended: + +```bash +python -m ccflow.examples.etl.explain ++extract.publisher.name=test +``` + + + +## What you learned + +- Each ETL stage is a plain `CallableModel`, wired together through configuration. +- `PublisherModel` pairs a model with a publisher to compute-and-write in one step. +- A single YAML file, with interpolation between stages, defines the whole pipeline. +- One CLI runs any stage, and the explain UI shows exactly what was composed. + +## Next steps + +You wrote the whole pipeline in one file. Real applications split configuration into swappable pieces and dispatch between many workflows from the command line. That is the subject of the next tutorial. + +- [Composing an ETL Application](Composing-an-ETL-Application) — break this into config groups and build a reusable, CLI-driven application. +- [Configuration and Hydra](Configuration-and-Hydra) — why this composition style is valuable. +- [Run Workflows from the CLI](Run-Workflows-from-the-CLI) — a focused reference for running and overriding. diff --git a/docs/wiki/tutorials/Composing-an-ETL-Application.md b/docs/wiki/tutorials/Composing-an-ETL-Application.md new file mode 100644 index 00000000..19a53d1a --- /dev/null +++ b/docs/wiki/tutorials/Composing-an-ETL-Application.md @@ -0,0 +1,254 @@ +# Composing an ETL Application + +In [Building an ETL Pipeline](Building-an-ETL-Pipeline) you wrote a whole pipeline in one YAML file. That is fine for one pipeline, but real applications need to *swap* pieces — a different source here, a different output there, a durable cache in production and a no-op cache in tests — and to run *many* workflows from one command. This tutorial grows the pipeline into a reusable, config-group-driven application with command-line dispatch. + +By the end you will have built the skeleton of a reusable ETL *toolkit*: a package of models plus a library of config groups that downstream applications assemble and run through a single entry point. The reasoning behind this style is in [Configuration and Hydra](Configuration-and-Hydra); here we build it. + +## From one file to config groups + +Recall the single-file config from the last tutorial. The first move is to give each stage its own file inside a **config group** — a directory of interchangeable options for one slice of the app. + +``` +config/ + base.yaml + extract/ + rest.yaml + transform/ + links.yaml + load/ + db.yaml +``` + +`base.yaml` composes the pieces with a `defaults` list: + +```yaml +# config/base.yaml +defaults: + - extract: rest + - transform: links + - load: db +``` + +Each stage lives in its own file — an *option* within its group: + +```yaml +# config/extract/rest.yaml +_target_: ccflow.PublisherModel +model: + _target_: ccflow.examples.etl.models.RestModel +publisher: + _target_: ccflow.publishers.GenericFilePublisher + name: raw + suffix: .html +field: value +``` + +```yaml +# config/transform/links.yaml +_target_: ccflow.PublisherModel +model: + _target_: ccflow.examples.etl.models.LinksModel + file: ${extract.publisher.name}${extract.publisher.suffix} +publisher: + _target_: ccflow.publishers.GenericFilePublisher + name: extracted + suffix: .csv +field: value +``` + +```yaml +# config/load/db.yaml +_target_: ccflow.examples.etl.models.DBModel +file: ${transform.publisher.name}${transform.publisher.suffix} +db_file: etl.db +table: links +``` + +This composes to exactly the same configuration as before — but now each concern is a small, independently reviewable file. + +## Swapping an option + +The payoff appears when a group has more than one option. Add a second way to extract — reading from a local file instead of over HTTP: + +```yaml +# config/extract/file.yaml +_target_: ccflow.PublisherModel +model: + _target_: ccflow.examples.etl.models.LocalReadModel # your own model + path: ./raw.html +publisher: + _target_: ccflow.publishers.GenericFilePublisher + name: raw + suffix: .html +field: value +``` + +Now the extract subsystem is swappable by name, at the command line, with nothing else changing: + +```bash +python -m ccflow.examples.etl +callable=extract # uses extract/rest (the default) +python -m ccflow.examples.etl +callable=extract extract=file # swaps in extract/file +``` + +That is the essence of config-group composition: an application is a small matrix of choices, and a run is one path through it. + +## Dispatching which callable to run + +You have already used `+callable=extract` to choose *which* step to run. That works because the shared entry point runs whatever the top-level `callable` key names. Making `callable` a first-class, overridable key is what turns one entry point into a dispatcher over a whole catalog of workflows. + +A common pattern is to compute `callable` from an optional selection so the same app can run a task directly *or* wrapped in something else. `ccflow` uses OmegaConf's `oc.select` resolver for this: + +```yaml +# config/base.yaml +defaults: + - _self_ + - extract: rest + - transform: links + - load: db + +# The concrete task the app runs by default: +task: ${load} + +# Run an optional wrapper if one is selected, otherwise the task itself: +callable: ${oc.select:wrapper,/task} +``` + +`${oc.select:wrapper,/task}` resolves to `wrapper` if a `wrapper` key has been selected, and otherwise falls back to the registered `/task`. Selecting a wrapper is then just another config-group choice (`+wrapper=/wrappers/retry`, say), and the same command runs either shape. Leading-slash names like `/task` refer to models by their path in the root registry. + +## Sharing an execution policy + +An application usually wants the *same* execution behavior everywhere — graph evaluation, memory caching, and logging — regardless of which workflow runs. Set it once with a shared `FlowOptions` block that the entry point applies: + +```yaml +# config/base.yaml (continued) +cli: + model: + _target_: ccflow.FlowOptions + evaluator: + _target_: ccflow.evaluators.MultiEvaluator + evaluators: + - _target_: ccflow.evaluators.GraphEvaluator + - _target_: ccflow.evaluators.MemoryCacheEvaluator + - _target_: ccflow.evaluators.LoggingEvaluator + cacheable: true +``` + +Now every run of this application evaluates its dependency graph, caches repeated sub-results, and logs — because the execution strategy is configuration, not code. See [Cache Results](Cache-Results) for how the graph and cache evaluators cooperate. + +## Packaging groups for reuse + +Everything so far lived beside one application. To build a *toolkit* — reusable ETL building blocks that many applications assemble — move the models and their config groups into an installable package. + +Give the package a config directory of domain-neutral, swappable groups: + +``` +mytoolkit/ + __init__.py # exports your CallableModels, publishers, etc. + cli.py # the shared entry point (below) + config/ + base.yaml # sensible defaults, marked global + cache/ + noop.yaml + execution/ + default.yaml + credentials/ + default.yaml + calendars/ + default.yaml +``` + +Mark the package base so its keys land at the root of any config that includes it: + +```yaml +# mytoolkit/config/base.yaml +# @package _global_ + +defaults: + - _self_ + - cache: noop + - execution: default + - credentials: default +``` + +The `# @package _global_` directive places this file's content at the top level rather than nested under `base`, which is what you want for a shared base that defines top-level keys. Each group (`cache`, `execution`, ...) ships a safe default option, and installed connector packages can contribute more options to the same groups — a package that provides `cache=redis` becomes selectable simply by being installed. + +A downstream application then keeps only *its* configuration and pulls the toolkit's groups in via a search path: + +```yaml +# app/config/pipeline.yaml +defaults: + - _self_ + - /cache: noop # a group provided by the toolkit + - /execution: default + +hydra: + searchpath: + - pkg://mytoolkit.config + +model: + _target_: myapp.MyPipeline + +task: ${model} +callable: ${oc.select:wrapper,/task} +``` + +`pkg://mytoolkit.config` tells Hydra to look inside the installed package for config groups, so the application composes its own file together with the toolkit's shared groups. Switching a subsystem — say, from the no-op cache to a durable one contributed by a connector package — is then a one-word change (`cache=redis`) with no code edits. + +## A shared entry point + +Finally, give the toolkit one command that every application uses. Build it on the same helpers from [Building an ETL Pipeline](Building-an-ETL-Pipeline): + +```python +# mytoolkit/cli.py +import hydra +from ccflow.utils.hydra import cfg_run, cfg_explain_cli + + +@hydra.main(config_path="config", config_name="base", version_base=None) +def main(cfg): + return cfg_run(cfg) + + +def explain(): + cfg_explain_cli(config_path="config", config_name="base", hydra_main=main) +``` + +Expose them as console scripts so they install as real commands: + +```toml +# pyproject.toml +[project.scripts] +cc-run = "mytoolkit.cli:main" +cc-run-explain = "mytoolkit.cli:explain" +``` + +Because these are ordinary Hydra apps, any downstream application can point the same command at its own config directory: + +```bash +# Run an application's pipeline through the shared entry point: +cc-run --config-dir ./app/config --config-name pipeline +context=[] + +# Swap a subsystem by naming a different group option: +cc-run --config-dir ./app/config --config-name pipeline cache=redis +context=[] + +# Inspect the composed configuration without running it: +cc-run-explain --config-dir ./app/config --config-name pipeline +``` + +One command, one place to look, and a matrix of config-group choices behind it. + +## What you built + +- A config directory where each subsystem is a **config group** of swappable options. +- Command-line **dispatch** of which callable runs, via a `callable` key resolved with `oc.select`. +- A shared **execution policy** applied to every run through a `FlowOptions` block. +- A reusable **toolkit**: models plus config groups shipped in a package, pulled into applications with `pkg://` search paths, and driven by one console entry point. + +This is the shape of a production `ccflow` application: a versioned base, swappable subsystems, and a single dispatchable command — with the same objects still available interactively for research. + +## Next steps + +- [Building a Configurable Calculator](Building-a-Configurable-Calculator) — the capstone tutorial: bring in the functional `@Flow.model` API and drive a registry of functions entirely from the CLI. +- [Configuration and Hydra](Configuration-and-Hydra) — the reasoning behind config groups, packages, and dispatch. +- [Run Workflows from the CLI](Run-Workflows-from-the-CLI) — a focused guide to running, overriding, and explaining. +- [Cache Results](Cache-Results) and [Retry on Failure](Retry-on-Failure) — reliability and performance for the workflows you dispatch. diff --git a/docs/wiki/tutorials/Configuring-Models.md b/docs/wiki/tutorials/Configuring-Models.md new file mode 100644 index 00000000..8bca9a35 --- /dev/null +++ b/docs/wiki/tutorials/Configuring-Models.md @@ -0,0 +1,310 @@ +# Configuring Models + +In [First Steps](First-Steps) you loaded a couple of configs into a registry. This tutorial slows down and builds the configuration framework up properly: typed schemas, hierarchy, serialization, the registry, and dependency injection. By the end you will be able to define your own configuration objects and wire them together, both interactively and from files. + +Follow along in a Python session. Every block runs as shown. We will lean on [pydantic](https://docs.pydantic.dev/latest/) throughout — `ccflow.BaseModel` is a pydantic model with a few `ccflow` conveniences added. + +Start with these imports: + +```python +from ccflow import BaseModel, ModelRegistry +from datetime import date +from pathlib import Path +from pprint import pprint +``` + +## Your first config + +Pydantic calls its classes "Models", so we do too — think of a model as a *configurable class*. Define one by subclassing `BaseModel`: + +```python +class MyFileConfig(BaseModel): + file: Path + asof: date = date(2024, 1, 1) + description: str = "" +``` + +That is a schema. Create an instance and watch pydantic conform the inputs to the declared types: + +```python +c = MyFileConfig(file="sample.txt") +print(c) +#> MyFileConfig(file=PosixPath('sample.txt'), asof=datetime.date(2024, 1, 1), description='') +``` + +Configs are mutable by default, so you can adjust them after construction: + +```python +c.description = "Sample description" +print(c) +#> MyFileConfig(file=PosixPath('sample.txt'), asof=datetime.date(2024, 1, 1), description='Sample description') +``` + +You can also build one from a dictionary — useful when configuration comes from a file: + +```python +config = {"file": "sample.txt", "asof": "2024-02-02"} +print(MyFileConfig.model_validate(config)) +#> MyFileConfig(file=PosixPath('sample.txt'), asof=datetime.date(2024, 2, 2), description='') +``` + +Notice the string path became a `PosixPath` and the string date became a `datetime.date`. + +The point of a schema is that mistakes are caught early. A value that cannot be conformed raises immediately: + +```python +try: + c.asof = "foo" +except ValueError as e: + print(e) +#> 1 validation error for MyFileConfig +# asof +# Input should be a valid date or datetime, input is too short ... +``` + +`ccflow.BaseModel` goes one step further than plain pydantic and rejects unknown or misnamed fields, so a typo cannot silently create a stray attribute: + +```python +try: + MyFileConfig(file="sample.txt", AsOf=date(2024, 1, 1)) +except ValueError as e: + print(e) +#> 1 validation error for MyFileConfig +# AsOf +# Extra inputs are not permitted ... +``` + +## Hierarchical configs + +Configuration is naturally nested, and models compose. Define a config whose fields are themselves configs: + +```python +class MyTransformConfig(BaseModel): + x: MyFileConfig + y: MyFileConfig = None + param: float = 0. +``` + +You can build it by composition: + +```python +x = MyFileConfig(file="source1.csv") +y = MyFileConfig(file="source2.csv") +transform = MyTransformConfig(x=x, y=y, param=1.) +print(transform.x.file) +#> source1.csv +``` + +Because this is ordinary object composition, editing the local `x` edits the object inside `transform`: + +```python +x.description = "First Source" +print(transform.x.description) +#> First Source +``` + +Keep that behavior in mind — it becomes important once you register configs. + +Pydantic will also coerce nested dictionaries into the declared types, so you can pass raw data all the way down: + +```python +print(MyTransformConfig(x={"file": "source1.csv", "asof": "2024-02-02"})) +#> MyTransformConfig(x=MyFileConfig(file=PosixPath('source1.csv'), asof=datetime.date(2024, 2, 2), description=''), y=None, param=0.0) +``` + +## Serializing configs + +Pydantic serializes models richly: + +```python +pprint(transform.model_dump(mode="json")) +``` + +`ccflow` adds one thing to the default: it records each model's `type_` alongside its fields. That lets you reconstruct the correct subclass from a serialized config: + +```python +transform2 = BaseModel.model_validate(transform.model_dump(mode="json")) +assert isinstance(transform2, MyTransformConfig) +``` + +For [Hydra](Configuration-and-Hydra) compatibility, that type field is also aliased to `_target_`: + +```python +pprint(transform.model_dump(mode="json", by_alias=True)) +#> {'param': 1.0, +# '_target_': '__main__.MyTransformConfig', +# 'x': {..., '_target_': '__main__.MyFileConfig'}, +# 'y': {..., '_target_': '__main__.MyFileConfig'}} +``` + +## Inheritance and templatization + +Because configs are classes, you can share fields through inheritance: + +```python +class DateRangeMixin(BaseModel): + start_date: date + end_date: date + +class RegionMixin(BaseModel): + region: str + +class MyConfig(DateRangeMixin, RegionMixin): + parameter: int + +print(MyConfig(parameter=4, region="US", start_date=date(2022, 1, 1), end_date=date(2023, 1, 1))) +#> MyConfig(region='US', start_date=datetime.date(2022, 1, 1), end_date=datetime.date(2023, 1, 1), parameter=4) +``` + +For generic/templated configs, see pydantic's docs on [Generic Models](https://docs.pydantic.dev/latest/concepts/models/#generic-models). + +## Registering configurations + +A `ModelRegistry` is a named collection of configs — a catalog. Create one and add configs to it: + +```python +registry = ModelRegistry(name="My Raw Data") +registry.add("source1", MyFileConfig(file="source1.csv", description="First")) +registry.add("source2", MyFileConfig(file="source2.csv", description="Second")) + +print(list(registry)) +#> ['source1', 'source2'] +``` + +The registry validates what goes in — only models are allowed: + +```python +try: + registry.add("bad_data", {"foo": 5, "bar": 6}) +except TypeError as e: + print(e) +#> model must be a child class of , not ''. +``` + +Look configs up with `__getitem__` or `get`, and note that adding an existing name needs `overwrite=True`: + +```python +assert registry["source1"] is registry.get("source1", default=None) + +try: + registry.add("source1", registry["source1"]) +except ValueError as e: + print(e) +#> Cannot add 'source1' to 'My Raw Data' as it already exists! +``` + +Registries can contain other registries, and a single **root** registry is available as a singleton to tie everything together: + +```python +root = ModelRegistry.root() +assert root is ModelRegistry.root() # singleton +root.add("data", registry, overwrite=True) + +print(list(root)) +#> ['data', 'data/source1', 'data/source2'] +``` + +From the root you can reach any config three equivalent ways: + +```python +root["data"]["source1"] # dictionary syntax +root["data/source1"] # path syntax +root.get("data").get("source1") # getter syntax +``` + +## Dependencies and dependency injection + +Here is where the registry earns its keep. A config can depend on another *by its name in the root registry*, and `ccflow` resolves the name to the actual instance: + +```python +root = ModelRegistry.root() +root.add("data", registry, overwrite=True) + +new_config = MyTransformConfig(x="data/source1") +print(new_config.x.file) +#> source1.csv +``` + +Because the reference resolves to the shared instance, editing the source edits it here too: + +```python +root["data"]["source1"].description = "Test" +assert new_config.x.description == "Test" +``` + +This is dependency injection: `new_config` did not need to know how `source1` was built, only its name. Register the composite and keep wiring by simple assignment: + +```python +root.add("transform", new_config, overwrite=True) +root["transform"].y = "data/source2" +assert root["transform"].y.file == Path("source2.csv") +``` + +Configs can report where they are registered and what they depend on: + +```python +print(new_config.get_registered_names()) +#> ['/transform'] +print(new_config.get_registry_dependencies()) +#> [['/data/source1'], ['/data/source2']] +``` + +> This shared-instance behavior is a deliberate difference from Hydra's `${...}` interpolation, which *copies* configuration. See [Configuration and Hydra](Configuration-and-Hydra) for why that matters. + +## Loading configs from data and files + +You rarely add configs one call at a time. `load_config` takes a nested dictionary and loads the whole thing — interpreting nested dictionaries as sub-registries and resolving string references: + +```python +all_configs = { + "data": { + "source1": {"_target_": "__main__.MyFileConfig", "file": "source1.csv", "description": "First"}, + "source2": {"_target_": "__main__.MyFileConfig", "file": "source2.csv", "description": "Second"}, + }, + "transform": { + "_target_": "__main__.MyTransformConfig", + "param": 1.0, + "x": "data/source1", + "y": "data/source2", + }, +} +root = ModelRegistry.root().clear() +root.load_config(all_configs) +print(list(root)) +#> ['data', 'data/source1', 'data/source2', 'transform'] +``` + +The `_target_` keys are what let `ccflow` (and Hydra) know which class to build. In fact, a config dict with `_target_` is exactly what `hydra.utils.instantiate` consumes: + +```python +from hydra.utils import instantiate +config = {"_target_": "__main__.MyTransformConfig", "param": 1.0, + "x": {"file": "source1.csv", "description": "First"}, "y": {"file": "source2.csv"}} +print(instantiate(config).param) +#> 1.0 +``` + +To load configuration straight from Hydra files, use `load_config_from_path`, pointing `config_key` at the part of the file that holds the registry: + +```python +import ccflow.examples +root = ModelRegistry.root().clear() +absolute_path = Path(ccflow.examples.__file__).parent / "config/conf.yaml" +root.load_config_from_path(path=absolute_path, config_key="registry") +``` + +Hydra's own [documentation](https://hydra.cc/docs/intro/) covers how to author those files; the [Composing an ETL Application](Composing-an-ETL-Application) tutorial builds a full file-based application step by step. + +## What you learned + +- `BaseModel` gives you typed, validated, self-documenting configuration. +- Configs compose into hierarchies and serialize round-trip via `_target_`. +- The `ModelRegistry` catalogs configs; the root registry links them by name. +- Name references are dependency injection over *shared instances*. +- `load_config` and `load_config_from_path` load whole trees from data or Hydra files. + +## Next steps + +- [Defining Workflows](Defining-Workflows) — make these configuration objects runnable. +- [Configure Complex Values](Configure-Complex-Values) — custom validation, Jinja/SQL templates, expressions, arrays, and arbitrary types in configs. +- [Bind Logic to Configs](Bind-Logic-to-Configs) — attach business logic to configuration classes. diff --git a/docs/wiki/tutorials/Defining-Workflows.md b/docs/wiki/tutorials/Defining-Workflows.md new file mode 100644 index 00000000..80f11a68 --- /dev/null +++ b/docs/wiki/tutorials/Defining-Workflows.md @@ -0,0 +1,238 @@ +# Defining Workflows + +So far your configuration objects have only held data. This tutorial makes them *run*. You will meet the three ingredients of a workflow step — a **result**, a **context**, and a **callable model** — build one, run it, and see how the `@Flow.call` decorator adds type checking, logging, and other behavior behind the scenes. + +Follow along in a Python session. For the full catalog of built-in results, contexts, and evaluators, see the [Reference](Reference); this tutorial teaches the pattern with a few representative examples. + +A workflow step in `ccflow` is a `CallableModel`: something you call *with a context* that returns *a result*. Three abstractions make this work: + +- a **result** type to hold what a step returns, +- a **context** type to parameterize the step at runtime, +- the **`@Flow.call`** decorator, through which the framework injects type checking, logging, caching, and alternative evaluation. + +## Results + +Every step returns a `ResultBase`. The simplest is `GenericResult`, which holds anything in its `value`: + +```python +from ccflow import GenericResult +print(GenericResult(value="Anything goes here")) +#> GenericResult(value='Anything goes here') +``` + +You can ask for type safety on the value using Python generics: + +```python +result = GenericResult[str](value="Any string") + +try: + GenericResult[str](value={"x": "foo", "y": 5.0}) +except ValueError as e: + print(e) +#> 1 validation error for GenericResult[str] ... +``` + +Pydantic validation also cuts boilerplate — a bare value is validated into the wrapper: + +```python +print(GenericResult.model_validate("Any string")) +#> GenericResult(value='Any string') +``` + +When you know the shape of your output, define a proper result schema by subclassing `ResultBase`: + +```python +from ccflow import ResultBase + +class MyResult(ResultBase): + x: str + y: float + +print(MyResult(x="foo", y=5.0)) +#> MyResult(x='foo', y=5.0) +``` + +`ccflow` ships typed results for common data structures (pandas, numpy, Arrow, xarray, Narwhals). The [Contexts and Results](Contexts-and-Results) reference lists them all. + +## Contexts + +A context carries the parameters that vary between runs. Some steps need none — use `NullContext`: + +```python +from ccflow import NullContext +print(NullContext()) +#> NullContext() +``` + +`GenericContext` mirrors `GenericResult` for ad-hoc parameters: + +```python +from ccflow import GenericContext +print(GenericContext[str].model_validate(100)) +#> GenericContext[str](value='100') +``` + +And you define your own when a workflow has specific parameters: + +```python +from ccflow import ContextBase +from datetime import datetime + +class LocationTimestampContext(ContextBase): + latitude: float + longitude: float + timestamp: datetime +``` + +Contexts are **frozen** (immutable) and hashable by default, so the framework can use them as cache keys. `ccflow` also provides date-oriented contexts (`DateContext`, `DateRangeContext`, and more) with convenient validation — see the [Contexts and Results](Contexts-and-Results) reference. + +## Your first callable model + +Put the pieces together with a `CallableModel`. You implement `__call__` as a function of the context and decorate it with `@Flow.call`. Here is the classic [FizzBuzz](https://en.wikipedia.org/wiki/Fizz_buzz) problem as a model: + +```python +from ccflow import CallableModel, Flow, GenericResult, GenericContext + +class FizzBuzzModel(CallableModel): + fizz: str = "Fizz" + buzz: str = "Buzz" + + @Flow.call + def __call__(self, context: GenericContext[int]) -> GenericResult[list[int | str]]: + n = context.value + result = [] + for i in range(1, n + 1): + if i % 3 == 0 and i % 5 == 0: + result.append(f"{self.fizz}{self.buzz}") + elif i % 3 == 0: + result.append(self.fizz) + elif i % 5 == 0: + result.append(self.buzz) + else: + result.append(i) + return result + +model = FizzBuzzModel() +print(model(15)) +#> GenericResult[list[Union[int, str]]](value=[1, 2, 'Fizz', 4, 'Buzz', 'Fizz', 7, 8, 'Fizz', 'Buzz', 11, 'Fizz', 13, 14, 'FizzBuzz']) +``` + +Two things worth noticing. You called `model(15)` with a bare integer, not a `GenericContext[int]` — and you returned a bare list, not a `GenericResult`. The `@Flow.call` decorator did the conversions for you. + +The model's fields (`fizz`, `buzz`) are configuration; the context (`15`) is the runtime parameter. That split is the whole idea: configure once, run across many contexts. + +## Type checking comes for free + +`@Flow.call` runs pydantic validation on the way in and out, so invalid inputs fail clearly: + +```python +try: + model("not an integer") +except ValueError as e: + print(e) +#> 1 validation error for GenericContext[int] ... +``` + +By default the decorator infers the context and result types from your `__call__` signature: + +```python +print(model.context_type) +#> +print(model.result_type) +#> +``` + +When the types depend on configuration, override the `context_type` / `result_type` properties instead of annotating: + +```python +from typing import Type + +class DynamicTypedModel(CallableModel): + input_type: Type + output_type: Type + + @property + def context_type(self): + return GenericContext[self.input_type] + + @property + def result_type(self): + return GenericResult[self.output_type] + + @Flow.call + def __call__(self, context): + return context.value + +print(DynamicTypedModel(input_type=int, output_type=str)(5)) +#> GenericResult[str](value='5') +``` + +## Controlling the Flow decorator + +`@Flow.call` is the seam where framework behavior is layered on, controlled by `FlowOptions`. You can set options four ways: as arguments to the decorator, via the `FlowOptionsOverride` context manager, on the model's `meta.options`, or per call with `_options`. + +Turn off result validation for one model: + +```python +class NoValidationModel(CallableModel): + @Flow.call(validate_result=False) + def __call__(self, context: GenericContext[str]) -> GenericResult[float]: + return "foo" + +print(NoValidationModel()("foo")) +#> foo +``` + +Raise the log level for a call and every sub-call, scoped by a context manager: + +```python +import logging +from ccflow import FlowOptionsOverride + +model = FizzBuzzModel() +with FlowOptionsOverride(options={"log_level": logging.WARN}): + _ = model(15) +#[FizzBuzzModel]: Start evaluation of __call__ on GenericContext[int](value=15). +#[FizzBuzzModel]: End evaluation of __call__ on GenericContext[int](value=15) (time elapsed: ...). +``` + +Or pass options to a single call: + +```python +_ = model(15, _options={"log_level": logging.WARN}) +``` + +The full set of options lives on the `FlowOptions` schema — see [Flow Options](Core-Types#flow-options) in the reference. The `meta` attribute on every `CallableModel` also carries a `name` and `description` (set automatically when models load from Hydra configs) and can hold `options` so they travel with the model in a config file. + +## Evaluators run your steps + +You have been running models the "standard" way — Python calls `__call__` directly. An **evaluator** changes *how* a model runs. The default logs each evaluation; others cache, evaluate an explicit dependency graph, retry on failure, or distribute work. Because an evaluator is set through `FlowOptions`, adding these behaviors does not change your step at all. + +You set an evaluator the same way you set any option: + +```python +from ccflow.evaluators import MemoryCacheEvaluator + +with FlowOptionsOverride(options={"cacheable": True, "evaluator": MemoryCacheEvaluator()}): + _ = model(15) +``` + +That is the whole idea; the practical guides cover the two you will reach for most: + +- [Cache Results](Cache-Results) — avoid redundant work with the `MemoryCacheEvaluator`, and evaluate dependency graphs. +- [Retry on Failure](Retry-on-Failure) — make flaky steps resilient. + +The full list of evaluators is in the [Built-in Models](Built-in-Models) reference. + +## What you learned + +- A workflow step is a `CallableModel` you call with a context to get a result. +- `@Flow.call` handles type conversion and validation, and is where framework behavior is injected. +- `FlowOptions` (via the decorator, an override, `meta`, or `_options`) tunes that behavior. +- Evaluators decide *how* steps run, without touching the steps themselves. + +## Next steps + +- [Building an ETL Pipeline](Building-an-ETL-Pipeline) — chain callable models into an end-to-end pipeline. +- [Bind Logic to Configs](Bind-Logic-to-Configs) — patterns for attaching logic, including publishers. +- [Core Concepts](Core-Concepts) — how contexts, results, and evaluators fit the bigger picture. diff --git a/docs/wiki/First-Steps.md b/docs/wiki/tutorials/First-Steps.md similarity index 52% rename from docs/wiki/First-Steps.md rename to docs/wiki/tutorials/First-Steps.md index 294bf98b..9ebd7506 100644 --- a/docs/wiki/First-Steps.md +++ b/docs/wiki/tutorials/First-Steps.md @@ -1,6 +1,10 @@ # First Steps -This short example shows some of the key features of the configuration framework in `ccflow`: +This is the shortest possible tour of `ccflow`. In a few minutes you will define two configuration objects, load them into a registry from plain data, look them up, and watch the registry keep linked objects in sync. Type it into a Python session and follow along — everything here runs as shown. + +You only need `ccflow` installed (see [Installation](Installation)). Everything below happens in Python; no files or command line yet. + +Paste the whole block into a Python session: ```python from ccflow import BaseModel, ModelRegistry @@ -68,3 +72,18 @@ root["data/source3"].file = "source3_amended.csv" print(root["transform"].x.file) #> source3_amended.csv ``` + +## What you just did + +- Defined two configuration schemas as `BaseModel` subclasses, so their fields are typed and validated. +- Loaded a nested dictionary of configs into the root `ModelRegistry` with `load_config`, which turned nested dictionaries into a hierarchy and resolved the string `"data/source1"` into a real reference. +- Looked configs up by path (`root["transform"]`). +- Rewired a dependency by name (`root["transform"].x = "data/source3"`) and saw that editing a low-level object (`root["data/source3"].file`) propagated up through the shared instance. + +That last point is the heart of `ccflow`: configuration is a graph of shared, strongly typed objects, not a tree of copied values. + +## Next steps + +- [Configuring Models](Configuring-Models) — build these ideas up properly: hierarchical configs, validation, serialization, and dependency injection. +- [Defining Workflows](Defining-Workflows) — make configuration objects *runnable*. +- [Core Concepts](Core-Concepts) — the vocabulary and the reasoning behind the design. diff --git a/docs/wiki/tutorials/Tutorials.md b/docs/wiki/tutorials/Tutorials.md new file mode 100644 index 00000000..0e00fc4d --- /dev/null +++ b/docs/wiki/tutorials/Tutorials.md @@ -0,0 +1,12 @@ +# Tutorials + +These tutorials are hands-on lessons. Work through them at the keyboard, in order — each one builds on the last, and every step is meant to succeed exactly as written. They teach by *doing*; the reasoning behind what you are doing lives in the [Explanation](Explanation) pages, and task-focused recipes live in the [How-to Guides](How-to-Guides). + +1. **[First Steps](First-Steps)** — define a couple of configuration objects, register them, and see how the registry links them together. The shortest path to the core idea. +1. **[Configuring Models](Configuring-Models)** — build up strongly typed, hierarchical configuration with pydantic models, register them, and wire dependencies between them. +1. **[Defining Workflows](Defining-Workflows)** — turn configuration into runnable steps with contexts, results, and callable models, and meet the evaluators that run them. +1. **[Building an ETL Pipeline](Building-an-ETL-Pipeline)** — assemble extract, transform, and load steps into an end-to-end pipeline driven by a single Hydra config. +1. **[Composing an ETL Application](Composing-an-ETL-Application)** — grow that pipeline into a reusable, config-group-driven application with command-line dispatch. +1. **[Building a Configurable Calculator](Building-a-Configurable-Calculator)** — the capstone: combine the functional `@Flow.model` API with config groups and CLI dispatch to build a fully configurable program from the command line. + +New to `ccflow`? Start at [First Steps](First-Steps). If you want to understand *why* the framework is shaped this way before diving in, read [Core Concepts](Core-Concepts) first.