fix: validation

docs/docs/advanced/lora-and-alora-adapters.md

@@ -194,6 +194,30 @@ directly (bypassing the normal `validate()` call), use `ALoraRequirement` from
 but still requires a matching adapter to actually be registered. If none is found,
 Mellea logs a warning and falls back to regular generation rather than erroring.
 
+### What the adapter sees
+
+The `requirement-check` adapter judges the last assistant turn of the conversation it is
+given, so the routing above only produces useful verdicts if that conversation actually
+reaches it. Mellea builds the adapter's message list from the validation context's
+`view_for_generation()`, and validation runs over the post-generation context — the same
+conversation the model generated into, with the generated output last.
+
+That only works on a context that renders history, which the default session does not
+provide: `start_session()` returns a session backed by `SimpleContext`, which retains
+`last_output()` but whose `view_for_generation()` is always empty by design. An
+adapter-backed requirement has no conversation to judge there, so an explicit
+`ALoraRequirement` raises `ValueError` and a plain `Requirement` logs a warning and falls
+back to LLM-as-a-judge. Ask for a chat context explicitly:
+
+```python
+import mellea
+
+# Adapter-backed requirements need a context that renders the assistant turn.
+m = mellea.start_session(context_type="chat")
+```
+
+See [What the validator sees](../concepts/requirements-system.md#what-the-validator-sees).
+
 ## Disable adapter validation
 
 To run without adapter validation (for benchmarking or debugging):

docs/docs/concepts/requirements-system.md

@@ -297,6 +297,48 @@ reserve LLM-based requirements for subjective criteria that cannot be coded dire
 
 For a full walkthrough of using LLM-as-a-judge for output quality evaluation, see [Evaluate with LLM-as-a-Judge](../how-to/evaluate-with-llm-as-a-judge.md).
 
+## What the validator sees
+
+Validation runs over the **post-generation context** — the same conversation the model
+just generated into, with the generated output as its last entry. This matters for all
+three validation approaches:
+
+- A `validation_fn` receives that context, so `ctx.last_output()` is the output under
+  judgement and `ctx.as_list()` is the conversation that produced it.
+- An LLM-as-a-judge requirement sends that conversation to the model, followed by the
+  specific output to judge. The judge prompt scopes the verdict to that output, while
+  telling the model the conversation is there to help it interpret the requirement —
+  which is what makes conversation-dependent requirements ("must answer the question
+  that was asked") judgeable at all.
+- An `ALoraRequirement` needs the conversation to reach the adapter at all — the
+  `requirement-check` adapter judges the last assistant turn of the conversation it is
+  given. See [LoRA and aLoRA Adapters](../advanced/lora-and-alora-adapters.md).
+
+The output under judgement is passed as the `ModelOutputThunk` itself, not a detached
+string copy, so its `parsed_repr` and any structured representation survive into the
+judge prompt.
+
+### Overriding the validation context
+
+`SamplingStrategy.sample()` takes an optional `validation_ctx` for validating over
+something other than the post-generation context — for example, a trimmed context that
+excludes long retrieved documents. When it is given, the sampled output is appended to it
+so it is still the validation target; when it is `None`, each attempt is validated over its
+own post-generation context.
+
+`instruct()`, `act()`, and their async counterparts always pass `None`, so the
+post-generation context is what you get unless you drive a strategy's `sample()` directly.
+
+`validate()` and `avalidate()` follow the same rule at a lower level: they validate over
+the `context` you hand them, in your own context type. Their `output` argument designates
+*which* output is under judgement rather than replacing the context — it is appended unless
+it is already the context's last entry. Passing an `output` from an earlier turn works too:
+it is appended so that it becomes the target, and the judge still sees the conversation
+around it.
+
+Preconditions are the one case that never sees the conversation: `precondition_requirements`
+are judged over the function arguments alone, in a fresh context.
+
 ## Composing requirements
 
 Requirements are composable: mix strings, `req()`, `check()`, and `Requirement`

docs/docs/how-to/write-custom-verifiers.md

@@ -77,6 +77,34 @@ result = m.instruct(
 print(str(result))
 ```
 
+### Which context your function receives
+
+The `ctx` argument is the **post-generation context**: the conversation the model just
+generated into, with the output under judgement as its last entry. So `ctx.last_output()`
+is the output to check, and `ctx.as_list()` gives you the turns that produced it — useful
+when a verdict depends on what was asked, not just what was answered:
+
+```python
+from mellea.core import Context, ValidationResult
+
+
+def validate_answers_the_question(ctx: Context) -> ValidationResult:
+    """Fail if the output repeats the user's question back instead of answering it."""
+    turns = ctx.as_list()
+    output = ctx.last_output()
+    text = output.value if output and output.value else ""
+
+    prior = [str(turn) for turn in turns[:-1]]
+    if prior and text.strip() in prior[-1]:
+        return ValidationResult(False, reason="The response echoes the question.")
+    return ValidationResult(True)
+```
+
+Under `instruct()` and `act()` this is always the post-generation context. Driving a
+sampling strategy's `sample()` directly lets you pass a `validation_ctx` to validate over
+something else instead — see
+[Overriding the validation context](../concepts/requirements-system.md#overriding-the-validation-context).
+
 ## Common validation patterns
 
 ### JSON validity

mellea/backends/huggingface.py

@@ -635,6 +635,10 @@ async def _generate_from_context(
             tool_calls (bool): If `True`, expose available tools to the model and
                 parse tool-call responses.
 
+        Raises:
+            ValueError: If `action` is an `ALoraRequirement` but `ctx` renders no
+                conversation, leaving the requirement-check adapter nothing to judge.
+
         Returns:
             tuple[ModelOutputThunk[C], Context]: A thunk holding the (lazy) model output
                 and an updated context that includes `action` and the new output.
@@ -690,6 +694,43 @@ async def _generate_from_context(
                 if issubclass(type(action), LLMaJRequirement):
                     reroute_to_alora = False
 
+                # The requirement-check adapter judges the last assistant turn of the
+                # conversation it is given, and this path never renders the requirement
+                # template -- so unlike LLM-as-a-judge it has no inlined copy of the output
+                # to fall back on. A context that renders nothing leaves it nothing to judge.
+                adapter_view = ctx.view_for_generation()
+                if reroute_to_alora and not adapter_view:
+                    # `None` means the history is non-linear and cannot be rendered at all;
+                    # `[]` means it renders, but to nothing. Neither gives the adapter an
+                    # assistant turn, so both are fatal here -- only the wording differs.
+                    empty_view_reason = (
+                        "its history is non-linear, so no conversation can be rendered"
+                        if adapter_view is None
+                        else "it renders no conversation"
+                    )
+                    if isinstance(action, ALoraRequirement):
+                        raise ValueError(
+                            f"cannot validate an ALoraRequirement over a "
+                            f"{type(ctx).__name__}: {empty_view_reason}, so the "
+                            f"{adapter_name} adapter has no assistant turn to judge. "
+                            "Validate over a context that renders history (e.g. ChatContext), "
+                            "or use a plain Requirement, which falls back to LLM-as-a-judge."
+                        )
+                    # Auto-rerouting a plain Requirement is an optimisation, not a request:
+                    # fall back to LLM-as-a-judge, which inlines the output and still works.
+                    warn_key = (
+                        f"alora_reroute_empty_view_{type(ctx).__name__}"
+                        f"_{adapter_view is None}"
+                    )
+                    if warn_key not in self._warned_about:
+                        self._warned_about.add(warn_key)
+                        MelleaLogger.get_logger().warning(
+                            f"not rerouting requirements to the {adapter_name} adapter: "
+                            f"{type(ctx).__name__} gives it nothing to judge -- "
+                            f"{empty_view_reason}; using LLM-as-a-judge instead."
+                        )
+                    reroute_to_alora = False
+
                 if reroute_to_alora:
                     # Keep the alora requirement handling separate for now.
                     mot = await self._generate_from_intrinsic(

mellea/backends/openai.py

@@ -833,6 +833,10 @@ async def _generate_from_context(
             tool_calls (bool): If `True`, expose available tools to the model and
                 parse tool-call responses.
 
+        Raises:
+            ValueError: If `action` is an `ALoraRequirement` but `ctx` renders no
+                conversation, leaving the requirement-check adapter nothing to judge.
+
         Returns:
             tuple[ModelOutputThunk[C], Context]: A thunk holding the (lazy) model output
                 and an updated context that includes `action` and the new output.
@@ -888,6 +892,43 @@ async def _generate_from_context(
                 if issubclass(type(action), LLMaJRequirement):
                     reroute_to_alora = False
 
+                # The requirement-check adapter judges the last assistant turn of the
+                # conversation it is given, and this path never renders the requirement
+                # template -- so unlike LLM-as-a-judge it has no inlined copy of the output
+                # to fall back on. A context that renders nothing leaves it nothing to judge.
+                adapter_view = ctx.view_for_generation()
+                if reroute_to_alora and not adapter_view:
+                    # `None` means the history is non-linear and cannot be rendered at all;
+                    # `[]` means it renders, but to nothing. Neither gives the adapter an
+                    # assistant turn, so both are fatal here -- only the wording differs.
+                    empty_view_reason = (
+                        "its history is non-linear, so no conversation can be rendered"
+                        if adapter_view is None
+                        else "it renders no conversation"
+                    )
+                    if isinstance(action, ALoraRequirement):
+                        raise ValueError(
+                            f"cannot validate an ALoraRequirement over a "
+                            f"{type(ctx).__name__}: {empty_view_reason}, so the "
+                            f"{adapter_name} adapter has no assistant turn to judge. "
+                            "Validate over a context that renders history (e.g. ChatContext), "
+                            "or use a plain Requirement, which falls back to LLM-as-a-judge."
+                        )
+                    # Auto-rerouting a plain Requirement is an optimisation, not a request:
+                    # fall back to LLM-as-a-judge, which inlines the output and still works.
+                    warn_key = (
+                        f"alora_reroute_empty_view_{type(ctx).__name__}"
+                        f"_{adapter_view is None}"
+                    )
+                    if warn_key not in self._warned_about:
+                        self._warned_about.add(warn_key)
+                        MelleaLogger.get_logger().warning(
+                            f"not rerouting requirements to the {adapter_name} adapter: "
+                            f"{type(ctx).__name__} gives it nothing to judge -- "
+                            f"{empty_view_reason}; using LLM-as-a-judge instead."
+                        )
+                    reroute_to_alora = False
+
                 if reroute_to_alora:
                     mot = await self._generate_from_intrinsic(
                         alora_action,

mellea/core/base.py

@@ -2180,6 +2180,11 @@ def as_list(self, last_n_components: int | None = None) -> list[Span]:
 
         If `last_n_components` is `None`, then all components are returned.
 
+        The same `Span` may legitimately appear more than once in the returned list: adding
+        an earlier output back onto a context to designate it as a validation target is the
+        motivating case. Only a repeated context *node* is a cycle, and that is what the
+        guard on this walk rejects.
+
         Args:
             last_n_components (int | None): Maximum number of most-recent components to include.
                 Pass `None` to return the full history.
@@ -2189,16 +2194,19 @@ def as_list(self, last_n_components: int | None = None) -> list[Span]:
         """
         context_list: list[Span] = []
         current_context: Context = self
+        visited_nodes: set[int] = set()
 
         last_n_count = 0
         while not current_context.is_root_node and (
             last_n_components is None or last_n_count < last_n_components
         ):
-            data = current_context.node_data
-            assert data is not None, "Data cannot be None (except for root context)."
-            assert data not in context_list, (
+            assert id(current_context) not in visited_nodes, (
                 "There might be a cycle in the context tree. That is not allowed."
             )
+            visited_nodes.add(id(current_context))
+
+            data = current_context.node_data
+            assert data is not None, "Data cannot be None (except for root context)."
             context_list.append(data)
             last_n_count += 1
 
@@ -2386,7 +2394,14 @@ class TemplateRepresentation:
     obj: Any
     args: dict[
         str,
-        str | Component | CBlock | Iterable | Mapping | TemplateRepresentation | None,
+        str
+        | Component
+        | CBlock
+        | ModelOutputThunk
+        | Iterable
+        | Mapping
+        | TemplateRepresentation
+        | None,
     ]
     tools: dict[str, AbstractMelleaTool] | None = (
         None  # the key must be the name of the function.

mellea/core/requirement.py

@@ -329,12 +329,29 @@ def __init__(
         self.check_only = check_only
         self.chunking: ChunkingStrategy | None = resolve_chunking_strategy(chunking)
 
-        # Used for validation. Do not manually populate.
-        self._output: str | None = None
+        # The span under judgement. Bound only on the transient copy created inside
+        # `validate` (see `_bind_validation_target`). Do not manually populate.
+        self._validation_target: Span | None = None
 
         # Per-stream chunker for streaming validation, built lazily.
         self._chunker: Chunker | None = None
 
+    def _bind_validation_target(self, target: Span) -> "Requirement":
+        """Return a shallow copy of this requirement bound to the span under judgement.
+
+        Binding happens on a copy so that a `Requirement` object can be reused across
+        validation calls (and across sampling iterations) without accumulating state.
+
+        Args:
+            target: The `Component`, `CBlock`, or `ModelOutputThunk` being validated.
+
+        Returns:
+            Requirement: A copy of this requirement whose `_validation_target` is `target`.
+        """
+        bound = copy(self)
+        bound._validation_target = target
+        return bound
+
     def __copy__(self) -> "Requirement":
         """Return a shallow copy with the live `_chunker` reset to `None`.
 
@@ -391,10 +408,10 @@ async def validate(
                 " Context has no appropriate last output"
             )
 
-            # Create a copy of the requirement that holds the output
-            # and its template gets populated with the output correctly.
-            req_copy = copy(self)
-            req_copy._output = last_output.value
+            # Bind the output being judged to a copy of this requirement, so that the
+            # judgement request carries the output itself -- not a detached string copy
+            # of it -- and so that `self` is left unmodified.
+            req_copy = self._bind_validation_target(last_output)
             llm_as_a_judge_result, val_ctx = await backend.generate_from_context(
                 req_copy, ctx, format=format, model_options=model_options
             )
@@ -562,29 +579,43 @@ async def stream_flush(
     def parts(self) -> list[Span]:
         """Returns all of the constituent parts of a Requirement.
 
+        Once a validation target has been bound (inside a `validate` call), that target is
+        the requirement's sole part. Exposing it here is what allows `generate_walk` to
+        await it if it has not been computed yet.
+
         Returns:
-            List of constituent components. Empty by default; subclasses override
-            to expose their internal structure.
+            List of constituent components. Empty unless a validation target is bound.
         """
-        return []
+        return [] if self._validation_target is None else [self._validation_target]
 
     def format_for_llm(self) -> TemplateRepresentation | str:
         """Returns a `TemplateRepresentation` for LLM-as-a-Judge evaluation of this requirement.
 
-        Populates the template with the requirement's `description` and the stored model
-        `_output`. Must only be called from within a `validate` call for this same requirement,
-        after `_output` has been set.
+        Populates the template with the requirement's `description` and the bound validation
+        target. Must only be called from within a `validate` call for this same requirement,
+        after the target has been bound by `_bind_validation_target`.
+
+        When the target is a `ModelOutputThunk` with a `Component` `parsed_repr`, the parsed
+        representation is used so that the judge sees the structured output rather than the
+        raw generated string.
 
         Returns:
             TemplateRepresentation | str: A `TemplateRepresentation` containing the description
             and the model output to be judged.
         """
-        assert self._output is not None, (
+        assert self._validation_target is not None, (
             "Object protocol error: should never try to templatize a Requirement except inside of a validate call for that same requirement."
         )
+
+        target: Span = self._validation_target
+        if isinstance(target, ModelOutputThunk) and isinstance(
+            target.parsed_repr, Component
+        ):
+            target = target.parsed_repr
+
         return TemplateRepresentation(
             obj=self,
-            args={"description": self.description, "output": self._output},
+            args={"description": self.description, "output": target},
             tools=None,
             template_order=["*", "Requirement"],
         )

mellea/core/sampling.py

@@ -148,7 +148,8 @@ async def sample(
             context: The context to be passed to the sampling strategy.
             backend: The backend used for generating samples.
             requirements: List of requirements to test against (merged with global requirements).
-            validation_ctx: Optional context to use for validation. If None, validation_ctx = ctx.
+            validation_ctx: Optional context to validate over. If None, each sample is validated
+                over its own post-generation context.
             format: output format for structured outputs.
             model_options: model options to pass to the backend during generation / validation.
             tool_calls: True if tool calls should be used during this sampling strategy.