Raise record contract to 2.0

.github/workflows/ci.yml

@@ -11,7 +11,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
+        python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
 
     steps:
       - uses: actions/checkout@v7

CHANGELOG.md

@@ -4,6 +4,32 @@ All notable changes to this project are documented in this file.
 
 ## Unreleased
 
+- **Breaking: the record shape and the Python floor changed.** See
+  [MIGRATING.md](MIGRATING.md).
+  - logquill now requires **Python 3.10 or newer** and is tested on 3.10–3.14.
+    Python 3.8 and 3.9 are end-of-life; pip on those interpreters keeps
+    installing 1.x.
+  - Every record now carries `schema_version` (`"2.0"`). Nothing was renamed or
+    removed, so a reader only breaks if it insists on exactly the five 1.x
+    keys. `parse_record()` reads 1.x and 2.x records alike, labelling a 1.x
+    record `"1.0"`.
+  - New reserved fields, shared with `logquill` on npm: a top-level `llm` block
+    (`model`, `tokens_in`, `tokens_out`, `cost_usd`, `latency_ms`,
+    `finish_reason`) for LLM calls, plus `meta.retry_count`, `meta.state_diff`
+    and `meta.mcp.server`/`meta.mcp.tool`. The text and logfmt formatters show
+    the `llm` block. Nothing writes these on its own yet.
+  - `TamperEvidentPlugin` now covers `schema_version` and `llm` in the hash, so
+    editing either is caught; hash chains written by 1.x still verify.
+  - Fixed: the New Relic transport rebuilt each record from its five 1.x fields
+    and would have dropped `schema_version` and `llm`; it now copies the record.
+  - The record format now has a machine-readable definition,
+    `schema/record.schema.json` (JSON Schema 2020-12), and a shared file of
+    golden records, `schema/golden_records.json`. `tests/test_contract.py`
+    checks the schema, the golden records, the parser, and everything the
+    logger actually writes against each other, so drift between the Python and
+    JavaScript packages now fails a test instead of relying on a reviewer to
+    notice.
+
 - Added the 1.0 features that hadn't shipped yet, plus hardening:
   - `logger.opt(lazy=True)` defers callable `meta` values until a record is
     really going to be emitted, so an expensive `DEBUG`/`TRACE` argument costs

CONTRIBUTING.md

@@ -38,6 +38,8 @@ legitimately needs more memory, raise the budget in the same PR and say why.
   4. `CHANGELOG.md` has an entry under `Unreleased`
   5. Nothing in the cross-language contract table silently diverged from `logquill-js`
      (open a tracking issue there if it changed)
+     — if you change the record shape, change `schema/record.schema.json` and
+     `schema/golden_records.json` together (see `schema/README.md`)
 - **CI must be green** (`ruff check`, `mypy logquill`, `pytest`, `pytest benchmarks`) and **at
   least one review approval** is required before merge — enforced by branch
   protection on `main`.

MIGRATING.md

@@ -0,0 +1,82 @@
+# Migrating from logquill 1.x to 2.0
+
+2.0 raises the minimum Python version and extends the record shape shared with
+`logquill` on npm. Application code that only calls `logger.info(...)` and
+friends needs no changes; what changes is what's on disk and on the wire, and
+which Python you run.
+
+## 1. Python 3.10 or newer
+
+`logquill` 2.0 requires Python 3.10+ (it was 3.8+). It is tested on 3.10
+through 3.14. On an older interpreter, pip keeps installing the 1.x line.
+
+## 2. Every record carries `schema_version`
+
+Records now start with `"schema_version": "2.0"`:
+
+```json
+{"schema_version":"2.0","timestamp":"2026-01-01T00:00:00.000Z","level":"INFO","logger":"app","message":"hello","meta":{}}
+```
+
+**If you read logs** — a parser, a dashboard, a SIEM rule — make sure it
+tolerates one extra top-level key. Nothing was renamed or removed; a strict
+"exactly these five keys" check is the only thing that breaks.
+
+**If you have existing 1.x logs**, they don't need converting. A record with no
+`schema_version` is a 1.x record, and `parse_record` reads both:
+
+```python
+import json
+
+from logquill import parse_record
+
+line = '{"timestamp":"2026-01-01T00:00:00.000Z","level":"INFO","logger":"app","message":"hi"}'
+
+record = parse_record(json.loads(line))  # a 1.x or 2.x line
+assert record["schema_version"] == "1.0"  # "2.0" for a 2.x record
+assert record["meta"] == {}
+```
+
+It fills in `"1.0"` and an empty `meta` if either is missing, keeps every other
+key, and raises `ValueError` — saying what to fix — for a record that isn't a
+LogQuill record, or one written by a newer major version.
+
+**Hash-chained logs** (`TamperEvidentPlugin`) keep verifying: a chain written by
+1.x verifies unchanged, and 2.x chains additionally cover `schema_version` and
+`llm`, so editing either is caught.
+
+## 3. New optional fields
+
+These are defined now so both languages agree on them. Nothing emits them on its
+own yet, and no record has them unless you add them.
+
+| Field | Where | Type |
+|---|---|---|
+| `llm.model`, `llm.finish_reason` | top level, `llm` block | string |
+| `llm.tokens_in`, `llm.tokens_out` | `llm` block | integer ≥ 0 |
+| `llm.cost_usd`, `llm.latency_ms` | `llm` block | number ≥ 0 |
+| `meta.retry_count` | `meta` | integer ≥ 0 |
+| `meta.state_diff` | `meta` | object |
+| `meta.mcp.server`, `meta.mcp.tool` | `meta` | string |
+
+`llm` is its own block, not part of `meta`, because cost and latency
+dashboards need stable names for it. A record that isn't an LLM call has no
+`llm` key.
+
+## 4. The published schema
+
+`schema/record.schema.json` (JSON Schema 2020-12) defines a record precisely.
+The top level is closed — `schema_version`, `timestamp`, `level`, `logger`,
+`message`, `meta` and `llm` only — so put your own fields in `meta`, which stays
+free-form. `schema/golden_records.json` holds example records that this package
+and `logquill` on npm both test against.
+
+## 5. Small things
+
+- `LogRecord` gained `schema_version` (required) and `llm` (optional), and
+  `create_record()` sets them. If you build `LogRecord` values by hand in your
+  own transport or plugin, add `schema_version=SCHEMA_VERSION`, or build them
+  from `create_record()`.
+- If a transport of yours rebuilds a record from its five 1.x fields, copy the
+  record instead (`{**record, "meta": new_meta}`) so `schema_version` and `llm`
+  aren't dropped.

README.md

@@ -3,7 +3,7 @@
 [![CI](https://github.com/nikhilvdev/logquill-python/actions/workflows/ci.yml/badge.svg)](https://github.com/nikhilvdev/logquill-python/actions/workflows/ci.yml)
 [![Publish](https://github.com/nikhilvdev/logquill-python/actions/workflows/release.yml/badge.svg)](https://github.com/nikhilvdev/logquill-python/actions/workflows/release.yml)
 [![PyPI](https://img.shields.io/pypi/v/logquill.svg)](https://pypi.org/project/logquill/)
-[![Python versions](https://img.shields.io/badge/python-3.8%2B-blue.svg)](pyproject.toml)
+[![Python versions](https://img.shields.io/badge/python-3.10%2B-blue.svg)](pyproject.toml)
 [![License](https://img.shields.io/github/license/nikhilvdev/logquill-python)](LICENSE)
 [![GitHub tag](https://img.shields.io/github/v/tag/nikhilvdev/logquill-python)](https://github.com/nikhilvdev/logquill-python/tags)
 [![Downloads](https://static.pepy.tech/badge/logquill)](https://pepy.tech/project/logquill)
@@ -37,6 +37,8 @@ for what's landed so far.
 
 ## Install
 
+Requires Python 3.10 or newer (2.0 raised the floor from 3.8; see [MIGRATING.md](MIGRATING.md)).
+
 ```bash
 pip install logquill
 ```
@@ -50,24 +52,56 @@ logger = Logger("app", level=Level.INFO)
 
 record = logger.info("user signed up", user_id=42, plan="pro")
 print(record)
-# {'timestamp': '2026-08-27T18:04:12.345Z', 'level': 'INFO', 'logger': 'app',
-#  'message': 'user signed up', 'meta': {'user_id': 42, 'plan': 'pro'}}
+# {'schema_version': '2.0', 'timestamp': '2026-08-27T18:04:12.345Z', 'level': 'INFO',
+#  'logger': 'app', 'message': 'user signed up', 'meta': {'user_id': 42, 'plan': 'pro'}}
 
 logger.debug("below threshold, dropped")  # -> None, filtered by level
 logger.set_level("debug")
 logger.debug("now visible")  # -> a record dict
 ```
 
 Every log call returns the record dict (or `None` if filtered by level) —
-`{"timestamp": ISO8601, "level": str, "logger": str, "message": str, "meta": dict}`,
+`{"schema_version": "2.0", "timestamp": ISO8601, "level": str, "logger": str, "message": str, "meta": dict}`,
 the same shape shared with [`logquill` on npm](https://www.npmjs.com/package/logquill).
 Use `JSONFormatter` to serialize a record to the canonical JSON line:
 
 ```python
 from logquill import JSONFormatter
 
 print(JSONFormatter().format(record))
-# '{"timestamp":"2026-08-27T18:04:12.345Z","level":"INFO","logger":"app","message":"user signed up","meta":{"user_id":42,"plan":"pro"}}'
+# '{"schema_version":"2.0","timestamp":"2026-08-27T18:04:12.345Z","level":"INFO","logger":"app","message":"user signed up","meta":{"user_id":42,"plan":"pro"}}'
+```
+
+### The record schema
+
+The record shape is defined precisely by [`schema/record.schema.json`](schema/record.schema.json)
+(JSON Schema 2020-12), the same file `logquill` on npm is tested against, with
+a shared set of [golden records](schema/golden_records.json) that both
+packages' test suites run. Two things to know:
+
+- `schema_version` is on every record. `parse_record()` reads a record another
+  process wrote, and accepts both 2.x records and logquill 1.x ones (which have
+  no `schema_version` and come back labelled `"1.0"`). It raises `ValueError`,
+  saying what to fix, for anything that isn't a LogQuill record.
+- An LLM call's cost and latency have their own top-level `llm` block
+  (`model`, `tokens_in`, `tokens_out`, `cost_usd`, `latency_ms`,
+  `finish_reason`), not free-form `meta`; `meta.retry_count`,
+  `meta.state_diff` and `meta.mcp.server`/`meta.mcp.tool` are reserved with
+  fixed types.
+
+```python
+import json
+
+from logquill import JSONFormatter, Logger, parse_record
+
+logger = Logger("app")
+line = JSONFormatter().format(logger.info("user signed up", user_id=42))
+
+assert parse_record(json.loads(line))["schema_version"] == "2.0"
+
+# a line written by logquill 1.x
+old = '{"timestamp":"2026-01-01T00:00:00.000Z","level":"INFO","logger":"app","message":"hi","meta":{}}'
+assert parse_record(json.loads(old))["schema_version"] == "1.0"
 ```
 
 ## Config

logquill/__init__.py

@@ -22,7 +22,7 @@
 from logquill.plugins.slack_alert_plugin import SlackAlertPlugin
 from logquill.plugins.tamper_evident_plugin import TamperEvidentPlugin
 from logquill.plugins.trace_context_plugin import TraceContextPlugin
-from logquill.records import LogRecord
+from logquill.records import SCHEMA_VERSION, LLMBlock, LogRecord, parse_record
 from logquill.serverless import with_azure_function, with_cloud_function, with_lambda
 from logquill.toggle import disable, enable, is_enabled
 from logquill.transports.batching_transport import BatchingTransport
@@ -78,6 +78,7 @@
     "KafkaTransport",
     "Level",
     "LogfmtFormatter",
+    "LLMBlock",
     "LogQuillAdapter",
     "LogQuillHandler",
     "LogRecord",
@@ -96,6 +97,7 @@
     "RedactPlugin",
     "RedisTransport",
     "RunPlugin",
+    "SCHEMA_VERSION",
     "SQLLogRow",
     "SQLiteTransport",
     "SQSTransport",
@@ -120,6 +122,7 @@
     "parse",
     "parse_level",
     "parse_logfmt",
+    "parse_record",
     "with_azure_function",
     "with_cloud_function",
     "with_lambda",

logquill/formatters/logfmt_formatter.py

@@ -81,6 +81,7 @@ class LogfmtFormatter:
 
         timestamp=2026-01-01T00:00:00.000Z level=INFO logger=app message="user signed up" user_id=42
 
+    An LLM call's `llm` block follows as `llm.model=...`, `llm.tokens_in=...`.
     `meta` keys are emitted as top-level pairs after the four record fields;
     nested dicts flatten to dotted keys (`http.status=200`); lists are
     emitted as a JSON string. Values containing whitespace, `=`, quotes, or
@@ -101,6 +102,9 @@ def format(self, record: LogRecord) -> str:
             f"logger={_quote(str(record['logger']))}",
             f"message={_quote(str(record['message']))}",
         ]
+        llm = record.get("llm")
+        if llm:
+            _flatten("llm", llm, 1, pairs)
         for meta_key, value in record["meta"].items():
             key = _key(meta_key)
             if key in _RESERVED_KEYS:

logquill/formatters/text_formatter.py

@@ -29,6 +29,9 @@ def format_text(record: Mapping[str, Any]) -> str:
         f"{record.get('timestamp', '?')} {str(record.get('level', '?')):<5} "
         f"{record.get('logger', '?')}: {record.get('message', '')}"
     )
+    llm = record.get("llm")
+    if llm:
+        line += f" llm={_dump_meta(llm)}"
     if meta:
         line += f" {_dump_meta(meta)}"
     if stack is not None:

logquill/parsing.py

@@ -9,16 +9,16 @@
 
 #: Matches one entry written by `TextFormatter` (single-line entries; a
 #: traceback printed on the lines after an entry isn't part of the match).
-#: Pass with `cast=TEXT_LOG_CASTS` to get `meta` back as a dict:
+#: Pass with `cast=TEXT_LOG_CASTS` to get `meta` (and an LLM call's `llm`) back as dicts:
 #:
 #:     parse("app.log", TEXT_LOG_PATTERN, cast=TEXT_LOG_CASTS)
 TEXT_LOG_PATTERN = (
     r"^(?P<timestamp>\S+) (?P<level>[A-Z]+)\s+(?P<logger>[^:\s]+): "
-    r"(?P<message>.*?)(?: (?P<meta>\{.*\}))?$"
+    r"(?P<message>.*?)(?: llm=(?P<llm>\{[^{}]*\}))?(?: (?P<meta>\{.*\}))?$"
 )
 
-#: `cast` mapping that decodes the `meta` group of `TEXT_LOG_PATTERN` from JSON.
-TEXT_LOG_CASTS: dict[str, Callable[[str], Any]] = {"meta": json.loads}
+#: `cast` mapping that decodes the `meta` and `llm` groups of `TEXT_LOG_PATTERN` from JSON.
+TEXT_LOG_CASTS: dict[str, Callable[[str], Any]] = {"meta": json.loads, "llm": json.loads}
 
 
 def parse(

logquill/plugins/tamper_evident_plugin.py

@@ -6,7 +6,7 @@
 from typing import Any
 
 from logquill.plugins.plugin import Plugin
-from logquill.records import LogRecord
+from logquill.records import LEGACY_SCHEMA_VERSION, LogRecord
 
 GENESIS_HASH = "0" * 64
 
@@ -68,15 +68,20 @@ def verify_chain(
 
 def _compute_hash(record: Mapping[str, Any], prev_hash: str) -> str:
     meta = record.get("meta", {})
-    payload = json.dumps(
-        {
-            "timestamp": record.get("timestamp"),
-            "level": record.get("level"),
-            "logger": record.get("logger"),
-            "message": record.get("message"),
-            "meta": {k: v for k, v in meta.items() if k not in ("hash", "prev_hash")},
-        },
-        sort_keys=True,
-        default=str,
-    )
+    content: dict[str, Any] = {
+        "timestamp": record.get("timestamp"),
+        "level": record.get("level"),
+        "logger": record.get("logger"),
+        "message": record.get("message"),
+        "meta": {k: v for k, v in meta.items() if k not in ("hash", "prev_hash")},
+    }
+    # `schema_version` and `llm` are covered when present, so editing either is
+    # caught. A `schema_version` of "1.0" is what `parse_record` labels a record
+    # that had none, so it's left out: a chain written by logquill 1.x verifies
+    # the same before and after parsing.
+    if record.get("schema_version", LEGACY_SCHEMA_VERSION) != LEGACY_SCHEMA_VERSION:
+        content["schema_version"] = record["schema_version"]
+    if "llm" in record:
+        content["llm"] = record["llm"]
+    payload = json.dumps(content, sort_keys=True, default=str)
     return hashlib.sha256(f"{prev_hash}{payload}".encode()).hexdigest()