Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,12 @@ the schema.
Schema and codegen changes must include `schema.lock` and all regenerated
files. Generated files and AUTOGEN regions must not be edited manually.

Code generation formats the generated Page modules with Ruff from the active
Python environment, so install the development dependencies before running it.
Ruff has a minimum version rather than an exact pin; use the same version when
comparing generated output across environments, and run the formatting and
codegen drift checks above before submitting changes.

## API reference

The generated Python API reference is published at
Expand Down
28 changes: 24 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ once, allowing the server to return its timeout response before the client deadl

`connect_timeout` defaults to `timeout` and bounds the entire WebSocket dial,
including connection setup and handshake. A value of `0` or less disables that
connection deadline. Connection timeouts raise the SDK's `browserless_bap.TimeoutError`
connection deadline. Connection timeouts raise the SDK's `browserless_bap.BAPTimeoutError`
with `operation == "connect"`.

A connect deadline expiring is terminal for that page's connection. To retry,
Expand Down Expand Up @@ -108,8 +108,8 @@ contents between the markers by hand.
| `content` | `html` | `str` | ✅ |
| `cookies` | `cookies` | `list[StandardCookie]` | ✅ |
| `emulate_media` | `emulateMediaType` | `EmulateMediaTypeResponse` | ✅ |
| `eval_on_selector` | `text` | `str` | ✅ |
| `eval_on_selector_all` | `mapSelector` | `list[MapSelectorResponse]` | ✅ |
| `eval_on_selector` | `text` | `str` | — |
| `eval_on_selector_all` | `mapSelector` | `list[MapSelectorResponse]` | — |
| `evaluate` | `evaluate` | `str \| None` | ✅ |
| `fulfill` | `fulfill` | `FulfillResponse` | — |
| `go_back` | `back` | `HTTPResponse \| None` | ✅ |
Expand Down Expand Up @@ -162,6 +162,26 @@ contents between the markers by hand.

<!-- AUTOGEN:method-table END -->

## Response keys and optional fields

Options and keyword arguments use `snake_case`; returned dictionaries keep the
server's field names (`liveURLId`, `httpOnly`, `childElementCount`), as Playwright
for Python does for cookies.

`query_selector()` and `query_selector_all()` fetch element HTML only with
`html=True`. Reading a handle's `inner_html` or `outer_html` without requesting it
raises `BAPValueError` with a hint. This also applies to element-scoped queries.
`response()` omits bodies by default; pass `body=True` to fetch them, or
`body_encoding=True` to fetch both the body and its `base64Encoded` flag.

`eval_on_selector()` returns the first matching element's text, and
`eval_on_selector_all()` returns element details. Unlike Playwright's methods,
neither accepts a page function.

SDK exceptions are `BAPTimeoutError`, `BAPConnectionError` and `BAPValueError`.
They still subclass Python's matching built-in exceptions; the builtin-named SDK
exports have been removed.

## Schema and generated code

`src/browserless_bap/mutations.graphql` is the vendored GraphQL schema. Its full digest,
Expand Down Expand Up @@ -237,7 +257,7 @@ card, import `LiveURLInstructions` from `browserless_bap.types`.

The SDK subscribes before minting and retains completion until the next mint,
so a quick completion is not lost between `live_url()` and `wait_for_live_complete()`.
Wait timeouts are in milliseconds and raise the SDK `TimeoutError`. By default
Wait timeouts are in milliseconds and raise `BAPTimeoutError`. By default
there is no client wait deadline; pass a positive timeout to bound the wait.
`timeout=0` also disables the wait deadline. Timing out or cancelling a wait does
not close the live URL.
Expand Down
21 changes: 19 additions & 2 deletions llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ All timeout values are milliseconds; the default per-step `timeout` is `30000`.
Mutations with server-side timeouts get `5000` ms of client headroom once.
`connect_timeout` defaults to `timeout`, bounds the entire WebSocket dial
(including setup and handshake), and is disabled by values of `0` or less.
Connection timeouts raise the SDK's `browserless_bap.TimeoutError` with
Connection timeouts raise the SDK's `browserless_bap.BAPTimeoutError` with
`operation == "connect"`.

A connect deadline expiring is terminal for that page's connection; create a
Expand All @@ -45,6 +45,23 @@ Generated Page methods: `add_cookies`, `add_script_tag`, `add_style_tag`, `authe

<!-- AUTOGEN:method-list END -->

## Response keys and optional fields

Options and keyword arguments use `snake_case`; returned dictionaries keep the
server's field names (`liveURLId`, `httpOnly`, `childElementCount`), as Playwright
for Python does for cookies.

`query_selector()` and `query_selector_all()` fetch element HTML only with
`html=True`, including element-scoped queries. Reading `inner_html` or
`outer_html` without requesting it raises `BAPValueError` with a fix hint.
`response()` omits bodies unless `body=True` or `body_encoding=True` (which implies
the body and adds `base64Encoded`).

`eval_on_selector()` returns text; `eval_on_selector_all()` returns element
details. Unlike Playwright's methods, neither accepts a page function.
SDK errors are `BAPTimeoutError`, `BAPConnectionError` and `BAPValueError`, still
subclassing the matching builtins; the builtin-named SDK exports are removed.

## Live URL handoffs

```python
Expand Down Expand Up @@ -84,7 +101,7 @@ card, import `LiveURLInstructions` from `browserless_bap.types`.

The SDK subscribes before minting and retains completion until the next mint,
so a quick completion is not lost between `live_url()` and `wait_for_live_complete()`.
Wait timeouts are in milliseconds and raise the SDK `TimeoutError`. By default
Wait timeouts are in milliseconds and raise `BAPTimeoutError`. By default
there is no client wait deadline; pass a positive timeout to bound the wait.
`timeout=0` also disables the wait deadline. Timing out or cancelling a wait does
not close the live URL.
Expand Down
11 changes: 9 additions & 2 deletions scripts/codegen/config/client_only.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,18 @@ class ClientParam:
name: str
annotation: str
transport_timeout: bool = False
opt_in: str | None = None


CLIENT_ONLY_PARAMS = {
"GotoOptions": (ClientParam("remote_response_metadata", "bool"),),
"ResponseFilterOptions": (ClientParam("body_encoding", "bool"),),
"GotoOptions": (
ClientParam("remote_response_metadata", "bool", opt_in="remoteMetadata"),
),
"ResponseFilterOptions": (
ClientParam("body", "bool", opt_in="body"),
ClientParam("body_encoding", "bool", opt_in="bodyEncoding"),
),
"QuerySelectorOptions": (ClientParam("html", "bool", opt_in="html"),),
"ScreenshotOptions": (ClientParam("path", "str"),),
"PDFOptions": (ClientParam("path", "str"),),
"AddScriptTagOptions": (ClientParam("path", "str"),),
Expand Down
44 changes: 44 additions & 0 deletions scripts/codegen/config/docs.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,52 @@
LLMS_END = "<!-- AUTOGEN:method-list END -->"

METHOD_DOCSTRINGS = {
"check": "Check the checkbox matching `selector`.",
"uncheck": "Uncheck the checkbox matching `selector`.",
"cookies": "Return the cookies on the page.",
"content": "Return the full HTML content of the page as a string.",
"watch_event": (
"Start tracking `event` now so a later `wait_for_event` call can return "
"immediately if that event has already fired."
),
"eval_on_selector": (
"Return the text content of the first element matching `selector`. "
"Unlike Playwright's `eval_on_selector`, this takes no page function: "
"it is a shortcut for `text(selector=...)`."
),
"eval_on_selector_all": (
"Return details (`innerHTML`, `innerText`, `id`, `class`) for every element "
"matching `selector`. Unlike Playwright's `eval_on_selector_all`, this "
"takes no page function: it is a shortcut for `map_selector(selector)`."
),
"query_selector": (
"Return a handle for the first element matching `selector`, or `None`. "
"The element's `innerHTML`/`outerHTML` are fetched only with `html=True`."
),
"query_selector_all": (
"Return handles for every element matching `selector`. "
"The elements' `innerHTML`/`outerHTML` are fetched only with `html=True`."
),
}


def lead_prose(description: str | None) -> str | None:
if not description:
return None
markers = (
"```",
"**Usage Example",
"**Field Definition",
"**Simple Example",
"**Advanced",
"Simple Example",
"Advanced Example",
"Example (",
"**Usage",
"**Example",
)
end = min(
(description.index(marker) for marker in markers if marker in description),
default=len(description),
)
return description[:end].strip() or None
2 changes: 0 additions & 2 deletions scripts/codegen/config/methods.py
Original file line number Diff line number Diff line change
Expand Up @@ -163,8 +163,6 @@ def options_name(mutation: str) -> str:
"pdf",
"query_selector",
"query_selector_all",
"eval_on_selector",
"eval_on_selector_all",
"wait_for_selector",
"wait_for_function",
"evaluate",
Expand Down
14 changes: 5 additions & 9 deletions scripts/codegen/config/schema_map.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
}

RESPONSE_TYPE_OVERRIDES = {"Headers": "RequestHeader"}
RESPONSE_FIELD_TYPE_OVERRIDES = {("QuerySelectorResponse", "childElementCount"): "int"}
RESPONSE_RENAME_EXCLUDE = {"WaitForFunction", "WaitForEvent"}

SCALAR_MAP = {
Expand All @@ -34,13 +35,8 @@
}

SUBSCRIPTION_OMIT_FIELDS = {"response": {"body"}}
UNSUPPORTED_FIELDS = {
"HTTPResponse": {
"fields": {"ip", "port"},
"variant": "HTTPResponseWithRemoteMetadata",
},
"ResponseResponse": {
"fields": {"base64Encoded"},
"variant": "ResponseResponseWithBodyEncoding",
},
OPTIONAL_FIELDS: dict[str, dict[str, tuple[str, ...]]] = {
"HTTPResponse": {"remoteMetadata": ("ip", "port")},
"ResponseResponse": {"body": ("body",), "bodyEncoding": ("body", "base64Encoded")},
"QuerySelectorResponse": {"html": ("innerHTML", "outerHTML")},
}
20 changes: 13 additions & 7 deletions scripts/codegen/emit_operations.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,25 @@
# ruff: noqa: E501
import json

from .config.schema_map import UNSUPPORTED_FIELDS
from .config.schema_map import OPTIONAL_FIELDS
from .schema import Schema, render_graphql_type
from .writer import HEADER


def response_fields(schema: Schema) -> dict[str, str]:
fields = {
return {
schema.response_name(name): schema.response_selection(name)
for name in schema.response_objects
}
for name, config in UNSUPPORTED_FIELDS.items():
fields[str(config["variant"])] = schema.response_selection(
name, keep_unsupported=True
)
return fields


def optional_fields(schema: Schema) -> dict[str, dict[str, str]]:
return {
schema.response_name(name): {
group: " ".join(fields) for group, fields in groups.items()
}
for name, groups in OPTIONAL_FIELDS.items()
}


def argument_types(schema: Schema) -> dict[str, dict[str, str]]:
Expand All @@ -40,6 +44,8 @@ def emit(schema: Schema) -> str:
"# fmt: off",
f"RESPONSE_FIELDS: dict[str, str] = {json.dumps(response_fields(schema), sort_keys=True, indent=4)}",
"",
f"OPTIONAL_FIELDS: dict[str, dict[str, str]] = {json.dumps(optional_fields(schema), sort_keys=True, indent=4)}",
"",
f"ARG_TYPES: dict[str, dict[str, str]] = {json.dumps(argument_types(schema), sort_keys=True, indent=4)}",
"# fmt: on",
"",
Expand Down
29 changes: 24 additions & 5 deletions scripts/codegen/emit_page.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@

import json

from .config.docs import METHOD_DOCSTRINGS
from .model import MethodModel
from .writer import HEADER

Expand Down Expand Up @@ -73,6 +72,7 @@ def _execute_lines(model: MethodModel, prefix: str) -> list[str]:
]
if timeout is not None:
lines.append(f" timeout=None if {timeout} is UNSET else {timeout},")
lines.extend(_opt_in_lines(model, " "))
lines.append(" )")
return lines

Expand All @@ -91,16 +91,33 @@ def _cast_execute_lines(model: MethodModel) -> list[str]:
lines.append(
f" timeout=None if {timeout} is UNSET else {timeout},"
)
lines.extend(_opt_in_lines(model, " "))
lines.extend((" ),", " )"))
return lines


def _opt_in_lines(model: MethodModel, indent: str) -> list[str]:
params = [param for param in model.params if param.opt_in is not None]
if not params:
return []
entries = ", ".join(
f"({json.dumps(param.opt_in)}, {param.py_name})" for param in params
)
return [
f"{indent}opt_in=tuple(",
f"{indent} group",
f"{indent} for group, enabled in ({entries},)",
f"{indent} if enabled is True",
f"{indent}),",
]


def _async_method(model: MethodModel) -> list[str]:
lines = [f" async def {model.py_name}("]
lines.extend(_signature_lines(model, " "))
lines.append(f" ) -> {model.return_type}:")
if docstring := METHOD_DOCSTRINGS.get(model.py_name):
lines.append(f" {json.dumps(docstring)}")
if docstring := model.doc:
lines.append(f" {json.dumps(docstring, ensure_ascii=False)}")
lines.extend(_argument_lines(model))
if model.flavor == "buffer":
has_path = any(param.py_name == "path" for param in model.params)
Expand Down Expand Up @@ -181,6 +198,7 @@ def emit_async(models: list[MethodModel]) -> str:
"# ruff: noqa: ASYNC109, F403, F405",
"from __future__ import annotations",
"",
"from collections.abc import Sequence",
"from typing import TYPE_CHECKING, Any, cast",
"",
"from ._codec import decode_base64, drop_unset, or_default, write_binary_file",
Expand All @@ -200,6 +218,7 @@ def emit_async(models: list[MethodModel]) -> str:
" response_type: str,",
" *,",
" timeout: float | None = None,",
" opt_in: Sequence[str] = (),",
" ) -> Any:",
" raise NotImplementedError",
"",
Expand Down Expand Up @@ -258,8 +277,8 @@ def emit_sync(models: list[MethodModel]) -> str:
lines.append(f" def {model.py_name}(")
lines.extend(_signature_lines(model, " "))
lines.append(f" ) -> {model.return_type}:")
if docstring := METHOD_DOCSTRINGS.get(model.py_name):
lines.append(f" {json.dumps(docstring)}")
if docstring := model.doc:
lines.append(f" {json.dumps(docstring, ensure_ascii=False)}")
lines.extend(_sync_result_lines(model))
lines.append("")
return "\n".join(lines)
15 changes: 13 additions & 2 deletions scripts/codegen/emit_responses.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import json
import keyword

from .config.schema_map import RESPONSE_FIELD_TYPE_OVERRIDES
from .schema import Schema
from .writer import HEADER

Expand Down Expand Up @@ -32,7 +33,12 @@ def emit(schema: Schema) -> str:
lines.append(f' "{name}",')
lines.append(" {")
for field in fields:
annotation = _nullable(schema.python_type(field.type, response=True))
annotation = _nullable(
RESPONSE_FIELD_TYPE_OVERRIDES.get(
(gql_name, field.name.value),
schema.python_type(field.type, response=True),
)
)
lines.append(
f" {json.dumps(field.name.value)}: "
f"{json.dumps(f'NotRequired[{annotation}]')},"
Expand All @@ -43,7 +49,12 @@ def emit(schema: Schema) -> str:
if not fields:
lines.append(" pass")
for field in fields:
annotation = _nullable(schema.python_type(field.type, response=True))
annotation = _nullable(
RESPONSE_FIELD_TYPE_OVERRIDES.get(
(gql_name, field.name.value),
schema.python_type(field.type, response=True),
)
)
lines.append(f" {field.name.value}: NotRequired[{annotation}]")
lines.extend(("", ""))
return "\n".join(lines).rstrip()
Loading
Loading