Skip to content

fix: encode element handle selectors so nested and special-character queries work - #6

Merged
Gikoskos merged 2 commits into
mainfrom
fix/element-handle-selectors
Nov 13, 2026
Merged

Gikoskos merged 2 commits into
mainfrom
fix/element-handle-selectors

Conversation

@artiom

@artiom artiom commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Encode element-handle selectors as URI-encoded JSON literals so nested queries and scoped selectors containing quotes, backslashes or newlines survive JavaScript string interpolation.
  • Forward timeout and visible from scoped queries in both async and sync APIs.
  • Build the indexed-handle expression once per nonempty query_selector_all result, with regenerated Page code.

Related issues

None.

Type of change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that changes existing behavior)
  • Documentation only
  • Chore / tooling

Checklist

  • ruff format --check .
  • ruff check .
  • mypy
  • pytest -m "not e2e" — 211 passed, 1 skipped, 5 deselected on Python 3.13
  • python scripts/codegen.py leaves no diff
  • Tests added or updated
  • Docs reviewed; no documentation changes needed
  • Did not hand-edit generated files or AUTOGEN regions
  • Title follows Conventional Commits (type: summary)
  • Title, description and commits contain no internal ticket IDs or internal links

Notes for reviewers

Live dev acceptance reported by the reviewer: review and results. The reviewer ran pytest -m e2e: 5/5 pass, including the new nested/special-character regression, which fails on main with "undefined" is not valid JSON. A 22-case async probe matches the TypeScript SDK throughout (main fails 11), and a sync run with the new options passes. These are reviewer-run results, not a claim that the author reran the dev suite. No production tests were run by this PR's author.

Offline checks also pass for the schema lock, package build, twine check, and fresh-wheel async/sync imports. A local Chromium probe simulating quote-only string interpolation failed all 17 nested/special-character cases on the original encoding and passed all 17 on this branch. That probe does not replace live-service acceptance.

Integration with #8: whichever PR lands second must preserve the encoded selectors, scoped timeout/visible options and html= forwarding in both async and sync APIs, regenerate generated files, retain forwarding tests and recount the live-suite skip expectation. Neither PR is imported into the other here.

ElementHandle.selector is an opaque expression: its text changes for scoped and indexed handles. Page-level selectors remain unchanged. Page-wide backslash/newline limitations and an HTML opt-in are not changed here. No runtime dependency, version or changelog changes.

Summary by CodeRabbit

  • New Features
    • Element-level queries now accept query options in both asynchronous and synchronous APIs, including options forwarded to the underlying page query.
    • Selector matching supports special characters and nested element queries, with more reliable handling of quoted, escaped, and Unicode selectors.
  • Bug Fixes
    • Empty query results now return an empty list without attempting selector processing.
    • Queries with no matches continue to return None for a single result or an empty list for multiple results.

…queries work

Co-authored-by: Artiom Lunev <artiom@browserless.io>
@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Selector expressions now encode selector literals and support scoped queries. Page-level multi-match queries build handles from one shared expression and return an empty list for empty results. Async and sync ElementHandle query methods accept options.

Changes

Selector Queries

Layer / File(s) Summary
Selector encoding and scoped queries
src/browserless_bap/element_handle.py, tests/test_element_handle.py, tests/test_page_overrides.py, tests/e2e/test_live.py
ElementHandle uses encoded selector expressions for document and parent-scoped queries. Async query methods forward options. Tests cover selector encoding, scoped selection, and query options.
Page-level multi-match handles
scripts/codegen/emit_page.py, src/browserless_bap/_page_generated.py, tests/test_page_overrides.py, tests/e2e/test_config.py
Generated methods return an empty list for falsey results. Otherwise, they build handles using indexed expressions based on one shared selector expression. Tests cover encoding calls and the updated skipped-test count.
Sync wrapper option forwarding
src/browserless_bap/sync_api/element_handle.py, tests/sync/test_wrappers.py
Sync ElementHandle query methods accept typed keyword options and forward them through _invoke. Wrapper tests verify that the options reach the async test double.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Suggested reviewers: gikoskos, adriansillo

Merge Risk: Minimal · up to a3eb0

This change fixes element-handle queries whose selectors contain quotes, backslashes or newlines, and it forwards timeout and visible options for scoped queries in both async and sync APIs. The offline tests are reported passing, and a reviewer reports a live development-service run passing. No concrete merge-blocking risk was found. The author should coordinate with PR #8, which touches the same query methods and generated files.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: element handle selectors are now encoded so nested and special-character queries work. The raw summary and objectives confirm this. The selector encoding change in ele…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Full details: Title check

Explanation

The title names the main change: element handle selectors are now encoded so nested and special-character queries work. The raw summary and objectives confirm this. The selector encoding change in element_handle.py, _page_generated.py, and the generated page code is the core fix, and the new e2e and unit tests cover it. The title does not mention the forwarded timeout and visible options, but that is secondary and the title does not need to cover every detail.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

A rabbit checks each selector line,
Encoded strings now travel fine.
Nested handles find their way,
Empty lists return without delay.
Options follow through the sync API,
The bunny hops, pleased as can be.

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Team
  • Run ID: 75c6a122-7545-4f38-bc5e-6db50ee4f5cd
Commits

Reviewing files that changed from the base of the PR and between dc35e51 and a3eb03c.

Files selected for processing: 1

  • src/browserless_bap/element_handle.py
Files skipped from review as they are similar to previous changes: 1

  • src/browserless_bap/element_handle.py

Included review availability: This review used your included allowance. 0 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

artiom commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator Author

All 13 current-head checks pass, including Python 3.11–3.14, lint/type checks, wheel, workflow lint, public-content validation and CodeQL. CodeRabbit completed with no actionable comments; there are no unresolved review threads and no merge conflict. No review fixes were needed. This remains a draft: the development-service live regression run is still outstanding because no verified development credential is available. Local Chromium interpolation checks do not replace that live acceptance. No production tests were run.

@Gikoskos

Copy link
Copy Markdown
Collaborator

Review: ready once the description is updated and the PR is marked ready

The encoding is correct. Against @browserless.io/bap-ts@2.2.0 from npm pack, I compared 20 selectors, each in 7 forms: the encoded selector alone, page-wide single and list lookups, an indexed handle, scoped single and list queries, and a 3-level nested chain. Of the 140 strings, 112 are byte-identical. The rest come from 4 inputs (non-ASCII text, emoji, U+2028 and 0x7F), which this PR sends as \uXXXX escapes. All 20 outputs decode back to the original selector in Node.

Live acceptance on dev, which the PR still lists as outstanding:

  • pytest -m e2e passes 5/5, including the new nested and special-character test.
  • The same new test fails on main with "undefined" is not valid JSON, so it catches the bug.
  • A 22-case async probe matches bap-ts on every case:
    • scoped searches with a quote, backslash, newline, non-ASCII text, emoji, U+2028 and 0x7F;
    • a 4-level nested chain;
    • typing into an input whose name contains a quote and a backslash;
    • hover and screenshot on a nested element;
    • a list lookup written as a JavaScript expression;
    • a parent element removed after it was grabbed.
  • Main fails 11 of the same 22.
  • A sync run with the new options also works.

Lint, mypy and the offline suite pass on 3.11 and 3.14 (211 passed, 1 skipped), and codegen leaves no diff.

Important

  1. Update the description and mark the PR ready. "Notes for reviewers" still says live acceptance is outstanding; the results above cover it.
  2. Coordinate with feat!: make element HTML and response bodies opt-in and rename builtin-shadowing errors #8. feat!: make element HTML and response bodies opt-in and rename builtin-shadowing errors #8 rewrites the same element-handle query_selector / query_selector_all (async and sync) on the old encoding, adds an html= keyword, and touches codegen and the generated page file. Whichever PR merges second must:
    • keep this encoding;
    • accept both the new options and html=;
    • regenerate the generated files instead of hand-merging them;
    • keep or add a test that each option is forwarded.

Minor

Co-authored-by: Artiom Lunev <artiom@browserless.io>

artiom commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator Author

Added the explanation that JSON escapes lone UTF-16 surrogates before URI encoding. The existing safety test already asserts on the raw URI-encoded payload: json.loads removes only the enclosing JavaScript string literal; unquote happens after both safety assertions, solely for the round-trip check. No assertion change is needed.

Updated the description to attribute the reported dev acceptance to the reviewer and preserve the integration requirements with #8. Main currently has no overlapping SDK changes, so there is no conflict to rebase or live-test skip count to recount yet. Fresh local verification: 211 offline tests pass, 1 skipped; formatting, lint, mypy, codegen drift and schema checks pass. The only pushed source change is a comment; runtime behavior and the reviewed live regression are unchanged. Readiness will follow the new head CI/review checks.

@artiom
artiom marked this pull request as ready for review November 13, 2026 14:29
@artiom
artiom requested a review from a team as a code owner November 13, 2026 14:29

artiom commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator Author

Review follow-up is complete. The description now records the reviewer-reported dev acceptance, the lone-surrogate explanation is added, and the existing raw-URI safety assertion is retained with the rationale above. All 13 current-head checks pass, including Python 3.11–3.14, lint, wheel, public rules/title, workflow lint and CodeQL. CodeRabbit reports no actionable comments; no review threads or merge conflict remain. Marked ready for review. Integration with #8 still requires preserving encoding, scoped options and html forwarding together, regenerating outputs and recounting the live-suite skip expectation when the second PR lands.

@Gikoskos Gikoskos left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved: the review findings are addressed, and live acceptance on dev passed.

@Gikoskos
Gikoskos merged commit f22f12b into main Nov 13, 2026
17 checks passed
@Gikoskos
Gikoskos deleted the fix/element-handle-selectors branch November 13, 2026 16:18
Gikoskos pushed a commit that referenced this pull request Nov 13, 2026
🤖 I have created a release *beep* *boop*
---


##
[0.3.0](v0.2.0...v0.3.0)
(2026-11-13)


### ⚠ BREAKING CHANGES

* `Page.query_selector()` and `Page.query_selector_all()` no longer
fetch `innerHTML`/`outerHTML`; pass `html=True`. Reading
`ElementHandle.inner_html` or `outer_html` on a handle queried without
it raises `BAPValueError`. `Page.response()` no longer returns `body`;
pass `body=True` (or `body_encoding=True`, which implies it). the
exceptions `TimeoutError`, `ConnectionError` and `ValueError` are
removed from `browserless_bap` and `browserless_bap.errors`; use
`BAPTimeoutError`, `BAPConnectionError` and `BAPValueError` (they still
subclass the matching built-in exceptions).
`browserless_bap.graphql.RESPONSE_FIELDS` no longer has the
`HTTPResponseWithRemoteMetadata` and `ResponseResponseWithBodyEncoding`
keys; use `OPTIONAL_FIELDS` with `selection_for()`.
`QuerySelectorResponse["childElementCount"]` is typed `int` (was
`float`), and
`browserless_bap.sync_api.Browserless.connect(transport=...)` is typed
`TransportFactory | None` (was `Any`).
* Opening a page now times out after `connect_timeout` (default: the
`timeout` value, 30000 ms). Pass `connect_timeout=0` to restore the
previous unbounded wait.

### Features

* bound connection time and harden the transport
([#9](#9))
([1689f3b](1689f3b))
* make element HTML and response bodies opt-in and rename
builtin-shadowing errors
([#8](#8))
([bc80671](bc80671))


### Bug Fixes

* encode element handle selectors so nested and special-character
queries work ([#6](#6))
([f22f12b](f22f12b))
* sync API listener delivery, wait_for_function predicates and parity
checks ([#7](#7))
([e5a106d](e5a106d))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: browserless-actions-bot[bot] <186328842+browserless-actions-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants