feat(datasource-pylon): Pylon datasource for the Ruby agent (EXT-4)

[christophebrun-forest]

Integration branch for EXT-4 — forest_admin_datasource_pylon, feature parity with the Zendesk datasource.

The ten stories of the epic are Done and merged, main is merged in, and the branch touches exactly three files outside the new package.

What is in the branch

A new forest_admin_datasource_pylon package: 44 files under lib, 24 spec files, 786 examples, 100.0% line coverage (1686/1686) and 94.5% branch coverage. Ten story PRs, all reviewed and merged into this branch:

PR	Story	Content
#341	EXT-5	foundation — gem, configuration, resilient client
#347	EXT-6	PylonIssue read-only, cursor pagination
#351	EXT-7	filters, search, count, PK short-circuit hardening
#354	EXT-8	all collections + relations
#358	EXT-9	messages / conversation thread
#359	EXT-10	custom fields introspection
#362	EXT-11	write operations (CRUD)
#366	EXT-12	action plugins: close issue, create issue with notification
#373	EXT-13	story 9 — proactive per-endpoint throttling on top of the 429 retry
#375	EXT-17	bound the boot-time custom-fields introspection

That is every sub-issue of EXT-4 and every PR ever opened against this branch. Re-checked against
Linear on 2026-08-27: EXT-4 has exactly these ten children, all Done, and no eleventh — so the
table above is the whole epic. EXT-4 itself is the only thing left In Progress, this PR being what
closes it.

Three follow-ups were filed out of the review rather than fixed here. They are not sub-issues of
EXT-4 — each is repo-wide, not Pylon's — and they are listed so the table is the full Linear picture:

Ticket	Scope	Why not here
EXT-19	.releaserc.js chains all fourteen successCmd pushes with ;	Fixing only the mambu/pylon boundary the diff touches would leave twelve packages masking failures
EXT-20	No datasource gemspec declares the Forest gems as runtime deps	Declaring them on Pylon alone makes it the only gem with a different contract
EXT-21	A non-positive page limit reads as "no limit" in one place, "empty" in another	Read path of every collection; the two readings have to be settled together

Shape of the package: client.rb + client/writes.rb over the Pylon API with a retry_policy, a rate_limiter / rate_limits / throttle trio metering each endpoint in its own sliding window, a base_collection specialised into cursor_collection / fetch_all_collection, one directory per collection (account, contact, issue, team, user) each with its own api_filters / schema_definition / serializer, a query/ layer translating condition trees into Pylon filters, pagination/cursor_walker, schema/custom_fields_introspector, and two action plugins.

Review pass on the whole branch

65190b9f merges main; the two conflicts were both the #368 sort-decorator cherry-pick, resolved in favour of main, so those files have left this diff. Fifteen commits then close out the review of the assembled branch — these are direct commits, so they appear in no story PR above:

Commit	Kind	Content
b9cb7d15	fix	refuse a Number filter cast to Infinity / NaN; deduplicate action issue ids; stop collect_pages on a cursor already followed
a93474b1	fix	MAX_ID_LOOKUPS bounds a page rather than a selection
24951097	fix	refuse two plugin configurations that misbehave silently
cc015514	fix	keep no column for a projection naming only relations
373f94de	refactor	drop the countable: flag nothing opts into
7033ba1d	fix	stop the custom-field introspection at its first failure
bd55e8ba	docs	the package README, last open item of EXT-13
2a1b4d6d	fix	serve the whole record for an empty projection
67cd786a	fix	refuse an internal note the form never offered
1fd03cbd	fix	keep a select with no option left read-only
96a0575a	docs	record what bounds CloseIssue and what does not
0a2d4e4d	chore	align version.rb with the other packages
d75bab2c	fix	intersect the id conditions of an and, on both the read and the write path
524cc95e	fix	let a refused selection reach the operator of a close action
b6bd085b	docs	correct which scope bounds CloseIssue

Defects fixed

• b9cb7d15 — a Number filter the agent cast to Infinity or NaN reached to_i, which raises, and then the JSON encoder, which raises too: a 500 either way, on a value the operator typed. Now refused with a 400 naming the field. Same commit: an action reading issue ids off a column deduplicates them, and collect_pages stops on a cursor it has already followed rather than only on one that did not move.
• a93474b1 — MAX_ID_LOOKUPS bounded a selection where it should bound a page: the ids were truncated to the first 20 before the window was applied, so an offset at or past 20 answered an empty page from records that exist. The window is now taken off the ids before any of them is read, which also spends one request per record the caller asked to see. A lookup carrying a residual condition cannot be paged that way and is refused past the cap rather than answered with a fraction of itself.
• 24951097 — a plugin scope that is neither a symbol nor a string raised NoMethodError instead of the error naming it; duplicate email-template titles, and a template titled like the "pick none" sentinel, are now refused at registration instead of silently sending the wrong message.
• cc015514 — project served the whole record when the projection named no column of the collection, so a row asked for as account:name came back carrying every native column too. Relevant now that main has landed serve only the columns of collections the caller may read. It also changed what an empty projection returns, which is not that case at all; 2a1b4d6d separates the two.
• 7033ba1d — EXT-17 bounded each boot-time introspection per request, not the three of them together, so a Pylon that hangs could add about a minute to a Rails boot rather than the twenty seconds intended. The first failure now stands for the object types after it.

Also

• 373f94de — drops the countable: flag: no collection ever passed it, Pylon having no count endpoint and no total.
• bd55e8ba — the package README, last open item of EXT-13.

Macroscope: 21 review threads across seven passes, every one answered in thread. The seventh pass landed one finding, on base_collection.rb:107, and it was real: fixed in d75bab2c, described below. Two of the last four findings were real and are fixed (67cd786a, 1fd03cbd); one is a false positive (nil.to_i is 0, so the guard already re-raises) and one is not reachable through the agent stack, which rewrites the date operators upstream — both answered with the reasoning. The normalize_templates thread was Macroscope reading a stale index; the method has been there since 24951097.

Recorded on Linear rather than fixed here: SnoozeIssue (EXT-12) and aggregate / Count (EXT-7) are in the epic scope but were never delivered; both are commented on their story with what exists instead. EXT-4's rate-limit figures were low by a factor of 4 to 15 and have been corrected against rate_limits.rb. The three findings declined as repo-wide now have tickets of their own, so they outlive these threads: EXT-19 (.releaserc.js chaining with ;), EXT-20 (no datasource gemspec declares the Forest gems) and EXT-21 (a non-positive page limit).

Behaviour outside the new package

Three files, each additively, to register the gem: .github/workflows/build.yml (lint matrix, test matrix, codecov files), .releaserc.js (prepareCmd, successCmd, git assets) and .rubocop.yml (MFA opt-out, the version.rb string cops, and the per-file metric excludes the package actually trips). All four points of CLAUDE.md are covered and verified.

Verification

Re-run locally on 0a2d4e4d, Ruby 4.0.2:

• forest_admin_datasource_pylon: 790 examples, 0 failures. Line coverage 100.0% (1691/1691). RuboCop 71 files, 0 offense.
• Earlier on the merged tree: forest_admin_datasource_customizer 703 examples, forest_admin_datasource_toolkit 478, forest_admin_agent 1164 — all 0 failures.

.releaserc.js was loaded through node to confirm both prepareCmd and successCmd are valid concatenations, and both YAML files were parsed. forest_admin_datasource_pylon is unclaimed on RubyGems, so the first publish creates it.

Three fixes from the seventh pass

Local re-run on b6bd085b, Ruby 4.0.2: 794 examples, 0 failures, line coverage 100.0% (1695/1695), branch 94.3%, RuboCop 71 files / 0 offense.

• d75bab2c — an and names the records all of its conditions name, so two id leaves name their intersection. Both the lookup short-circuit and the write-resolution bound kept only the first and left the second to be applied later, which put the cap around the wider set: a selection of 25 ids narrowed to 3 was refused over the 25 it named, after reading all 25 one request each. Worth being exact about the failure — the refusal is an explicit UnsupportedOperatorError, never a short page, so this over-refused rather than under-delivered. extract_id_lookup now partitions the id leaves out of the branch and intersects them, leaving the residual, ensure_residual_appliable! and guard_nil_comparisons untouched; disjoint sets reduce to the empty lookup with no request at all. writes.rb#filtered_ids moves with it, that count being what refuse_unresolvable_selection is worded around — its own comment had been excusing the same shape. Narrow on the read side: only PylonIssue reaches the and branch, cursor collections short-circuiting on a bare leaf alone.
• 524cc95e — IssueTargets.resolve_issue_ids rescued StandardError and answered [], which the action reports as "No Pylon issue selected." That swallowed the datasource's own refusals: a bulk CloseIssue on PylonIssue, with a scope or segment set and more than MAX_ID_LOOKUPS records selected, raises inside list and the operator was told nothing was selected about a selection they can see they made. A Pylon outage read the same way. A ValidationError now travels, the agent surfacing its message; everything else still degrades, so a renamed column is still "no issue selected" rather than a stack trace in the panel.
• b6bd085b — the README's CloseIssue caveat had the scope question backwards. It said a scope on PylonIssue does not bound what the action closes, on the grounds that the state is written through the client. The ids are resolved first, through the collection the action sits on, and Actions#get_record_selection intersects the operator's scope into the filter that reads them — so mounted on PylonIssue, the scope bounds exactly what it closes. What the caveat describes is the host-collection form, where the ids come off a column and that column is the authority. As written it told an operator a scope bought them nothing when it bought them the whole selection.

Open items before merge

• CI is green on the current head. Run 33084670532 on b6bd085b, and 33076526570 before it on 0a2d4e4d: 55 jobs successful, one skipped (Release package, which only runs on main / beta). Both pylon lint jobs and both pylon test jobs pass, Ruby 3.4 included — the earlier runs for 65190b9f and bd55e8ba had waited hours on a runner without ever starting, so this is the first run to exercise the branch on anything but 4.0.
• Deliberately not fixed here, each with the reasoning recorded in thread:
  • CloseIssue's batch is uncapped: one request per selected record, so a wide bulk selection is a long sequential run the request may time out on, leaving what it already closed closed. Stated in the README (96a0575a, reworded in b6bd085b); putting a cap on a bulk action is a product call, not a defect fix. Worth knowing that Writes::MAX_WRITE_REQUESTS refuses exactly this shape at 20 requests on the collection itself, so the datasource currently answers the same question two ways.
  • NIL_UNSAFE_OPERATORS and residual_leaf_appliable? are derived from two different sets, so the guard is narrower than what the residual admits and only the upstream OperatorsEquivalence rewrite closes the gap. Not reachable today; deriving the guard from the equivalence touches the filtering path of every collection.
  • Configuration#validate! checks neither that base_url parses nor that it is https, so a typo surfaces as a raw URI error at the first request and an http:// base url would carry the bearer token in clear. Refusing http would break local mock servers, so this wants a warning rather than a refusal, in its own change.
• A human review.

Left for a follow-up

• One hardening for the base_url change, not done here: Configuration exposes api_key through attr_reader and defines no inspect, so any incidental inspect of a Configuration, Client or Datasource — a Rails error page, a logger.debug of the datasource — prints the bearer token. Same family as the http:// item above, same one-line place to fix it.

🤖 Generated with Claude Code

Note

Add forest_admin_datasource_pylon package with collections, client, and plugins

• Introduces a complete Pylon datasource for the Ruby agent: Datasource, Configuration, Client, and five collections (PylonIssue, PylonAccount, PylonContact, PylonUser, PylonTeam) with cursor-based and fetch-all base classes
• Adds query translation (ConditionTreeTranslator, FilterValue, OperatorMaps), rate limiting (RateLimiter, RateLimits, Throttle), retry policy, and custom-field introspection
• Includes two action plugins: CloseIssue (single/bulk) and CreateIssueWithNotification (form builder + payload shaping), plus IssueTargets resolver and shared Writes module with write budgets
• Ships full RSpec test coverage across client, collections, plugins, pagination, and configuration; CI workflow and release config updated to build/publish the gem
• Risk: new package with no prior consumers; reviewers should verify operator allow-lists in collections/*/api_filters.rb and rate-limit budgets in rate_limits.rb match the live Pylon API contract

^{Macroscope summarized bd55e8b.}

[linear-code]

EXT-4

[macroscopeapp]

🟠 High .releaserc.js:47

A failed forest_admin_datasource_mambu_payments build or push is reported as a successful successCmd when the subsequent Pylon publish succeeds, so semantic-release can mark a partially published release as successful. The Mambu subshell is separated from the Pylon subshell with ;, making Pylon's status the overall status; chain them with && so the Mambu failure propagates.

Suggested change

	'( cd packages/forest_admin_datasource_mambu_payments && gem build && gem push forest_admin_datasource_mambu_payments-*.gem );' +
	'( cd packages/forest_admin_datasource_mambu_payments && gem build && gem push forest_admin_datasource_mambu_payments-*.gem ) &&' +

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @.releaserc.js around line 47:

A failed `forest_admin_datasource_mambu_payments` build or push is reported as a successful `successCmd` when the subsequent Pylon publish succeeds, so semantic-release can mark a partially published release as successful. The Mambu subshell is separated from the Pylon subshell with `;`, making Pylon's status the overall status; chain them with `&&` so the Mambu failure propagates.

[christophebrun-forest]

Valid, but not this PR's to fix.

All fourteen packages in successCmd are chained with ;, and have been since long before this branch — the mambu/pylon boundary is just where the diff happens to touch the list. Changing only that one separator would leave the other twelve masking failures exactly as they do today, so the fix belongs in a repo-wide change against main.

The underlying point stands and is worth its own ticket: a gem push failing anywhere in the chain is currently reported as a successful release.

[macroscopeapp]

Agreed — fixing all fourteen packages consistently makes more sense than patching just the diff boundary. Would you like me to open a PR against main that chains all the successCmd gem operations with && instead of ; so any build or push failure properly propagates?

[christophebrun-forest]

Filed as EXT-19 so it outlives this thread — all fourteen packages, as a change against main.

[macroscopeapp]

🟠 High forest_admin_datasource_pylon/forest_admin_datasource_pylon.gemspec:32

Installing this gem from RubyGems omits forest_admin_datasource_toolkit and forest_admin_datasource_customizer, so require 'forest_admin_datasource_pylon' fails with LoadError in a clean application. The development Gemfile does not provide runtime gem dependencies; declare both Forest gems in this gemspec.

  spec.add_dependency 'activesupport', '>= 6.1'
+  spec.add_dependency 'forest_admin_datasource_toolkit'
+  spec.add_dependency 'forest_admin_datasource_customizer'

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @packages/forest_admin_datasource_pylon/forest_admin_datasource_pylon.gemspec around line 32:

Installing this gem from RubyGems omits `forest_admin_datasource_toolkit` and `forest_admin_datasource_customizer`, so `require 'forest_admin_datasource_pylon'` fails with `LoadError` in a clean application. The development `Gemfile` does not provide runtime gem dependencies; declare both Forest gems in this gemspec.

[christophebrun-forest]

Acknowledged, deliberately consistent with the monorepo.

No datasource gem here declares the Forest gems as runtime dependencies — zendesk, graphql_hasura, snowflake and mambu_payments all omit both, and every one of them requires the toolkit at load exactly as this one does. They are brought in by the agent application through forest_admin_rails / forest_admin_agent.

Declaring them on Pylon alone would make it the only gem in the repo with a different dependency contract, without fixing the other four. If the contract is wrong it is wrong repo-wide, which is a separate change against main.

[christophebrun-forest]

Filed as EXT-20 so it outlives this thread — all five datasource gemspecs, with the pinning question stated rather than assumed.

[qltysh]

Coverage Impact

⬆️ Merging this pull request will increase total coverage on main by 0.6%.

Modified Files with Diff Coverage (43)

Rating	File	% Diff	Uncovered Line #s
	...forest_admin_datasource_pylon/collections/cursor_collection.rb	100.0%
	...pylon/lib/forest_admin_datasource_pylon/collections/contact.rb	100.0%
	...orest_admin_datasource_pylon/collections/contact/serializer.rb	100.0%
	..._pylon/lib/forest_admin_datasource_pylon/query/filter_value.rb	100.0%
	...tasource_pylon/lib/forest_admin_datasource_pylon/datasource.rb	100.0%
	...pylon/lib/forest_admin_datasource_pylon/query/operator_maps.rb	100.0%
	..._pylon/lib/forest_admin_datasource_pylon/collections/writes.rb	100.0%
	...st_admin_datasource_pylon/lib/forest_admin_datasource_pylon.rb	100.0%
	...b/forest_admin_datasource_pylon/collections/base_collection.rb	100.0%
	...asource_pylon/lib/forest_admin_datasource_pylon/rate_limits.rb	100.0%
	...ce_pylon/lib/forest_admin_datasource_pylon/collections/team.rb	100.0%
	...ce_pylon/lib/forest_admin_datasource_pylon/collections/user.rb	100.0%
	...rest_admin_datasource_pylon/collections/contact/api_filters.rb	100.0%
	...est_admin_datasource_pylon/collections/fetch_all_collection.rb	100.0%
	..._admin_datasource_pylon/collections/issue/messages_embedder.rb	100.0%
	...e_pylon/plugins/create_issue_with_notification/form_builder.rb	100.0%
	...lon/lib/forest_admin_datasource_pylon/plugins/issue_targets.rb	100.0%
	...source_pylon/lib/forest_admin_datasource_pylon/rate_limiter.rb	100.0%
	...pylon/lib/forest_admin_datasource_pylon/plugins/close_issue.rb	100.0%
	...dmin_datasource_pylon/collections/account/schema_definition.rb	100.0%
	...t_admin_datasource_pylon/collections/issue/id_lookup_reader.rb	100.0%
	...source_pylon/plugins/create_issue_with_notification/payload.rb	100.0%
	...rest_admin_datasource_pylon/collections/account/api_filters.rb	100.0%
	.../forest_admin_datasource_pylon/plugins/close_issue/messages.rb	100.0%
	...min_datasource_pylon/plugins/create_issue_with_notification.rb	100.0%
	...asource_pylon/lib/forest_admin_datasource_pylon/issue_enums.rb	100.0%
	...n_datasource_pylon/lib/forest_admin_datasource_pylon/client.rb	100.0%
	...ource_pylon/lib/forest_admin_datasource_pylon/configuration.rb	100.0%
	...forest_admin_datasource_pylon/collections/relation_embedder.rb	100.0%
	.../forest_admin_datasource_pylon/collections/issue/serializer.rb	100.0%
	..._admin_datasource_pylon/collections/issue/schema_definition.rb	100.0%
	...est_admin_datasource_pylon/collections/record_serialization.rb	100.0%
	...orest_admin_datasource_pylon/collections/account/serializer.rb	100.0%
	...forest_admin_datasource_pylon/collections/issue/api_filters.rb	100.0%
	...e_pylon/lib/forest_admin_datasource_pylon/collections/issue.rb	100.0%
	...pylon/lib/forest_admin_datasource_pylon/collections/account.rb	100.0%
	.../lib/forest_admin_datasource_pylon/pagination/cursor_walker.rb	100.0%
	...dmin_datasource_pylon/collections/contact/schema_definition.rb	100.0%
	...rest_admin_datasource_pylon/query/condition_tree_translator.rb	100.0%
	...ource_pylon/lib/forest_admin_datasource_pylon/client/writes.rb	100.0%
	...st_admin_datasource_pylon/schema/custom_fields_introspector.rb	100.0%
	...source_pylon/lib/forest_admin_datasource_pylon/retry_policy.rb	100.0%
	...datasource_pylon/lib/forest_admin_datasource_pylon/throttle.rb	100.0%
	Total	100.0%

🚦 See full report on Qlty Cloud »

🛟 Help

• Diff Coverage: Coverage for added or modified lines of code (excludes deleted files). Learn more.

• Total Coverage: Coverage for the whole repository, calculated as the sum of all File Coverage. Learn more.

• File Coverage: Covered Lines divided by Covered Lines plus Missed Lines. (Excludes non-executable lines including blank lines and comments.)

  • Indirect Changes: Changes to File Coverage for files that were not modified in this PR. Learn more.

[macroscopeapp]

🟡 Medium issue/id_lookup_reader.rb:29

A missing or mismatched ID consumes a slot in the requested page, so records_by_id can return fewer rows and hide later reachable IDs. page_of_ids must be applied after fetch_by_ids resolves the IDs, not before the reads.

-          return fetch_by_ids(page_of_ids(lookup.ids, filter)) if lookup.residual.nil?
+          return page_window(fetch_by_ids(lookup.ids), filter) if lookup.residual.nil?

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @packages/forest_admin_datasource_pylon/lib/forest_admin_datasource_pylon/collections/issue/id_lookup_reader.rb around line 29:

A missing or mismatched ID consumes a slot in the requested page, so `records_by_id` can return fewer rows and hide later reachable IDs. `page_of_ids` must be applied after `fetch_by_ids` resolves the IDs, not before the reads.

[christophebrun-forest]

This is the behaviour a93474b deliberately replaced, so the suggested diff would restore the bug it fixed.

page_window(fetch_by_ids(lookup.ids), filter) reads the whole selection to then drop all but a window of it — one request per record nobody asked to see — and, because the selection is capped at MAX_ID_LOOKUPS before that, answers any offset at or past 20 with an empty page. That is the finding in the thread on issue.rb:139 from the previous pass.

The trade is a real one and it is stated in the code: the window is over the ids, so an id that no longer resolves leaves a gap rather than pulling the next one in behind it. Closing that gap costs a request per record outside the page to discover, and dangling ids in an id IN (...) selection are the exception — Forest builds those lists from ids it has just read. A spec pins each half of the behaviour.

[qltysh]

41 new issues

Tool	Category	Rule	Count
qlty	Structure	Function with many parameters (count = 4): search_issues	23
qlty	Structure	Function with high complexity (count = 10): collect_pages	16
qlty	Duplication	Found 17 lines of identical code in 4 locations (mass = 76)	1
qlty	Structure	High total complexity (count = 57)	1

[qltysh]

Found 17 lines of identical code in 4 locations (mass = 76) [qlty:identical-code]

[qltysh]

Function with many parameters (count = 4): search_issues [qlty:function-parameters]

[qltysh]

Function with many parameters (count = 4): search_accounts [qlty:function-parameters]

[qltysh]

Function with many parameters (count = 4): search_contacts [qlty:function-parameters]

[qltysh]

Function with many parameters (count = 5): search_resource [qlty:function-parameters]

[qltysh]

Function with many parameters (count = 4): success_message [qlty:function-parameters]

[qltysh]

Function with high complexity (count = 6): requester_default [qlty:function-complexity]

[qltysh]

Function with high complexity (count = 5): interpolate [qlty:function-complexity]

[qltysh]

Function with high complexity (count = 7): resolve_issue_ids [qlty:function-complexity]

[qltysh]

Function with many parameters (count = 5): initialize [qlty:function-parameters]