feat(datasource-pylon): Pylon datasource for the Ruby agent (EXT-4) #369
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| *.gem | ||
| .bundle/ | ||
| Gemfile.lock | ||
| Gemfile-test.lock | ||
| coverage/ | ||
| pkg/ | ||
| tmp/ | ||
| .rspec_status |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| --format documentation | ||
| --color | ||
| --require spec_helper |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| source 'https://rubygems.org' | ||
|
|
||
| gemspec | ||
|
|
||
| gem 'forest_admin_datasource_customizer' | ||
| gem 'forest_admin_datasource_toolkit' | ||
| gem 'rake', '~> 13.0' | ||
| gem 'rubocop', '1.86.1' | ||
| gem 'rubocop-performance', '1.26.1' | ||
| gem 'rubocop-rspec', '3.9.0' | ||
|
|
||
| group :development, :test do | ||
| gem 'rspec', '~> 3.0' | ||
| gem 'simplecov', '~> 0.22', require: false | ||
| gem 'webmock', '~> 3.0' | ||
| end | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| source 'https://rubygems.org' | ||
|
|
||
| # Specify your gem's dependencies in forest_admin_datasource_pylon.gemspec | ||
| gemspec | ||
|
|
||
| gem 'rake', '~> 13.0' | ||
| gem 'rubocop', '1.86.1' | ||
| gem 'rubocop-performance', '1.26.1' | ||
| gem 'rubocop-rspec', '3.9.0' | ||
|
|
||
| group :development, :test do | ||
| gem 'forest_admin_datasource_customizer', path: '../forest_admin_datasource_customizer' | ||
| gem 'forest_admin_datasource_toolkit', path: '../forest_admin_datasource_toolkit' | ||
| gem 'rspec', '~> 3.0' | ||
| gem 'simplecov', '~> 0.22', require: false | ||
| gem 'simplecov-html', '~> 0.12.3' | ||
| gem 'simplecov_json_formatter', '~> 0.1.4' | ||
| gem 'webmock', '~> 3.0' | ||
| end |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,159 @@ | ||
| # Forest — Pylon datasource | ||
|
|
||
| Surface [Pylon](https://usepylon.com) issues, accounts, contacts, users and teams as Forest | ||
| collections, with filters, free-text search, relations, the conversation thread, custom fields, | ||
| CRUD writes and two action plugins. | ||
|
|
||
| ## Installation | ||
|
|
||
| ```ruby | ||
| # Gemfile | ||
| gem 'forest_admin_datasource_pylon' | ||
| ``` | ||
|
|
||
| ## Usage | ||
|
|
||
| ```ruby | ||
| # app/lib/forest_admin_rails/create_agent.rb | ||
| ForestAdminAgent::Builder::AgentFactory.instance.add_datasource( | ||
| ForestAdminDatasourcePylon::Datasource.new(api_key: ENV['PYLON_API_KEY']) | ||
| ) | ||
| ``` | ||
|
|
||
| A single Bearer token is the whole configuration. `GET /me` is the health check: it returns the | ||
| organization owning the token, which is enough to prove the credentials are usable. | ||
|
|
||
| ## Collections | ||
|
|
||
| | Collection | Pylon resource | Read | Write | | ||
| | --- | --- | --- | --- | | ||
| | `PylonIssue` | `/issues` | `POST /issues/search`, cursor-paginated | create / update / delete | | ||
| | `PylonAccount` | `/accounts` | search + list, cursor-paginated | create / update / delete | | ||
| | `PylonContact` | `/contacts` | search + list, cursor-paginated | create / update / delete | | ||
| | `PylonUser` | `/users` | whole dataset, unpaginated | update | | ||
| | `PylonTeam` | `/teams` | whole dataset, unpaginated | create / update | | ||
|
|
||
| `PylonIssue`, `PylonAccount` and `PylonContact` also carry their Pylon custom fields, introspected | ||
| at boot from `GET /custom-fields`. | ||
|
|
||
| ## What the API cannot do, and what this does about it | ||
|
|
||
| Pylon is a ticketing API, not a database, and several things Forest asks for have no equivalent. | ||
| Where that happens this datasource **refuses with a message naming the reason** rather than | ||
| answering something that looks right and is not. All of these arrive as a 400 carrying the text. | ||
|
|
||
| - **No aggregate endpoint and no total.** `aggregate` is refused on the cursor-backed collections: | ||
| counting the pages a walk collected would answer a fraction of a collection as if it were the | ||
| whole of it, so those are not advertised as countable. `PylonUser` and `PylonTeam` are the | ||
| exception — their endpoint hands back every record, so a count or a group over it is the figure | ||
| a server-side aggregation would have given. | ||
| - **No sort parameter on `/issues/search`.** Issues always come back newest first. A requested | ||
| order is reported in the log rather than silently swallowed. | ||
| - **No `id` filter on the search endpoints.** A primary-key lookup is short-circuited to | ||
| `GET /issues/{id}`, one request per id. That fan-out is capped per page (`MAX_ID_LOOKUPS`), and | ||
| a selection naming more ids *and* filtering them further is refused, since which records the | ||
| page holds could only be known by reading all of them. | ||
| - **No joins.** A condition on a relation is answered by reading the foreign collection for its | ||
| keys and sending them as an `in`. Past `MAX_RELATION_KEYS` the condition is refused. | ||
| - **Search and id lookup are different endpoints.** Combining a free-text search with a filter on | ||
| `id` is refused: neither endpoint can do the other's half. | ||
| - **Writes are one record per request.** A write reaching more records than one pass covers is | ||
| refused up front; one that fails halfway reports exactly which records were written, so a retry | ||
| can target the untouched ones rather than performing the write twice. | ||
|
|
||
| ## Rate limits | ||
|
|
||
| Pylon meters **per endpoint**, not per token, from 30 to 300 requests a minute depending on the | ||
| endpoint. `RateLimits` holds the documented budget of every endpoint this client calls, and | ||
| `RateLimiter` spaces requests out so each one is spent rather than exceeded — a sliding window per | ||
| endpoint, in front of the 429 retry rather than instead of it. | ||
|
|
||
| The limiter is a smoother, not a guarantee: past `DEFAULT_MAX_WAIT` a request goes out anyway and | ||
| the 429 retry takes over, with one log line per endpoint per window saying so. Under real | ||
| saturation — several agents or processes on the same token — the retry is the defence. | ||
|
|
||
| To meter on your own side instead, pass `rate_limiter: nil`: | ||
|
|
||
| ```ruby | ||
| ForestAdminDatasourcePylon::Datasource.new(api_key: ENV['PYLON_API_KEY'], rate_limiter: nil) | ||
| ``` | ||
|
|
||
| ## Boot-time introspection | ||
|
|
||
| Custom fields are read while the datasource is being constructed, one call per object type. That | ||
| sits in front of a Rails boot, so it runs on its own connection with short timeouts and a single | ||
| quick retry (`boot_open_timeout`, `boot_timeout`, `boot_retry_policy`), and the first failure | ||
| stops the remaining object types from being tried. | ||
|
|
||
| An introspection that fails costs the custom columns, not the datasource: the agent boots on the | ||
| native schema and says so in the log. | ||
|
|
||
| ## Action plugins | ||
|
|
||
| ### `CloseIssue` | ||
|
|
||
| Moves the selected issues to a state, single and bulk, one request per issue with a per-id rescue | ||
| so one failure does not abort the batch. | ||
|
|
||
| ```ruby | ||
| @agent.collection :Ticket do |collection| | ||
| collection.use(ForestAdminDatasourcePylon::Plugins::CloseIssue, { | ||
| datasource: pylon_datasource, # required | ||
| issue_id_field: 'pylon_issue_id', # omit when the action sits on PylonIssue itself | ||
| state: 'closed', # default | ||
| scopes: %i[single bulk] # default | ||
| }) | ||
| end | ||
| ``` | ||
|
|
||
| Two things to know before granting it. | ||
|
|
||
| **Which scope bounds it.** The state is written straight through the Pylon client rather than | ||
| through `PylonIssue`, because the action is registered on the host collection — but the ids were | ||
| read before that write, through the collection the action sits on, and the agent intersects the | ||
| operator's scope into the filter that read them. So the scope that bounds this is the scope of the | ||
| **host** collection: mounted on `PylonIssue`, a scope or segment on `PylonIssue` bounds exactly what | ||
| it closes; mounted on a business collection with `issue_id_field`, what bounds it is the records the | ||
| operator may see there, and the issue ids those records carry. In that second form the column is | ||
| the authority — an operator who can write it can name any Pylon issue — so treat it as one. | ||
|
|
||
| **The batch is capped at `MAX_TARGETS` (20) issues**, the budget a filter-driven write already gets: | ||
| Pylon takes one request per issue, so a wider selection is a long run of sequential writes the | ||
| request may time out on, leaving the issues closed up to that point closed and reporting which ones | ||
| to nobody. Past the cap the run is refused before its first write, the way `update` and `delete` | ||
| refuse a selection costing more than `MAX_WRITE_REQUESTS`. The cap counts the issues named, not the | ||
| records selected: a column of issue ids is not a key, so a hundred host records naming ten issues is | ||
| a batch of ten. | ||
|
|
||
| ### `CreateIssueWithNotification` | ||
|
|
||
| Creates an issue from a form and notifies the requester, optionally writing the new issue id back | ||
| onto the host record. | ||
|
|
||
| ```ruby | ||
| @agent.collection :Customer do |collection| | ||
| collection.use(ForestAdminDatasourcePylon::Plugins::CreateIssueWithNotification, { | ||
| datasource: pylon_datasource, # required | ||
| sender_email: 'support@example.com', # required when the destination is email | ||
| issue_id_field: 'pylon_issue_id', # optional id writeback | ||
| email_templates: [{ title: 'Outage', content: '<p>Sorry {{ record.name }}</p>' }] | ||
| }) | ||
| end | ||
| ``` | ||
|
|
||
| `{{ record.field }}` tokens are interpolated from the host record, HTML-escaped in the message | ||
| body. Template titles must be unique and cannot be `"No template"`, which names the option that | ||
| picks none of them. | ||
|
|
||
| The message body is HTML the operator writes and Pylon delivers to the requester, and the | ||
| requester address is a free-text field. Restrict both actions to the roles that should be able to | ||
| send mail on your organization's behalf. | ||
|
|
||
| ## Development | ||
|
|
||
| ```bash | ||
| cd packages/forest_admin_datasource_pylon | ||
| bundle install | ||
| bundle exec rspec | ||
| bundle exec rubocop | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| require 'bundler/gem_tasks' | ||
| require 'rspec/core/rake_task' | ||
|
|
||
| RSpec::Core::RakeTask.new(:spec) | ||
|
|
||
| task default: :spec |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟠 High
.releaserc.js:47A failed
forest_admin_datasource_mambu_paymentsbuild or push is reported as a successfulsuccessCmdwhen the subsequent Pylon publish succeeds, so semantic-release can mark a partially published release as successful. The Mambu subshell is separated from the Pylon subshell with;, making Pylon's status the overall status; chain them with&&so the Mambu failure propagates.🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Valid, but not this PR's to fix.
All fourteen packages in
successCmdare chained with;, and have been since long before this branch — the mambu/pylon boundary is just where the diff happens to touch the list. Changing only that one separator would leave the other twelve masking failures exactly as they do today, so the fix belongs in a repo-wide change againstmain.The underlying point stands and is worth its own ticket: a
gem pushfailing anywhere in the chain is currently reported as a successful release.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed — fixing all fourteen packages consistently makes more sense than patching just the diff boundary. Would you like me to open a PR against
mainthat chains all thesuccessCmdgem operations with&&instead of;so any build or push failure properly propagates?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Filed as EXT-19 so it outlives this thread — all fourteen packages, as a change against
main.