Skip to content

BED-8693: Complete data collection regardless of single object errors#17

Merged
StranDutton merged 2 commits into
mainfrom
fix/BED-8693-error-resilience
Jun 24, 2026
Merged

BED-8693: Complete data collection regardless of single object errors#17
StranDutton merged 2 commits into
mainfrom
fix/BED-8693-error-resilience

Conversation

@StranDutton

@StranDutton StranDutton commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Adds try/except error handling to all resource and transformer functions in the OpenHound GitHub extension so that a failure on a single object (org, repo, team, etc.) doesn't abort the entire collection.

summary of changes:

  • organization.py: Wrapped multi-org resource functions with per-object error handling
  • enterprise.py: Wrapped multi-org function with per-object error handling
  • Resource functions (multi-org loops): catch + log + continue to next org
  • Unit tests added to confirm error resilience logic

Log format follows the existing safe_resource_wrapper convention

The extra bit in the error log messaging is included because they get emitted as top-level keys in JSON logs. It's only useful for log aggregation in BHE service deployments (they're not visible in CLI mode)

Complementary to PR #32 which handles schema/data-type failures. This PR handles API/network failures, but both apply to the same ticket

Validation: Successfully ran a collection to SpecterOps GitHub org

@StranDutton StranDutton self-assigned this Jun 24, 2026
@d3vzer0

d3vzer0 commented Jun 24, 2026

Copy link
Copy Markdown
Collaborator

@StranDutton Wrapping every resource and transformer in a try/except should not be required. The safe_resource_wrapper function should take care of generic exception handling for every @app.resource and @app.transformer. I believe we're only missing exception handling for DLT's defer.

@StranDutton

Copy link
Copy Markdown
Contributor Author

@d3vzer0 I removed the exception handling for the transformers, since the safe_resource_wrapper catches exceptions at that level and DLT moves on to the next item, so my additions there were redundant. The issue I am trying solve has more to do with the multi-org resource functions.

I have a unit test that proves the bug in the PR, but the scenario occurs when, say, you're iterating over 3 orgs, and an exception gets thrown during iteration of org2. The safe resource wrapper does catch that and calls continue, but at this point, the generator is already dead and can't be resumed after an unhandled exception. The try/except has to be inside the loop to isolate each org and prevent data loss for the remaining orgs if an exception is thrown during collection of 1 of them. In the case that I mentioned, we would silently lose org 3.

@d3vzer0 d3vzer0 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Lets also think about a future update to the core OpenHound library to support generic exception handling for fan-out resources similar to GitHub (ie. when an enterprise starts collection for multiple orgs) :)

@StranDutton StranDutton merged commit 1989d73 into main Jun 24, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants