BED-8693: Complete data collection regardless of single object errors#17
Conversation
|
@StranDutton Wrapping every resource and transformer in a try/except should not be required. The |
|
@d3vzer0 I removed the exception handling for the transformers, since the safe_resource_wrapper catches exceptions at that level and DLT moves on to the next item, so my additions there were redundant. The issue I am trying solve has more to do with the multi-org resource functions. I have a unit test that proves the bug in the PR, but the scenario occurs when, say, you're iterating over 3 orgs, and an exception gets thrown during iteration of org2. The safe resource wrapper does catch that and calls continue, but at this point, the generator is already dead and can't be resumed after an unhandled exception. The try/except has to be inside the loop to isolate each org and prevent data loss for the remaining orgs if an exception is thrown during collection of 1 of them. In the case that I mentioned, we would silently lose org 3. |
d3vzer0
left a comment
There was a problem hiding this comment.
Looks good to me! Lets also think about a future update to the core OpenHound library to support generic exception handling for fan-out resources similar to GitHub (ie. when an enterprise starts collection for multiple orgs) :)
Adds try/except error handling to all resource and transformer functions in the OpenHound GitHub extension so that a failure on a single object (org, repo, team, etc.) doesn't abort the entire collection.
summary of changes:
organization.py: Wrapped multi-org resource functions with per-object error handlingenterprise.py: Wrapped multi-org function with per-object error handlingLog format follows the existing
safe_resource_wrapperconventionThe
extrabit in the error log messaging is included because they get emitted as top-level keys in JSON logs. It's only useful for log aggregation in BHE service deployments (they're not visible in CLI mode)Complementary to PR #32 which handles schema/data-type failures. This PR handles API/network failures, but both apply to the same ticket
Validation: Successfully ran a collection to SpecterOps GitHub org