Skip to content

fix: pass a payload that is not JSON to a custom mapper unchanged #387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hamzahalq merged 1 commit into from
Oct 20, 2026
Merged

fix: pass a payload that is not JSON to a custom mapper unchanged #387

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions SW.Bitween.Api/Services/MapperEnrichment.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ namespace SW.Bitween;
public static class MapperEnrichment
{
/// <summary>
/// The payload as the enrichment reads it: <c>JToken.Parse</c>, which throws on what is not JSON,
/// and null for JSON that is not an object (a root array, a string), which is not enriched.
/// The payload as the enrichment reads it: null for what is not a JSON object, which is not
/// enriched - other JSON (a root array, a string), and what is not JSON at all.
/// </summary>
/// <remarks>
/// <paramref name="shared"/> says the object is the pipeline's shared parse, which
Expand All @@ -35,7 +35,16 @@ public static JObject Read(XchangeFile file, out bool shared)

shared = false;
NativeAdapters.PayloadParseTally.Parsed();
return JToken.Parse(file.Data) as JObject;
try
{
return JToken.Parse(file.Data) as JObject;
}
catch (JsonReaderException)
{
// XML or CSV from a receiver, read by a custom mapper. Failing it here failed every such
// exchange before its mapper ran, for values it had no use for.
return null;
}
}

/// <summary>
Expand Down
8 changes: 3 additions & 5 deletions SW.Bitween.Api/Services/XchangeService.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -421,9 +421,7 @@ private async Task<XchangeFile> RunMapper(Xchange xchange, XchangeFile xchangeFi
// A mapper that takes its context separately gets the payload exactly as it arrived. Anything
// else keeps the enrichment below, unchanged - every template already written reads
// __partner__ out of the payload, so this is load-bearing behaviour, not an implementation
// detail. Note the JToken.Parse: it runs before anything knows which mapper is configured,
// and it throws on a payload that is not JSON, which is why a mapper that wants to handle
// XML or CSV has to be able to opt out of this entire block.
// detail. A payload that is not JSON goes to the mapper as it arrived.
string mappingContextJson = null;

if (nativeAdapterDiscovery.MapperReceivesOwnContext(xchange.MapperId))
Expand All @@ -434,8 +432,8 @@ private async Task<XchangeFile> RunMapper(Xchange xchange, XchangeFile xchangeFi
{
// Inject __partner__ adapter properties into the input JSON so Scriban templates
// can reference them as {{ __partner__?.propkey }}
// Only applies when the data is a JSON object; skip enrichment for non-object payloads
// (e.g. a receiver returning a JSON-encoded string).
// Only applies when the data is a JSON object; skip enrichment for anything else
// (e.g. a receiver returning a JSON-encoded string, or XML).
var jObjEnriched = MapperEnrichment.Read(xchangeFile, out var shared);

if (jObjEnriched != null)
Expand Down
2 changes: 1 addition & 1 deletion SW.Bitween.UnitTests/PayloadParsing/Golden/expected/json-bom.json.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Not valid JSON: Unexpected character encountered while parsing value: . Path
=== schema-xml
Not valid XML: Data at the root level is invalid. Line 1, position 1.
=== enriched
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: . Path '', line 0, position 0.
(not an object, not enriched)
=== enriched-scriban
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: . Path '', line 0, position 0.
=== scriban
Expand Down
2 changes: 1 addition & 1 deletion SW.Bitween.UnitTests/PayloadParsing/Golden/expected/json-deep.json.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Not valid JSON: The reader's MaxDepth of 64 has been exceeded. Path 'd.d.d.d.d.d
=== schema-xml
Not valid XML: Data at the root level is invalid. Line 1, position 1.
=== enriched
!Newtonsoft.Json.JsonReaderException: The reader's MaxDepth of 64 has been exceeded. Path 'd.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d', line 1, position 333.
(not an object, not enriched)
=== enriched-scriban
!Newtonsoft.Json.JsonReaderException: The reader's MaxDepth of 64 has been exceeded. Path 'd.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d', line 1, position 333.
=== scriban
Expand Down
2 changes: 1 addition & 1 deletion SW.Bitween.UnitTests/PayloadParsing/Golden/expected/json-empty.json.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Not valid JSON: Error reading JToken from JsonReader. Path '', line 0, position
=== schema-xml
Not valid XML: Root element is missing.
=== enriched
!Newtonsoft.Json.JsonReaderException: Error reading JToken from JsonReader. Path '', line 0, position 0.
(not an object, not enriched)
=== enriched-scriban
!Newtonsoft.Json.JsonReaderException: Error reading JToken from JsonReader. Path '', line 0, position 0.
=== scriban
Expand Down
2 changes: 1 addition & 1 deletion SW.Bitween.UnitTests/PayloadParsing/Golden/expected/json-invalid.json.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Not valid JSON: Unexpected end of content while loading JObject. Path 'id', line
=== schema-xml
Not valid XML: Data at the root level is invalid. Line 1, position 1.
=== enriched
!Newtonsoft.Json.JsonReaderException: Unexpected end of content while loading JObject. Path 'id', line 1, position 7.
(not an object, not enriched)
=== enriched-scriban
!Newtonsoft.Json.JsonReaderException: Unexpected end of content while loading JObject. Path 'id', line 1, position 7.
=== scriban
Expand Down
2 changes: 1 addition & 1 deletion SW.Bitween.UnitTests/PayloadParsing/Golden/expected/json-trailing-content.json.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Not valid JSON: Additional text encountered after finished reading JSON content:
=== schema-xml
Not valid XML: Data at the root level is invalid. Line 1, position 1.
=== enriched
!Newtonsoft.Json.JsonReaderException: Additional text encountered after finished reading JSON content: {. Path '', line 1, position 13.
(not an object, not enriched)
=== enriched-scriban
!Newtonsoft.Json.JsonReaderException: Additional text encountered after finished reading JSON content: {. Path '', line 1, position 13.
=== scriban
Expand Down
2 changes: 1 addition & 1 deletion SW.Bitween.UnitTests/PayloadParsing/Golden/expected/json-trailing-garbage.json.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Not valid JSON: Additional text encountered after finished reading JSON content:
=== schema-xml
Not valid XML: Data at the root level is invalid. Line 1, position 1.
=== enriched
!Newtonsoft.Json.JsonReaderException: Additional text encountered after finished reading JSON content: x. Path '', line 1, position 12.
(not an object, not enriched)
=== enriched-scriban
!Newtonsoft.Json.JsonReaderException: Additional text encountered after finished reading JSON content: x. Path '', line 1, position 12.
=== scriban
Expand Down
2 changes: 1 addition & 1 deletion SW.Bitween.UnitTests/PayloadParsing/Golden/expected/json-whitespace-only.json.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Not valid JSON: Error reading JToken from JsonReader. Path '', line 2, position
=== schema-xml
Not valid XML: Root element is missing.
=== enriched
!Newtonsoft.Json.JsonReaderException: Error reading JToken from JsonReader. Path '', line 2, position 1.
(not an object, not enriched)
=== enriched-scriban
!Newtonsoft.Json.JsonReaderException: Error reading JToken from JsonReader. Path '', line 2, position 1.
=== scriban
Expand Down
2 changes: 1 addition & 1 deletion SW.Bitween.UnitTests/PayloadParsing/Golden/expected/text-csv.txt.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Not valid JSON: Unexpected character encountered while parsing value: i. Path ''
=== schema-xml
Not valid XML: Data at the root level is invalid. Line 1, position 1.
=== enriched
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: i. Path '', line 0, position 0.
(not an object, not enriched)
=== enriched-scriban
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: i. Path '', line 0, position 0.
=== scriban
Expand Down
2 changes: 1 addition & 1 deletion SW.Bitween.UnitTests/PayloadParsing/Golden/expected/xml-bom.xml.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Not valid JSON: Unexpected character encountered while parsing value: . Path
=== schema-xml
Not valid XML: Data at the root level is invalid. Line 1, position 1.
=== enriched
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: . Path '', line 0, position 0.
(not an object, not enriched)
=== enriched-scriban
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: . Path '', line 0, position 0.
=== scriban
Expand Down
2 changes: 1 addition & 1 deletion SW.Bitween.UnitTests/PayloadParsing/Golden/expected/xml-cdata.xml.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Not valid JSON: Unexpected character encountered while parsing value: <. Path ''
=== schema-xml
(valid)
=== enriched
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
(not an object, not enriched)
=== enriched-scriban
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
=== scriban
Expand Down
2 changes: 1 addition & 1 deletion SW.Bitween.UnitTests/PayloadParsing/Golden/expected/xml-dtd.xml.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Not valid JSON: Unexpected character encountered while parsing value: <. Path ''
=== schema-xml
Not valid XML: For security reasons DTD is prohibited in this XML document. To enable DTD processing set the DtdProcessing property on XmlReaderSettings to Parse and pass the settings into XmlReader.Create method.
=== enriched
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
(not an object, not enriched)
=== enriched-scriban
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
=== scriban
Expand Down
2 changes: 1 addition & 1 deletion SW.Bitween.UnitTests/PayloadParsing/Golden/expected/xml-entities.xml.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Not valid JSON: Unexpected character encountered while parsing value: <. Path ''
line 1, position 95: The element 'order' has invalid child element 'raw'. List of possible elements expected: 'lines'.
Not valid XML: An error occurred while parsing EntityName. Line 1, position 104.
=== enriched
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
(not an object, not enriched)
=== enriched-scriban
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
=== scriban
Expand Down
2 changes: 1 addition & 1 deletion SW.Bitween.UnitTests/PayloadParsing/Golden/expected/xml-html-encoded.xml.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Not valid JSON: Unexpected character encountered while parsing value: &. Path ''
=== schema-xml
Not valid XML: Data at the root level is invalid. Line 1, position 1.
=== enriched
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: &. Path '', line 0, position 0.
(not an object, not enriched)
=== enriched-scriban
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: &. Path '', line 0, position 0.
=== scriban
Expand Down
2 changes: 1 addition & 1 deletion SW.Bitween.UnitTests/PayloadParsing/Golden/expected/xml-invalid-chars.xml.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Not valid JSON: Unexpected character encountered while parsing value: <. Path ''
=== schema-xml
Not valid XML: '', hexadecimal value 0x07, is an invalid character. Line 1, position 30.
=== enriched
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
(not an object, not enriched)
=== enriched-scriban
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
=== scriban
Expand Down
2 changes: 1 addition & 1 deletion SW.Bitween.UnitTests/PayloadParsing/Golden/expected/xml-invalid.xml.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Not valid JSON: Unexpected character encountered while parsing value: <. Path ''
=== schema-xml
Not valid XML: The 'id' start tag on line 1 position 9 does not match the end tag of 'order'. Line 1, position 17.
=== enriched
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
(not an object, not enriched)
=== enriched-scriban
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
=== scriban
Expand Down
2 changes: 1 addition & 1 deletion SW.Bitween.UnitTests/PayloadParsing/Golden/expected/xml-namespaces.xml.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Not valid JSON: Unexpected character encountered while parsing value: <. Path ''
=== schema-xml
(valid)
=== enriched
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
(not an object, not enriched)
=== enriched-scriban
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
=== scriban
Expand Down
2 changes: 1 addition & 1 deletion SW.Bitween.UnitTests/PayloadParsing/Golden/expected/xml-order.xml.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Not valid JSON: Unexpected character encountered while parsing value: <. Path ''
=== schema-xml
(valid)
=== enriched
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
(not an object, not enriched)
=== enriched-scriban
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
=== scriban
Expand Down
2 changes: 1 addition & 1 deletion SW.Bitween.UnitTests/PayloadParsing/Golden/expected/xml-plain.xml.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Not valid JSON: Unexpected character encountered while parsing value: <. Path ''
=== schema-xml
(valid)
=== enriched
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
(not an object, not enriched)
=== enriched-scriban
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
=== scriban
Expand Down
2 changes: 1 addition & 1 deletion SW.Bitween.UnitTests/PayloadParsing/Golden/expected/xml-trailing-content.xml.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Not valid JSON: Unexpected character encountered while parsing value: <. Path ''
=== schema-xml
Not valid XML: There are multiple root elements. Line 1, position 30.
=== enriched
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
(not an object, not enriched)
=== enriched-scriban
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
=== scriban
Expand Down
2 changes: 1 addition & 1 deletion SW.Bitween.UnitTests/PayloadParsing/Golden/expected/xml-utf16-declaration.xml.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Not valid JSON: Unexpected character encountered while parsing value: <. Path ''
=== schema-xml
(valid)
=== enriched
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
(not an object, not enriched)
=== enriched-scriban
!Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
=== scriban
Expand Down
2 changes: 1 addition & 1 deletion docs/caveats.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ These behaviours were found while documenting Bitween from its source code, and
| Validators only run for API gateways and the legacy API call endpoint. | `Services/XchangeService.cs` |
| Exchanges created by fan-out do not carry the parent's references. | `XchangeService.CreateXchangesForHits` |
| A retry without reset keeps the response subscription but drops the response message type name. | `Domain/Xchange/Xchange.cs` |
| Mappers other than the rules-based mapper fail on payloads that are not JSON, and overwrite payload keys named `__partner__` or `__globals__`. | `XchangeService.RunMapper` |
| Mappers other than the rules-based mapper overwrite payload keys named `__partner__` or `__globals__`. | `XchangeService.RunMapper` |
| An API gateway attached to an inactive subscription returns 404. | `Controllers/GatewayController.cs` |
| Bulk retry includes exchanges that have no result yet, so work still in flight can run twice. Deliberate: after a broker outage those are the exchanges a bulk retry is for. | `Resources/Xchanges/BulkRetryPlanner.cs` |
| The dashboard's *Failures to act on* tile says it covers 14 days, but counts failures of any age. | `ClientApp/src/pages/dashboard/DashboardPage.tsx` |
Expand Down
2 changes: 1 addition & 1 deletion docs/exchange-pipeline.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ When the exchange has a mapper, the mapper turns the input into the output file.
How the mapper sees partner and global values depends on the mapper.

- The **rules-based mapper**, `NativeMapper`, receives partner and global values as separate context. The payload reaches it untouched, so it can map XML as well as JSON.
- **Every other mapper**, including the legacy JSON mapper and custom mappers, gets them written into the payload as `__partner__` and `__globals__` keys, and only when the payload is a JSON object. A payload that is not JSON fails the exchange before the mapper runs. A real payload key with either name is overwritten.
- **Every other mapper**, including the legacy JSON mapper and custom mappers, gets them written into the payload as `__partner__` and `__globals__` keys, and only when the payload is a JSON object. Any other payload, such as XML or CSV, reaches the mapper as it arrived. A real payload key with either name is overwritten.

A mapper that returns nothing fails the exchange. See [Mapping](mapping.md).

Expand Down
Loading