feat(mapper): add entity-proto mappers by yushan8 · Pull Request #211 · uber/tango

yushan8 · 2026-07-17T03:36:57Z

Summary

Adds entity-proto mappers for target graph and changed targets responses
ResultToTargetGraph — converts targethasher.Result into ID-mapped entity types
GetTargetGraphResponseToProto / ChangedTargetsResponseToProto — entity→proto conversions for gRPC streaming
ProtoToGetTargetGraphRequest — proto→entity request conversion

This PR is addresses this comment to split up the Result -> Optimized target entity conversion, and OptimizedTarget -> GetTargetGraphResponseToProto #206. These mappers are used to avoid leaking proto definitions into internals such as storage and orchestrator.

Stack

feat(mapper): add entity-proto mappers #224 (streaming split utilities)
This PR (entity-proto mappers)
refactor: replace proto serialization with JSON and migrate callsites #222 (storage JSON rewrite)
refactor: migrate callsites to entity types and consolidate config #212 (callsite migration)

Test plan

make build && make test && make gazelle

…210) ## Summary - Add `entity.OptimizedTarget`, `entity.Metadata`, and `entity.GetTargetGraphResponse` as compact, proto-free representations of the target graph wire types - Add `entity.ChangedTarget` and `entity.GetChangedTargetsResponse` for the changed-targets path - These entity types mirror the proto definitions but live in a dependency-free package, preventing proto from leaking into storage, orchestrator, and other non-transport layers ## Test plan - [x] `make build && make test && make gazelle` ## Stack 1. **#210 — this PR** (entity types) 2. #211 (mappers + streaming) 3. #222 (storage JSON rewrite) 4. #212 (callsite migration + cleanup) --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…eaming Move SplitTargetGraph from internal/mapper to internal/streaming where the other split functions live. Rename chunk→split for consistency. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Remove protoToOptimizedTarget, protoToMetadata, and round-trip tests that converted back from proto. Replace with direct proto field assertions. Use t.Context() instead of context.Background(). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…kage Move ResultToTargetGraph and idmapper out of internal/ so external consumers (e.g. the internal monorepo compute package) can import them. Proto↔entity conversion helpers remain in internal/mapper. Also fixes trailing newline lint issue in target_graph.go. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

## Summary - Adds entity-proto mappers for target graph and changed targets responses - `ResultToTargetGraph` — converts `targethasher.Result` into ID-mapped entity types - `GetTargetGraphResponseToProto` / `ChangedTargetsResponseToProto` — entity→proto conversions for gRPC streaming - `ProtoToGetTargetGraphRequest` — proto→entity request conversion This PR is addresses this comment to split up the Result -> Optimized target entity conversion, and OptimizedTarget -> `GetTargetGraphResponseToProto` #206. These mappers are used to avoid leaking proto definitions into internals such as storage and orchestrator. ## Stack 1. #224 (streaming split utilities) 2. **This PR** (entity-proto mappers) 3. #222 (storage JSON rewrite) 4. #212 (callsite migration) ## Test plan - [x] `make build && make test && make gazelle` --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

…#222) ## Summary Intent: Proto definitions are currently leaking into the storage. We should `Put` the actual entities to be encoded/decoded from the storage, not the proto definitions. - Remove all proto imports from `core/storage/` — serialize entity structs directly via `encoding/json` - Move write functions from reader files into `graphwriter.go` - Controller reads `entity.GetTargetGraphResponse` and `entity.GetChangedTargetsResponse` internally, converting to proto via mapper only at `stream.Send` - Changed-targets pipeline fully migrated to entity types - Orchestrator uses `mapper.ResultToTargetGraph` + `SplitBySize` - Consolidate `service.chunking.*` into a single `max_message_bytes` config field - Remove old chunking code from `core/common` - Update `query-bench` example ## Test plan - [x] `grep -r "tangopb\|gogo/protobuf" core/storage/` returns zero results - [x] `make build && make test` — all 19 test targets pass - [x] `make gazelle` — BUILD files in sync ## Stack 1. #210 (entity types) 2. #211 (mappers + streaming) 3. **#222 — this PR** (storage JSON + callsite migration + cleanup) Ran `integration tests` and `make bench` to make sure storage is compatible with current changes. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

yushan8 requested review from a team as code owners July 17, 2026 03:36

yushan8 mentioned this pull request Jul 17, 2026

refactor: migrate callsites to entity types and consolidate config #212

Closed

3 tasks

yushan8 marked this pull request as draft July 17, 2026 04:11

yushan8 mentioned this pull request Jul 17, 2026

feat(mapper): add wire-size-based chunking and ResultToGetTargetGraphResponse #206

Closed

3 tasks

yushan8 force-pushed the yushan/streaming-and-mappers branch from 9bd4bdf to d05117c Compare July 17, 2026 05:13

yushan8 force-pushed the yushan/entity-target-graph branch from 4cb3028 to fba8c52 Compare July 17, 2026 05:13

yushan8 force-pushed the yushan/streaming-and-mappers branch 3 times, most recently from ecb2996 to 36f3f33 Compare July 17, 2026 22:57

yushan8 force-pushed the yushan/entity-target-graph branch from fba8c52 to 34e6a59 Compare July 17, 2026 22:57

yushan8 force-pushed the yushan/streaming-and-mappers branch 4 times, most recently from b621c93 to 03833f5 Compare July 20, 2026 02:03

yushan8 force-pushed the yushan/entity-target-graph branch 2 times, most recently from ac6aea4 to facc62d Compare July 20, 2026 03:20

yushan8 force-pushed the yushan/streaming-and-mappers branch 2 times, most recently from 8ba38ac to 6fc3266 Compare July 20, 2026 03:28

yushan8 force-pushed the yushan/entity-target-graph branch from facc62d to ef4db65 Compare July 20, 2026 03:28

yushan8 mentioned this pull request Jul 20, 2026

feat(entity): add entity types for target graph and changed targets #210

Merged

1 task

yushan8 changed the title ~~feat(mapper): add streaming split, graph chunk mappers, and graphstore adapter~~ feat(mapper): add entity-proto mappers and streaming splitters Jul 20, 2026

yushan8 mentioned this pull request Jul 20, 2026

refactor: replace proto serialization with JSON and migrate callsites #222

Merged

3 tasks

yushan8 force-pushed the yushan/entity-target-graph branch from ef4db65 to 1b416d5 Compare July 20, 2026 03:42

yushan8 force-pushed the yushan/streaming-and-mappers branch 2 times, most recently from 0f5a89c to 5fb83ae Compare July 20, 2026 03:46

yushan8 force-pushed the yushan/entity-target-graph branch from 1b416d5 to ac18fbe Compare July 20, 2026 03:46

yushan8 marked this pull request as ready for review July 20, 2026 04:43

yushan8 force-pushed the yushan/streaming-and-mappers branch from 5fb83ae to 7f55e85 Compare July 20, 2026 04:52

yushan8 force-pushed the yushan/entity-target-graph branch from ac18fbe to 9789ce1 Compare July 20, 2026 04:52

yushan8 force-pushed the yushan/streaming-and-mappers branch from 7f55e85 to e762db8 Compare July 20, 2026 05:27

yushan8 force-pushed the yushan/entity-target-graph branch from 9789ce1 to 061115e Compare July 20, 2026 05:27

yushan8 force-pushed the yushan/streaming-and-mappers branch from e762db8 to dff47a4 Compare July 20, 2026 16:52

yushan8 force-pushed the yushan/entity-target-graph branch from 6b968d3 to 2da7020 Compare July 20, 2026 17:16

yushan8 force-pushed the yushan/streaming-and-mappers branch from dff47a4 to 9c060bc Compare July 20, 2026 17:17

Base automatically changed from yushan/entity-target-graph to main July 20, 2026 18:03

yushan8 mentioned this pull request Jul 20, 2026

feat(mapper): add entity-proto mappers #224

Merged

1 task

yushan8 force-pushed the yushan/streaming-and-mappers branch from 9c060bc to 63d2138 Compare July 20, 2026 18:21

yushan8 changed the base branch from main to yushan/streaming-split July 20, 2026 18:21

yushan8 force-pushed the yushan/streaming-and-mappers branch from 63d2138 to 7b87081 Compare July 20, 2026 18:41

yushan8 force-pushed the yushan/streaming-split branch from 2f87a76 to eacdca8 Compare July 20, 2026 19:31

yushan8 force-pushed the yushan/streaming-and-mappers branch from 7b87081 to 23bf3ac Compare July 20, 2026 19:32

yushan8 changed the title ~~feat(mapper): add entity-proto mappers and streaming splitters~~ feat(mapper): add entity-proto mappers Jul 20, 2026

yushan8 force-pushed the yushan/streaming-and-mappers branch 2 times, most recently from 8f360d1 to 15fe2f7 Compare July 20, 2026 22:20

yushan8 and others added 4 commits July 20, 2026 15:32

feat(mapper): add entity-proto mappers and streaming splitters

502fa2f

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

refactor: rename ChunkTargetGraph to SplitTargetGraph and move to str…

a832fa1

…eaming Move SplitTargetGraph from internal/mapper to internal/streaming where the other split functions live. Rename chunk→split for consistency. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

yushan8 force-pushed the yushan/streaming-and-mappers branch from 15fe2f7 to 814e971 Compare July 20, 2026 22:32

xytan0056 approved these changes Jul 20, 2026

View reviewed changes

yushan8 merged commit 60f3ec2 into yushan/streaming-split Jul 20, 2026
10 checks passed

yushan8 deleted the yushan/streaming-and-mappers branch July 20, 2026 23:36

yushan8 restored the yushan/streaming-and-mappers branch July 21, 2026 04:36

yushan8 deleted the yushan/streaming-and-mappers branch July 22, 2026 00:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(mapper): add entity-proto mappers#211

feat(mapper): add entity-proto mappers#211
yushan8 merged 4 commits into
yushan/streaming-splitfrom
yushan/streaming-and-mappers

yushan8 commented Jul 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

yushan8 commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Stack

Test plan

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yushan8 commented Jul 17, 2026 •

edited

Loading