Generate a protobuf schema artifact from ElasticGraph schemas

[jwils]

Note

First of a four-PR stack. Supporting changes merged earlier (#1296 artifact comment prefixes, #1297 gem scaffold), so this diff is contained entirely to the elasticgraph-proto_ingestion directory. Per review feedback the original PR has been split: wire-stable field/enum-value numbering, proto2 + custom headers, and enum sourcing/external references are now stacked follow-ups.

Why

• continue the pluggable ingestion serializer proposal after pulling JSON Schema into its own gem
• revive the earlier protobuf prototype from Add elasticgraph-proto gem with stable proto field-number mappings #1056 on top of the new serializer extension points

What

• fill in the elasticgraph-proto_ingestion extension gem with core generation: schema_artifacts:dump emits a proto3 schema.proto covering the schema's indexed types
• map built-in ElasticGraph scalars to proto types, with t.protobuf type: for custom scalars (resolved via type_ref.with_reverted_override so built-ins renamed with type_name_overrides keep working)
• generate messages for object/interface/union types and enums (with a zero-valued *_UNSPECIFIED entry), escaping proto reserved words and wrapping lists of lists so the output stays valid
• keep schema.proto on public GraphQL field names; validate proto package names
• hold extension state on a ProtoIngestionState container behind a single proto_ingestion_state reader (matching Group JSON ingestion schema state #1281)

Field and enum value numbers are assigned sequentially in definition order in this PR; the stacked follow-up adds the proto_field_numbers.yaml sidecar that keeps them wire-stable across schema evolution.

Stacked follow-ups

1. this PR — core schema.proto generation
2. wire-stable field/enum value numbers via a proto_field_numbers.yaml sidecar
3. syntax: :proto2 support and custom file-level headers:
4. Source enum values from external proto enums and reference external enum types #1286 — enum value sourcing from existing proto enums + external proto type references

Verification

• script/run_gem_specs elasticgraph-proto_ingestion (100% line + branch coverage at this commit)
• script/type_check, script/lint, script/spellcheck
• script/quick_build green at the stack head (whose tree is identical to the previously reviewed single-PR revision)

References

• Feature Proposal: Support for Protobuf ingestion #1059
• Add elasticgraph-proto gem with stable proto field-number mappings #1056
• Pluggable ingestion serializers with json_ingestion #1079

Update — 2026-07-10

• The current stack is Generate a protobuf schema artifact from ElasticGraph schemas #1080 → Pin protobuf field and enum value numbers in a sidecar artifact #1304 → Map DateTime to google.protobuf.Timestamp and document temporal string formats #1306 → Support proto2 syntax and custom proto file headers #1305 → Source enum values from external proto enums and reference external enum types #1286.
• Proto extension state now uses a mutable Struct, and keyword package-name segments are validated without being rewritten.
• Lists of lists now raise an actionable schema error instead of generating wrapper messages; this supersedes the earlier wrapping note above.

---

Update — 2026-07-15

• Interface and union messages now wrap concrete subtype messages in a oneof, matching the JSON Schema oneOf representation.
• Concrete subtype messages omit the redundant __typename discriminator.
• Proto type rendering is now stateless: the generator selects the reachable type graph up front, then each extended type renders itself without mutating shared traversal state.
• No-block extension coverage now completes the definitions so the fixture remains valid under CI GraphQL-schema validation.

---

Update — 2026-07-19

• Replaced keyword suffixing with fully qualified local message and enum references, preserving source type and field names while disambiguating contextual protobuf words and built-in scalar names.
• Removed the now-unnecessary keyword collision tracking and verified a generated schema containing contextual names with protoc 35.1.

[myronmarston]

Not done reviewing but wanted to submit what I have so far.

[myronmarston]

Still not done with my review but want to submit what I have so far.

One thing I haven't seen is something which verifies that the proto schema we produce is a valid proto schema--e.g. a test which parses a proto.schema file we generate using the google-protobuf gem, demonstrating that we generate valid protos. (For now, it just looks like we do but I can't say for sure that it's valid!).

One idea--maybe in a followup PR you can wire this up with the test schema (similar to the warehouse artifact) and then you can add a test to the test suite that verifies that the generated proto schema from our main test schema is valid? I'm open to other ideas here...

[myronmarston]

Finally made it through the entire PR! I'm hoping this is the last round of feedback.

[myronmarston]

One other thing I just noticed: the tests generate a few warnings which dirty up the output. Can we change the tests that generate these warnings (so they no longer generate them) without impacting what they cover?

$ be rspec elasticgraph-proto_ingestion/spec

Randomized with seed 37739
.........WARNING: Since a `Account.count` field exists, ElasticGraph is not able to
define its typical `AccountFieldsListFilterInput.count` field, which allows clients to filter on the count
of values for a `[Account]` field. Clients will still be able to filter on the `count`
at a leaf field path. However, there are a couple ways this naming conflict can be avoided if desired:

1. Pick a different name for the `Account.count` field.
2. Change the name used by ElasticGraph for this field. To do that, pass a
   `schema_element_name_overrides: {"count" => "alt_name"}` option alongside
   `schema_element_name_form: ...` when defining `ElasticGraph::SchemaDefinition::RakeTasks`
   (typically in the `Rakefile`).
...............Object types must have fields, but Query doesn't have any. Define a field for this type, remove it from your schema, or add `has_no_fields(true)` to its definition.

This will raise an error in a future GraphQL-Ruby version.
.......

[myronmarston]

Well done! Using the dot-prefixed FQN form to avoid naming collisions is a really nice approach. I'm really happy with how we landed on this.

Left some non-blocking feedback. Use your judgement about which things to address and feel free to address them in a followup PR. (Frankly, I'd prefer that as it would be a smaller diff to go through!).