Skip to content

MINOR: Add Protobuf message formatter for kafka-console-consumer - #23260

Open
ssbytedev wants to merge 1 commit into
apache:trunkfrom
ssbytedev:protobuf-message-formatter
Open

MINOR: Add Protobuf message formatter for kafka-console-consumer#23260
ssbytedev wants to merge 1 commit into
apache:trunkfrom
ssbytedev:protobuf-message-formatter

Conversation

@ssbytedev

Copy link
Copy Markdown

Implements KIP-1337 (currently Under Discussion) as a reference
implementation to make the proposal concrete for discussion.

What

Adds ProtobufMessageFormatter for kafka-console-consumer, implementing the existing
org.apache.kafka.common.MessageFormatter extension point.

Why

Allows inspecting Protobuf payloads from the console consumer using local .proto
schema files, without requiring a schema registry.

How

--formatter org.apache.kafka.tools.consumer.ProtobufMessageFormatter
--formatter-property proto.dir=/path/to/protos
--formatter-property message.type=com.example.MyEvent

At configure() time, compiles every *.proto file under proto.dir via protoc
(--descriptor_set_out, --include_imports), resolves the configured message.type
against the resulting descriptors, and caches both. At writeTo() time, decodes
each record with DynamicMessage.parseFrom and prints canonical ProtoJSON via
protobuf-java-util JsonFormat. Requires protoc on PATH.

Testing

  • ProtobufMessageFormatterTest: happy path, missing/invalid config, no .proto files
    found, invalid .proto syntax, message.type not found, malformed record bytes,
    null (tombstone) and empty-value records - compiling real .proto sources via
    protoc, not synthetic descriptors
  • ConsoleConsumerOptionsTest: --formatter/--formatter-property CLI wiring
    without --skip-message-on-error, tombstone)
  • checkstyleMain/checkstyleTest, spotlessCheck, spotbugsMain/spotbugsTest all pass

Open questions from the KIP not yet resolved here

  • Import resolution across multiple proto.dir files/subdirectories is supported
    via protoc --proto_path but not exercised by a dedicated test yet
  • google.protobuf.Any scope is not addressed
  • Tombstone (null value) behavior (currently prints the literal null) is a
    placeholder pending KIP agreement

Adds ProtobufMessageFormatter implementing org.apache.kafka.common.MessageFormatter,
matching KIP-1337: compiles .proto definitions from a user-specified proto.dir via
protoc at configure() time, resolves the configured message.type, and decodes
records with DynamicMessage, printing canonical ProtoJSON.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@github-actions github-actions Bot added triage PRs from the community tools dependencies Pull requests that update a dependency file build Gradle build or GitHub Actions labels Aug 24, 2026
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build Gradle build or GitHub Actions ci-approved dependencies Pull requests that update a dependency file needs-attention tools triage PRs from the community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants