Local Mode: `peer.service` edges missing from `maple service-map`

[koolay]

Local Mode: peer.service edges missing from maple service-map

Summary

In Maple local mode (v0.0.19 and v0.0.20), maple service-map never shows edges derived from peer.service (e.g. my-api → postgresql, my-api → redis). Only cross-service parent-child span edges appear (e.g. frontend → api).

The data is correctly ingested — maple attributes values peer.service shows the values and service_map_db_edges_hourly aggregates them — but the service map query never surfaces them because service_map_edges_hourly is always empty: the materialized view that should populate it from ingested spans does not exist in local mode.

Environment

• maple v0.0.20 (also reproduced on v0.0.19)
• macOS aarch64, installed via brew install Makisuo/tap/maple
• Local mode: maple start --reset -d

Reproduction

# 1. Start fresh
maple stop; maple start --reset -d

# 2. Send spans with peer.service from any OTel-instrumented service, e.g.:
#
#    Go (redisotel):
#      redisotel.InstrumentTracing(rdb,
#          redisotel.WithAttributes(attribute.String("peer.service", "redis")))
#
#    Go (otelhttp client transport):
#      otelhttp.NewTransport(base, otelhttp.WithSpanOptions(
#          trace.WithAttributes(attribute.String("peer.service", "postgresql"))))
#
#    Node.js (manual span):
#      tracer.startSpan("query", {
#          kind: SpanKind.CLIENT,
#          attributes: { "peer.service": "postgresql" }
#      })

# 3. Verify attributes are ingested
maple attributes values peer.service
# → shows: postgresql, redis, etc.

# 4. Check the service map
maple service-map
# → Only shows cross-service parent-child edges (frontend→api)
# → Does NOT show peer.service edges (api→postgresql, api→redis)

# 5. Confirm table is empty
maple query "SELECT count() FROM service_map_edges_hourly"
# → 0

Root Cause

The maple service-map query (visible via maple service-map --debug) uses three data sources via UNION ALL:

1. service_map_edges_hourly — pre-aggregated edges for completed hours
2. Cross-service parent-child join (current hour, first partial window) — WHERE p.ServiceName != c.ServiceName
3. Cross-service parent-child join (current hour, second partial window) — same condition

Source 1 is always empty because no MV writes into it:

maple query "SELECT count() FROM service_map_edges_hourly"
# → 0

maple query "SHOW CREATE TABLE service_map_edges_hourly_mv"
# → Error: There is no metadata of table `service_map_edges_hourly_mv`

The table service_map_edges_hourly_ingest exists (ENGINE = Null) with a corresponding service_map_edges_hourly_ingest_mv that passes through INSERTs to service_map_edges_hourly. This is a push model — the ingest pipeline should INSERT pre-computed edges into service_map_edges_hourly_ingest during span processing — but local mode's ingest pipeline does not perform this step.

Sources 2 and 3 only produce edges where the parent span's ServiceName differs from the child span's ServiceName. This works for cross-service calls (frontend → api) where both sides export their own spans, but cannot produce edges for databases/caches because their spans are emitted by the calling service (e.g. both the HTTP handler span and the query SELECT child span have ServiceName=my-api).

Impact

Any dependency that doesn't run its own OTel SDK (PostgreSQL, Redis, Kafka, external HTTP APIs without instrumentation) is invisible on the service map in local mode, even when peer.service is correctly set on every Client span. This contradicts the OTel Conventions documentation which states:

Emit peer.service on every Client or Producer span. The value must match the service.name of the downstream service. Maple's materialized view groups on (SourceService, TargetService, DeploymentEnv) per hour where peer.service is non-empty.

Workaround

Create the missing MV manually after each maple start --reset:

CREATE MATERIALIZED VIEW IF NOT EXISTS service_map_peer_edges_mv
TO service_map_edges_hourly
AS SELECT
    OrgId,
    toStartOfHour(toDateTime(Timestamp)) AS Hour,
    ServiceName AS SourceService,
    SpanAttributes['peer.service'] AS TargetService,
    ResourceAttributes['deployment.environment.name'] AS DeploymentEnv,
    count() AS CallCount,
    countIf(StatusCode = 'Error') AS ErrorCount,
    sum(Duration / 1000000) AS DurationSumMs,
    max(Duration / 1000000) AS MaxDurationMs,
    countIf(TraceState LIKE '%th:%') AS SampledSpanCount,
    countIf((TraceState = '') OR (TraceState NOT LIKE '%th:%')) AS UnsampledSpanCount,
    sum(if(match(TraceState, 'th:[0-9a-f]+'),
        1.0 / greatest(1.0 - reinterpretAsUInt64(reverse(unhex(rightPad(
            extract(TraceState, 'th:([0-9a-f]+)'), 16, '0')))) / pow(2.0, 64), 0.0001),
        1.0)) AS SampleRateSum
FROM traces
WHERE SpanKind IN ('Client', 'Producer')
    AND SpanAttributes['peer.service'] != ''
    AND ServiceName != SpanAttributes['peer.service']
GROUP BY OrgId, Hour, SourceService, TargetService, DeploymentEnv;

Via maple query (single line):

maple query "CREATE MATERIALIZED VIEW IF NOT EXISTS service_map_peer_edges_mv TO service_map_edges_hourly AS SELECT OrgId, toStartOfHour(toDateTime(Timestamp)) AS Hour, ServiceName AS SourceService, SpanAttributes['peer.service'] AS TargetService, ResourceAttributes['deployment.environment.name'] AS DeploymentEnv, count() AS CallCount, countIf(StatusCode = 'Error') AS ErrorCount, sum(Duration / 1000000) AS DurationSumMs, max(Duration / 1000000) AS MaxDurationMs, countIf(TraceState LIKE '%th:%') AS SampledSpanCount, countIf((TraceState = '') OR (TraceState NOT LIKE '%th:%')) AS UnsampledSpanCount, sum(if(match(TraceState, 'th:[0-9a-f]+'), 1.0 / greatest(1.0 - reinterpretAsUInt64(reverse(unhex(rightPad(extract(TraceState, 'th:([0-9a-f]+)'), 16, '0')))) / pow(2.0, 64), 0.0001), 1.0)) AS SampleRateSum FROM traces WHERE SpanKind IN ('Client', 'Producer') AND SpanAttributes['peer.service'] != '' AND ServiceName != SpanAttributes['peer.service'] GROUP BY OrgId, Hour, SourceService, TargetService, DeploymentEnv"

After creating the MV, new spans with peer.service will populate service_map_edges_hourly. Verification:

maple query "SELECT SourceService, TargetService, sum(CallCount) as calls FROM service_map_edges_hourly GROUP BY SourceService, TargetService ORDER BY calls DESC" --format table

Expected output after some traffic:

SourceService  TargetService  calls
─────────────  ─────────────  ─────
my-api         postgresql     49
my-api         redis          29
my-agent       my-api         4
my-frontend    my-api         3

Note: maple service-map still won't show the current (incomplete) hour's peer edges because its query uses Hour < toStartOfHour(now()). They appear after the hour rolls over. To verify immediately, query the table directly as shown above.

Secondary Issue: maple service-map excludes current-hour aggregated edges

Even after the MV workaround, maple service-map only shows service_map_edges_hourly rows where Hour < toStartOfHour(now()) — meaning the current hour's peer edges are always excluded from the result until the next hour boundary. The real-time fallback (sources 2 and 3 in the UNION ALL) only covers cross-service parent-child, not peer-based edges.

This means a user who just set up peer.service has to wait up to 60 minutes before seeing their database/cache nodes on the map, even though the data is already in the hourly table.

Suggested Fix

Option A: Add the MV to local mode's schema bootstrap

During maple start schema initialization, create a materialized view equivalent to service_map_peer_edges_mv (as shown in the workaround above) so that peer.service-based edges are aggregated in real time, matching the hosted Maple behavior.

Option B: Add a real-time peer.service fallback in the service-map query

In addition to (or instead of) the MV, add a fourth UNION ALL branch to the maple service-map query that directly queries current-hour spans with peer.service:

UNION ALL
SELECT
    ServiceName AS sourceService,
    SpanAttributes['peer.service'] AS targetService,
    count() AS bucketCallCount,
    countIf(StatusCode = 'Error') AS bucketErrorCount,
    sum(Duration / 1000000) AS bucketDurationSumMs,
    max(Duration / 1000000) AS bucketMaxDurationMs,
    sum(...) AS bucketEstimatedSpanCount
FROM traces
WHERE SpanKind IN ('Client', 'Producer')
    AND SpanAttributes['peer.service'] != ''
    AND ServiceName != SpanAttributes['peer.service']
    AND Timestamp >= toStartOfHour(now())
    AND Timestamp < now()
    AND OrgId = 'local'
GROUP BY sourceService, targetService

This ensures peer-based edges (databases, caches, external APIs) are visible immediately without waiting for the hour to roll over.

Related

• OTel Conventions: Service Map — documents peer.service as the mechanism for service-map edges
• service_map_db_edges_hourly — correctly populated via its own MV in local mode; shows DB edges are tracked internally but not surfaced in maple service-map
• Hosted Maple likely handles this in the ingest pipeline (writes to service_map_edges_hourly_ingest), which local mode does not replicate