DSM: pathway timestamp corruption between dd-trace-rb (plain LEB128) and dd-trace-go (zigzag varint)

[olivia-gusto]

Summary

Data Streams Monitoring reports wildly incorrect (years-to-decades off) latency for any pathway where a Ruby producer instrumented with datadog (dd-trace-rb) feeds a Go consumer instrumented with dd-trace-go v2. Root cause: the two SDKs disagree on the wire format for the two pathway timestamps (pathway_start, edge_start) packed into the dd-pathway-ctx-base64 header. dd-trace-rb encodes them as plain unsigned LEB128; dd-trace-go decodes them as zigzag-encoded signed varint. The decode does not raise an error — it "succeeds" while silently producing a garbage timestamp offset by years to decades from the true value.

Environment

• Ruby producer: datadog gem 2.34.0, via the WaterDrop/Karafka integration (produce-side checkpoint, c.tracing.instrument :waterdrop).
• Go consumer: dd-trace-go/v2 v2.9.0, contrib/twmb/franz-go/v2 v2.9.0 (ddkgo.WithTracing(ddkgo.WithDataStreams(), ...)).
• Kafka client on the consumer side: franz-go (twmb/kgo).
• Observed symptom: a single-hop DSM pathway showing average latency in the tens of billions of seconds (effectively "now minus roughly the Unix epoch"), affecting 100% of messages on that pathway, with no exceptions logged on either producer or consumer side.

Root cause (verified by direct source inspection on both SDKs)

Ruby side — dd-trace-rb 2.34.0, lib/datadog/data_streams/pathway_context.rb, encode_var_int_64 / self.decode_varint (~lines 101-156): implements plain unsigned LEB128 per the DWARF5 spec (the docstring cites DWARF5 section 7.6 directly). No zigzag transform anywhere.

Go side — dd-trace-go/v2@v2.9.0, internal/datastreams/propagator.go:

// Encode encodes the pathway
func (p Pathway) Encode() []byte {
	data := make([]byte, 8, 20)
	binary.LittleEndian.PutUint64(data, p.hash)
	encoding.EncodeVarint64(&data, p.pathwayStart.UnixNano()/int64(time.Millisecond))
	encoding.EncodeVarint64(&data, p.edgeStart.UnixNano()/int64(time.Millisecond))
	return data
}

func Decode(ctx context.Context, data []byte) (p Pathway, outCtx context.Context, err error) {
	...
	pathwayStart, err := encoding.DecodeVarint64(&data)
	...
	p.pathwayStart = time.Unix(0, pathwayStart*int64(time.Millisecond))
	...
}

encoding.EncodeVarint64 / DecodeVarint64 come from github.com/DataDog/sketches-go@v1.4.8/ddsketch/encoding/encoding.go:87-96:

func EncodeVarint64(b *[]byte, v int64) {
	EncodeUvarint64(b, uint64(v>>(64-1)^(v<<1))) // zigzag map
}
func DecodeVarint64(b *[]byte) (int64, error) {
	v, err := DecodeUvarint64(b)
	return int64((v >> 1) ^ -(v & 1)), err // zigzag un-map
}

This is a signed, zigzag-encoded varint — reused from sketches-go's ddsketch bin-index encoder, which needs to represent negative bin indices. Pathway timestamps are never negative, so it looks like this encoding was picked for convenience (an existing helper already on the import graph via ddsketch) without checking cross-SDK wire compatibility against dd-trace-rb's plain-unsigned implementation.

Go-to-Go DSM pathways are unaffected, since both the Go producer and Go consumer apply the same zigzag transform consistently. But any Ruby-producer → Go-consumer pathway is broken: the Go decoder applies a zigzag un-map to a value that was never zigzag-mapped on the Ruby side, producing a timestamp off by years to decades depending on the parity of the raw millisecond value. This is silent numeric corruption — no error, no missing-header fallback — so it can be very hard to notice in code review; it typically only surfaces as anomalous DSM dashboard data.

Suggested fix

Either:

• Change dd-trace-go's Encode/Decode in internal/datastreams/propagator.go to use EncodeUvarint64/DecodeUvarint64 (plain unsigned) for the two timestamp fields, matching dd-trace-rb's implementation, or
• Change dd-trace-rb's pathway_context.rb to zigzag-encode, matching dd-trace-go.

Flagging both sides since either SDK could be the "correct" one per whatever internal wire-format spec exists for DSM pathway context — your team is better placed to pick the canonical fix and coordinate a compatibility-preserving rollout across all DSM-instrumented SDKs. This likely affects any customer running a Ruby producer feeding a Go consumer (or vice versa) with DSM enabled.

Reproduction

Happy to share a minimal repro if useful — a standalone WaterDrop::Producer + real Datadog::DataStreams::Processor (Ruby) producing to a real local Kafka broker, decoded on the Go side with dd-trace-go's own Decode(), deterministically shows the corrupted timestamp for any produce call.

[olivia-gusto]

Follow-up: checked every other Datadog language tracer that implements DSM, and this isn't a "which side is canonical" question — Ruby is the sole outlier; every other SDK already agrees on ZigZag:

SDK	Scheme	Source
Go (dd-trace-go)	ZigZag	sketches-go's EncodeVarint64/DecodeVarint64 (as above)
Python (dd-trace-py)	ZigZag	ddtrace/internal/datastreams/encoding.py: encode_var_int_64 does v >> 63 ^ (v << 1)
JavaScript (dd-trace-js)	ZigZag	packages/dd-trace/src/datastreams/encoding.js, explicitly documented as zig-zag
Java (dd-trace-java)	ZigZag	VarEncodingHelper.encodeSignedVarLong/decodeSignedVarLong (via sketches-java) call zigZagEncode/zigZagDecode
.NET (dd-trace-dotnet)	ZigZag	VarEncodingHelper.WriteVarLongZigZag/ReadVarLongZigZag
Ruby (dd-trace-rb)	Plain unsigned LEB128	pathway_context.rb — no ZigZag step
C++ (dd-trace-cpp)	N/A	Doesn't implement DSM

Went further than reading source — actually ran Go's, Python's, and Java's real, unmodified encode/decode code (cloned each repo, compiled/executed directly, no reimplementation) against a fixed pathway fixture and against a real Kafka message produced by a patched dd-trace-rb through the actual production code path. All three independently agree byte-for-byte on encode, and all three decode the same real message to the same correct hash/timestamp.

So this is a single-SDK bug in dd-trace-rb, not an ambiguous cross-SDK format disagreement — Ruby forgot the ZigZag step that every other DSM-capable tracer applies. Opened a fix directly against dd-trace-rb: DataDog/dd-trace-rb#6170. No change needed on the Go side; flagging here for visibility/tracking since this issue was filed against dd-trace-go first.

[mensfeld]

@olivia-gusto karafka maintainer here - just to be sure, this is purely on DD side or something that is not correct on my side in waterdrop?

[olivia-gusto]

@olivia-gusto karafka maintainer here - just to be sure, this is purely on DD side or something that is not correct on my side in waterdrop?

hi @mensfeld nice to meet you. I dug into this and can confirm it's 100% a Datadog issue, not WaterDrop or Karafka. Your instrumentation is doing exactly what it should afaik: It hooks into the dd-trace-rb DSM processor, and the Ruby side correctly encodes the timestamps as plain unsigned LEB128.

The actual bug is in dd-trace-*. The Go decoder is expecting a signed, zigzag-encoded varint. It looks like it's just reusing sketches-go encoding helpers (which need zigzag for negative bin indices), but pathway timestamps are never negative anyway. So when the plain-unsigned bytes from a Ruby producer hit the Go decoder, it scrambles the numbers. No errors get thrown, you just end up with a garbage timestamp.

Go-to-Go setups work perfectly since both sides are applying the same zigzag transform, but cross-language setups involving Ruby completely break down. In fact, all Ruby-to-other-language pathways break down.

The fix was implemented here and my team has been running it for a few days with no issues so far!