Stream packets to Kafka

[sandman137]

Essentially
sensor -> receiver -> Kafka

[r8ygun]

@sandman137 @vadorovsky I'm curious about how you'd expect this to work. Would each packet received by the receiver end up as a single Kafka message, or would you expect to batch them up a bit (say to 1MB like the default send limit)?

[shyam-dev]

We usually send out a TLV buffer that contains a certain number of packets where the total size of the packets is within a configurable size. Thus, for example, if we set the size of a kafka message as 65k, we will fill it up with packets such that their total size is less than 65k. On the receiver end, we can read out individual packets from this buffer.

…

On Tue, Apr 19, 2022 at 6:51 AM Ryan Harvey ***@***.***> wrote: @sandman137 <https://github.com/sandman137> @vadorovsky <https://github.com/vadorovsky> I'm curious about how you'd expect this to work. Would each packet received by the receiver end up as a single Kafka message, or would you expect to batch them up a bit (say to 1MB like the default send limit)? — Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAK6V6HDGQ5E7PRZNRCPDYLVF227TANCNFSM5SGRN3YA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[vadorovsky]

I agree with @shyam-dev - I would go for 65K as a default batch for Kafka, of course it would be configurable. I wouldn't go for megabytes like in S3, which is an object file store.

[r8ygun]

I've started working on this on my fork - happy if you want to assign it.

[sandman137]

Awesome, just assigned it to you @r8ygun

[r8ygun]

@sandman137 I've raised a draft PR (#20) with some questions/notes. It's not code complete, but it would be good to get your/others input before I mark it as ready as there will be some stuff to do depending on what behaviour you all expect.

[sandman137]

@r8ygun thank you. I will let @vadorovsky comment on the same and take it to closure.

[vadorovsky]

Ryan's PR was merged, thanks for your contribution!

But I will keep this issue open. As you can see, the arm64 build fails:

https://github.com/deepfence/PacketStreamer/runs/6415308027?check_suite_focus=true

That's because https://github.com/confluentinc/confluent-kafka-go is wrapping a C library and providing vendored .a files built for x64... As a workaround for other architectures they suggest to build kafka manually.

That doesn't sound good for me and I would rather try some library written entirely in Go which (hopefully) doesn't require lots of gymnastics in order to get multi-arch builds, like:

https://github.com/segmentio/kafka-go

I will try using it as soon as I have time for that.