Add blog post: Kafka bucket notifications in Ceph RGW - Part 1 - #1183
Add blog post: Kafka bucket notifications in Ceph RGW - Part 1#1183sujay-d07 wants to merge 1 commit into
Conversation
Signed-off-by: sujay-d07 <sujaydongre07@gmail.com>
anthonyeleven
left a comment
anthonyeleven
left a comment
There was a problem hiding this comment.
Good stuff. Was AI used here?
|
|
||
| Getting Kafka bucket notifications working over plaintext takes about ten minutes. | ||
| Getting them working over TLS, with SASL, against a broker that actually checks who | ||
| you are — that took considerably longer, mostly because Kafka security and Ceph |
There was a problem hiding this comment.
The tense of the verbs in the first two sentences should agree, so here
s/that took/that takes/
| Getting them working over TLS, with SASL, against a broker that actually checks who | ||
| you are — that took considerably longer, mostly because Kafka security and Ceph | ||
| bucket notifications are almost always documented separately. Kafka's docs explain | ||
| listeners and JAAS. Ceph's docs list SNS topic attributes. Nobody sits you down and |
There was a problem hiding this comment.
suggest /sits you down/connects the dots/
|
|
||
| This post is the thing I wanted while adding Kafka security coverage to RGW's | ||
| bucket notification test suite. We start with a pipeline that has no security at | ||
| all, then add one property at a time — identity, then encryption, then both, then |
There was a problem hiding this comment.
suggest s/property/enhancement/
| Part 2 picks up with GSSAPI (Kerberos) and OAuthBearer. | ||
|
|
||
| ## The one thing worth understanding first | ||
|
|
There was a problem hiding this comment.
I'm not at all familiar with bucket notifications, but given this heading, I'm not seeing a "one thing". I suggest adding a short sentence laying that out before the following detail. Think TL;DR
| progression works: securing the connection means changing topic attributes. The | ||
| bucket notification configuration never changes. Not once, in any section below. | ||
|
|
||
| Here's where we're going, and which port each mechanism uses in this guide: |
| KAFKA_VERSION=3.9.0 | ||
|
|
||
| wget https://archive.apache.org/dist/kafka/${KAFKA_VERSION}/kafka_2.13-${KAFKA_VERSION}.tgz | ||
| tar -xzf kafka_2.13-${KAFKA_VERSION}.tgz |
There was a problem hiding this comment.
There are probably still some tar implementations that don't accept the -, so I'd leave it out.
|
|
||
| ### A Ceph dev cluster | ||
|
|
||
| `install-deps.sh` figures out your distro's package names, so these steps are the |
There was a problem hiding this comment.
personal peeve: "distro" bugs me and I wrote "distribution"
| ```bash | ||
| cd $KAFKA_HOME | ||
|
|
||
| # Use your real broker IP or hostname — the SAN has to match what clients connect to |
There was a problem hiding this comment.
another personal peeve: IP address
| bash /path/to/ceph/src/test/rgw/bucket_notification/kafka-security.sh | ||
| ``` | ||
|
|
||
| You get five files, all with the password `mypassword`: |
There was a problem hiding this comment.
Thank you for spelling out five
| --add-config 'SCRAM-SHA-512=[password=alice-secret]' | ||
| ``` | ||
|
|
||
| Check it took: |
There was a problem hiding this comment.
Check that it took effect:
| @@ -0,0 +1,855 @@ | |||
| --- | |||
| title: "Kafka Bucket Notifications in Ceph RGW — Part 1" | |||
There was a problem hiding this comment.
title should be about bucket notification and kafka security - i think that by now there are already blog posts dealing with bucket notification with kafka
|
|
||
| ## The one thing worth understanding first | ||
|
|
||
| Ceph RGW implements the S3 API, and bucket notifications let you push an event to a |
There was a problem hiding this comment.
the audience of this post knows what the RGW is doing, and also what bucket notifications are. please focus on bucket notifications with kafka security
| between mechanisms then costs you one `aws sns create-topic` call instead of a | ||
| broker restart. | ||
|
|
||
| ## Setting up |
There was a problem hiding this comment.
i think we can drop this entire section.
audience probably already have kafka cluster up and running, and they are not going to deploy ceph from source.
same goes for aws cli - no need to provide instructions here
|
|
I would like to promote this blog on social media through the Ceph LinkedIn and Twitter accounts. I'll wait until the changes have been made. |
4 participants
Really excited to share Part 1 of my blog on Kafka Bucket Notifications in Ceph RGW! :)
Part 1 includes the plaintext, SASL_PLAINTEXT (PLAIN and SCRAM), SSL, SASL_SSL, and mTLS mechanisms.
Each section includes the commands needed to reproduce the setup and steps to verify that events actually make it to the Kafka broker.