Skip to content
This repository was archived by the owner on Nov 22, 2024. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,9 @@ The Streamlet model offers an abstraction to:

In the current version, Cloudflow includes backend `Streamlet` implementations for Akka, Apache Spark - Structured Streaming, and Apache Flink.
Using these implementations you can write business logic in the native API of the backend.
Additionally, Cloudflow can be extended with new streaming backends.
Along with the currently supported built-in integration of Flink and Spark streamlets via the Cloudflow operator, Cloudflow also supports external integrations for these streaming platforms through additional plugins. Using this new integration, users now have more control on deployment and management of Flink and Spark streamlets while still using the same Cloudflow streamlet API for developing their business logic. However, Akka will continue to be supported natively as in earlier versions.
Integration support for Flink and Spark, thus being externalized, makes Cloudflow easily extensible with new streaming backends. The new externalized integration has been marked _Experimental_ in the current version.
For more details on externalized Flink and Spark integrations, please have a look at https://lightbend.github.io/cloudflow-contrib/[Cloudflow Contrib] documentation.

The following table shows the current support for Java and Scala implementations for each runtime.

Expand Down Expand Up @@ -263,6 +265,7 @@ $ kubectl cloudflow deploy /cloudflow/examples/call-record-aggregator/target/cal

This method is not only dev-friendly, but is also compatible with the typical CI/CD deployments.
This allows you to take the application from dev to production in a controlled way.
The deployment procedure will be different with the _cloudflow contrib_ approach where Flink and Spark applications are supported through external plugins. Akka applications will be fully deployed using `kubectl cloudflow` as above. However, for Spark and Flink applications, you need to use an extra plugin and carry out a few extra steps to make them known to the Cloudflow engine. For details please have a look at https://lightbend.github.io/cloudflow-contrib/[Cloudflow Contrib] documentation.

== Conclusion

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,8 @@ In <<flink-streamlet-deployment-model>>, we can visualize the chain of delegatio
- The Flink Job Manager then requests task manager resources from Kubernetes to deploy the distributed processing.
- Finally, if and when resources are available, the Flink-bound task managers start as Kubernetes pods. The task managers are the components tasked with the actual data processing, while the Job Manager serves as coordinator of the (stream) data process.

In case you are using the cloudflow-contrib model of integration, you need to go through some additional steps to complete the deployment of your Flink streamlets. This https://lightbend.github.io/cloudflow-contrib/docs/0.0.4/get-started/flink-native.html[section] on cloudflow-contrib has more details.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this at all?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just thought of adding this here since there is no mention of cloudflow-contrib in this section on flink streamlets.


If you make any changes to the streamlet and deploy the application again, the existing Flink application will be stopped, and the new version will be started to reflect the changes.
It could be that Flink streamlets are restarted in other ways, for instance, by administrators.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,8 @@ In the Spark streamlet deployment model shown above, we can visualize the chain
In this architecture, the Spark driver runs the Cloudflow-specific logic that connects the streamlet to our managed data streams, at which point the streamlet starts consuming from inlets.
The streamlet advances through the data streams that are provided on inlets and writes data to outlets.

In case you are using the cloudflow-contrib model of integration, you need to go through some additional steps to complete the deployment of your Spark streamlets. This https://lightbend.github.io/cloudflow-contrib/docs/0.0.4/get-started/spark-native.html[section] on cloudflow-contrib has more details.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this comment?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just thought of adding this here since there is no mention of cloudflow-contrib in this section on spark streamlets.


If you make any changes to the streamlet and deploy the application again, the existing Spark applications will be stopped, and the new version will be started to reflect the changes.
It could be that Spark streamlets are restarted in other ways, for instance by administrators.

Expand Down