From c999559204b0ec7936c24e93154cb3f21d53091c Mon Sep 17 00:00:00 2001 From: Debasish Ghosh Date: Fri, 7 May 2021 11:01:52 +0530 Subject: [PATCH 1/5] Started introducing cloudflow-contrib thoughts into cloudflow docs --- .../docs/modules/ROOT/pages/app-development-process.adoc | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/shared-content-source/docs/modules/ROOT/pages/app-development-process.adoc b/docs/shared-content-source/docs/modules/ROOT/pages/app-development-process.adoc index b6f796fa7..c97b660fb 100644 --- a/docs/shared-content-source/docs/modules/ROOT/pages/app-development-process.adoc +++ b/docs/shared-content-source/docs/modules/ROOT/pages/app-development-process.adoc @@ -27,7 +27,9 @@ The Streamlet model offers an abstraction to: In the current version, Cloudflow includes backend `Streamlet` implementations for Akka, Apache Spark - Structured Streaming, and Apache Flink. Using these implementations you can write business logic in the native API of the backend. -Additionally, Cloudflow can be extended with new streaming backends. +Native support for Flink and Spark streamlets are supported as _legacy_ versions and will be discontinued in future. The current version introduces Flink and Spark integrations with more controls in the hands of the users. Users now have more control on deployment and management of Flink and Spark streamlets while still using the same Cloudflow streamlet API for developing their business logic. However, Akka will continue to be supported natively as in earlier versions. +Integration support for Flink and Spark, thus being externalized, makes Cloudflow easily extensible with new streaming backends. +For more details on externalized Flink and Spark integrations, please have a look at xref:develop:cloudflow-contrib-change-me.adoc[Cloudflow Contrib] documentation. The following table shows the current support for Java and Scala implementations for each runtime. @@ -264,6 +266,8 @@ $ kubectl cloudflow deploy /cloudflow/examples/call-record-aggregator/target/cal This method is not only dev-friendly, but is also compatible with the typical CI/CD deployments. This allows you to take the application from dev to production in a controlled way. +The deployment procedure will be a bit different with the _cloudflow contrib_ approach where Flink and Spark applications are supported through external plugins. Akka applications will be fully depoyed using `kubectl cloudflow` as above. However for Spark and Flink applications, you need to use an extra plugin and carry out a few extra steps to make them known to the cloudflow engine. For details please have a look at xref:develop:cloudflow-contrib-change-me.adoc[Cloudflow Contrib] documentation. + == Conclusion As a developer, Cloudflow gives you a set of powerful tools to accelerate the application development process: From 683ff73c51c5c5da89e0baf20e174aa85269cdc1 Mon Sep 17 00:00:00 2001 From: Debasish Ghosh Date: Fri, 7 May 2021 11:16:53 +0530 Subject: [PATCH 2/5] Trying to fix error in dcos build --- .../docs/modules/ROOT/pages/app-development-process.adoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/shared-content-source/docs/modules/ROOT/pages/app-development-process.adoc b/docs/shared-content-source/docs/modules/ROOT/pages/app-development-process.adoc index c97b660fb..805024240 100644 --- a/docs/shared-content-source/docs/modules/ROOT/pages/app-development-process.adoc +++ b/docs/shared-content-source/docs/modules/ROOT/pages/app-development-process.adoc @@ -265,7 +265,6 @@ $ kubectl cloudflow deploy /cloudflow/examples/call-record-aggregator/target/cal This method is not only dev-friendly, but is also compatible with the typical CI/CD deployments. This allows you to take the application from dev to production in a controlled way. - The deployment procedure will be a bit different with the _cloudflow contrib_ approach where Flink and Spark applications are supported through external plugins. Akka applications will be fully depoyed using `kubectl cloudflow` as above. However for Spark and Flink applications, you need to use an extra plugin and carry out a few extra steps to make them known to the cloudflow engine. For details please have a look at xref:develop:cloudflow-contrib-change-me.adoc[Cloudflow Contrib] documentation. == Conclusion From fd019d5271ad62fb7d5fb71b415c0c70d9dc0376 Mon Sep 17 00:00:00 2001 From: Debasish Ghosh Date: Mon, 10 May 2021 11:16:50 +0530 Subject: [PATCH 3/5] Incorporated review feedback changes --- .../docs/modules/ROOT/pages/app-development-process.adoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/shared-content-source/docs/modules/ROOT/pages/app-development-process.adoc b/docs/shared-content-source/docs/modules/ROOT/pages/app-development-process.adoc index 805024240..46980b57a 100644 --- a/docs/shared-content-source/docs/modules/ROOT/pages/app-development-process.adoc +++ b/docs/shared-content-source/docs/modules/ROOT/pages/app-development-process.adoc @@ -27,9 +27,9 @@ The Streamlet model offers an abstraction to: In the current version, Cloudflow includes backend `Streamlet` implementations for Akka, Apache Spark - Structured Streaming, and Apache Flink. Using these implementations you can write business logic in the native API of the backend. -Native support for Flink and Spark streamlets are supported as _legacy_ versions and will be discontinued in future. The current version introduces Flink and Spark integrations with more controls in the hands of the users. Users now have more control on deployment and management of Flink and Spark streamlets while still using the same Cloudflow streamlet API for developing their business logic. However, Akka will continue to be supported natively as in earlier versions. -Integration support for Flink and Spark, thus being externalized, makes Cloudflow easily extensible with new streaming backends. -For more details on externalized Flink and Spark integrations, please have a look at xref:develop:cloudflow-contrib-change-me.adoc[Cloudflow Contrib] documentation. +Along with the currently supported built-in integration of Flink and Spark streamlets via the Cloudflow operator, Cloudflow also supports external integrations for these streaming platforms through additional plugins. The current version introduces Flink and Spark integrations with more controls in the hands of the users. Users now have more control on deployment and management of Flink and Spark streamlets while still using the same Cloudflow streamlet API for developing their business logic. However, Akka will continue to be supported natively as in earlier versions. +Integration support for Flink and Spark, thus being externalized, makes Cloudflow easily extensible with new streaming backends. The new externalized integration has been marked _Experimental_ in the current version. +For more details on externalized Flink and Spark integrations, please have a look at https://lightbend.github.io/cloudflow-contrib/docs/0.0.4/index.html[Cloudflow Contrib] documentation. The following table shows the current support for Java and Scala implementations for each runtime. @@ -265,7 +265,7 @@ $ kubectl cloudflow deploy /cloudflow/examples/call-record-aggregator/target/cal This method is not only dev-friendly, but is also compatible with the typical CI/CD deployments. This allows you to take the application from dev to production in a controlled way. -The deployment procedure will be a bit different with the _cloudflow contrib_ approach where Flink and Spark applications are supported through external plugins. Akka applications will be fully depoyed using `kubectl cloudflow` as above. However for Spark and Flink applications, you need to use an extra plugin and carry out a few extra steps to make them known to the cloudflow engine. For details please have a look at xref:develop:cloudflow-contrib-change-me.adoc[Cloudflow Contrib] documentation. +The deployment procedure will be a bit different with the _cloudflow contrib_ approach where Flink and Spark applications are supported through external plugins. Akka applications will be fully deployed using `kubectl cloudflow` as above. However for Spark and Flink applications, you need to use an extra plugin and carry out a few extra steps to make them known to the Cloudflow engine. For details please have a look at https://lightbend.github.io/cloudflow-contrib/docs/0.0.4/index.html[Cloudflow Contrib] documentation. == Conclusion From d493515ee59dafc39481305a7963ce2dc8e7cb66 Mon Sep 17 00:00:00 2001 From: Debasish Ghosh Date: Mon, 10 May 2021 14:55:54 +0530 Subject: [PATCH 4/5] cloudflow-contrib references added to Spark and Flink support sections --- .../docs/modules/develop/pages/use-flink-streamlets.adoc | 2 ++ .../docs/modules/develop/pages/use-spark-streamlets.adoc | 2 ++ 2 files changed, 4 insertions(+) diff --git a/docs/shared-content-source/docs/modules/develop/pages/use-flink-streamlets.adoc b/docs/shared-content-source/docs/modules/develop/pages/use-flink-streamlets.adoc index eba3fbf8e..a4b31a97a 100644 --- a/docs/shared-content-source/docs/modules/develop/pages/use-flink-streamlets.adoc +++ b/docs/shared-content-source/docs/modules/develop/pages/use-flink-streamlets.adoc @@ -87,6 +87,8 @@ In <>, we can visualize the chain of delegatio - The Flink Job Manager then requests task manager resources from Kubernetes to deploy the distributed processing. - Finally, if and when resources are available, the Flink-bound task managers start as Kubernetes pods. The task managers are the components tasked with the actual data processing, while the Job Manager serves as coordinator of the (stream) data process. +In case you are using the cloudflow-contrib model of integration, you need to go through some additional steps to complete the deployment of your Flink streamlets. This https://lightbend.github.io/cloudflow-contrib/docs/0.0.4/get-started/flink-native.html[section] on cloudflow-contrib has more details. + If you make any changes to the streamlet and deploy the application again, the existing Flink application will be stopped, and the new version will be started to reflect the changes. It could be that Flink streamlets are restarted in other ways, for instance, by administrators. diff --git a/docs/shared-content-source/docs/modules/develop/pages/use-spark-streamlets.adoc b/docs/shared-content-source/docs/modules/develop/pages/use-spark-streamlets.adoc index 67c0e389f..08cbd8071 100644 --- a/docs/shared-content-source/docs/modules/develop/pages/use-spark-streamlets.adoc +++ b/docs/shared-content-source/docs/modules/develop/pages/use-spark-streamlets.adoc @@ -118,6 +118,8 @@ In the Spark streamlet deployment model shown above, we can visualize the chain In this architecture, the Spark driver runs the Cloudflow-specific logic that connects the streamlet to our managed data streams, at which point the streamlet starts consuming from inlets. The streamlet advances through the data streams that are provided on inlets and writes data to outlets. +In case you are using the cloudflow-contrib model of integration, you need to go through some additional steps to complete the deployment of your Spark streamlets. This https://lightbend.github.io/cloudflow-contrib/docs/0.0.4/get-started/spark-native.html[section] on cloudflow-contrib has more details. + If you make any changes to the streamlet and deploy the application again, the existing Spark applications will be stopped, and the new version will be started to reflect the changes. It could be that Spark streamlets are restarted in other ways, for instance by administrators. From 932207958e1aeea0b06b25460a825a6e15015056 Mon Sep 17 00:00:00 2001 From: Debasish Ghosh Date: Mon, 17 May 2021 11:06:23 +0530 Subject: [PATCH 5/5] REview feedbacks --- .../docs/modules/ROOT/pages/app-development-process.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/shared-content-source/docs/modules/ROOT/pages/app-development-process.adoc b/docs/shared-content-source/docs/modules/ROOT/pages/app-development-process.adoc index 46980b57a..da9186b9e 100644 --- a/docs/shared-content-source/docs/modules/ROOT/pages/app-development-process.adoc +++ b/docs/shared-content-source/docs/modules/ROOT/pages/app-development-process.adoc @@ -27,9 +27,9 @@ The Streamlet model offers an abstraction to: In the current version, Cloudflow includes backend `Streamlet` implementations for Akka, Apache Spark - Structured Streaming, and Apache Flink. Using these implementations you can write business logic in the native API of the backend. -Along with the currently supported built-in integration of Flink and Spark streamlets via the Cloudflow operator, Cloudflow also supports external integrations for these streaming platforms through additional plugins. The current version introduces Flink and Spark integrations with more controls in the hands of the users. Users now have more control on deployment and management of Flink and Spark streamlets while still using the same Cloudflow streamlet API for developing their business logic. However, Akka will continue to be supported natively as in earlier versions. +Along with the currently supported built-in integration of Flink and Spark streamlets via the Cloudflow operator, Cloudflow also supports external integrations for these streaming platforms through additional plugins. Using this new integration, users now have more control on deployment and management of Flink and Spark streamlets while still using the same Cloudflow streamlet API for developing their business logic. However, Akka will continue to be supported natively as in earlier versions. Integration support for Flink and Spark, thus being externalized, makes Cloudflow easily extensible with new streaming backends. The new externalized integration has been marked _Experimental_ in the current version. -For more details on externalized Flink and Spark integrations, please have a look at https://lightbend.github.io/cloudflow-contrib/docs/0.0.4/index.html[Cloudflow Contrib] documentation. +For more details on externalized Flink and Spark integrations, please have a look at https://lightbend.github.io/cloudflow-contrib/[Cloudflow Contrib] documentation. The following table shows the current support for Java and Scala implementations for each runtime. @@ -265,7 +265,7 @@ $ kubectl cloudflow deploy /cloudflow/examples/call-record-aggregator/target/cal This method is not only dev-friendly, but is also compatible with the typical CI/CD deployments. This allows you to take the application from dev to production in a controlled way. -The deployment procedure will be a bit different with the _cloudflow contrib_ approach where Flink and Spark applications are supported through external plugins. Akka applications will be fully deployed using `kubectl cloudflow` as above. However for Spark and Flink applications, you need to use an extra plugin and carry out a few extra steps to make them known to the Cloudflow engine. For details please have a look at https://lightbend.github.io/cloudflow-contrib/docs/0.0.4/index.html[Cloudflow Contrib] documentation. +The deployment procedure will be different with the _cloudflow contrib_ approach where Flink and Spark applications are supported through external plugins. Akka applications will be fully deployed using `kubectl cloudflow` as above. However, for Spark and Flink applications, you need to use an extra plugin and carry out a few extra steps to make them known to the Cloudflow engine. For details please have a look at https://lightbend.github.io/cloudflow-contrib/[Cloudflow Contrib] documentation. == Conclusion