CloudflowApplication size reach etcd request size limit

[jenoOvchi]

Describe the bug
Today our CloudflowApplication size reach etcd request size limit. After kubectl-cloudflow deploy we find an error in the Cloudflow Operator logs:
2021-02-16 14:13:53,550 INFO [CloudflowApplication$Status$] - Setting error status for app pipeline-cais-dev
2021-02-16 14:13:53,879 ERROR [Materializer] - [action] Upstream failed.
cloudflow.operator.action.ActionException: Action updateStatus failed: Status(v1,Status,ListMeta(,,None),Some(Failure),Some(etcdserver: request is too large),None,None,Some(500))
at cloudflow.operator.action.SkuberActionExecutor$$anonfun$execute$3.applyOrElse(SkuberActionExecutor.scala:53)
at cloudflow.operator.action.SkuberActionExecutor$$anonfun$execute$3.applyOrElse(SkuberActionExecutor.scala:50)
at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:453)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
Caused by: skuber.api.client.package$K8SException: Status(v1,Status,ListMeta(,,None),Some(Failure),Some(etcdserver: request is too large),None,None,Some(500))
at skuber.api.client.impl.KubernetesClientImpl.$anonfun$toKubernetesResponse$1(KubernetesClientImpl.scala:648)
at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:256)
at akka.http.scaladsl.util.FastFuture$.$anonfun$transformWith$1(FastFuture.scala:37)
at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
at akka.http.scaladsl.util.FastFuture$.transformWith$extension(FastFuture.scala:45)
at akka.http.scaladsl.util.FastFuture$.transformWith$extension(FastFuture.scala:37)
at akka.http.scaladsl.util.FastFuture$FulfilledFuture.transformWith(FastFuture.scala:86)
at scala.concurrent.Future.flatMap(Future.scala:254)
at scala.concurrent.Future.flatMap$(Future.scala:254)
at akka.http.scaladsl.util.FastFuture$FulfilledFuture.flatMap(FastFuture.scala:77)
at skuber.api.client.impl.KubernetesClientImpl.toKubernetesResponse(KubernetesClientImpl.scala:646)
at skuber.api.client.impl.KubernetesClientImpl.$anonfun$makeRequestReturningObjectResource$1(KubernetesClientImpl.scala:188)
at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:433)
... 11 common frames omitted
2021-02-16 14:13:53,879 ERROR [Operator] - The status changes stream failed, terminating.
cloudflow.operator.action.ActionException: Action updateStatus failed: Status(v1,Status,ListMeta(,,None),Some(Failure),Some(etcdserver: request is too large),None,None,Some(500))
at cloudflow.operator.action.SkuberActionExecutor$$anonfun$execute$3.applyOrElse(SkuberActionExecutor.scala:53)
at cloudflow.operator.action.SkuberActionExecutor$$anonfun$execute$3.applyOrElse(SkuberActionExecutor.scala:50)
at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:453)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:93)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: skuber.api.client.package$K8SException: Status(v1,Status,ListMeta(,,None),Some(Failure),Some(etcdserver: request is too large),None,None,Some(500))
at skuber.api.client.impl.KubernetesClientImpl.$anonfun$toKubernetesResponse$1(KubernetesClientImpl.scala:648)
at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:256)
at akka.http.scaladsl.util.FastFuture$.$anonfun$transformWith$1(FastFuture.scala:37)
at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
at akka.http.scaladsl.util.FastFuture$.transformWith$extension(FastFuture.scala:45)
at akka.http.scaladsl.util.FastFuture$.transformWith$extension(FastFuture.scala:37)
at akka.http.scaladsl.util.FastFuture$FulfilledFuture.transformWith(FastFuture.scala:86)
at scala.concurrent.Future.flatMap(Future.scala:254)
at scala.concurrent.Future.flatMap$(Future.scala:254)
at akka.http.scaladsl.util.FastFuture$FulfilledFuture.flatMap(FastFuture.scala:77)
at skuber.api.client.impl.KubernetesClientImpl.toKubernetesResponse(KubernetesClientImpl.scala:646)
at skuber.api.client.impl.KubernetesClientImpl.$anonfun$makeRequestReturningObjectResource$1(KubernetesClientImpl.scala:188)
at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:433)
... 11 common frames omitted

Full size of this Cloudflow Application is 1,6 Mb. It can't be deployed because etcd limit is 1 Mb. Is it possible to reduce Cloudflow Application size? Why it is so big? I think it's a big blocker for real projects with Cloudflow.

To Reproduce
Deploy Cloudflow Application with manifest larger than 1 Mb.

Expected behavior
Application has been correctly deployed.

[RayRoestenburg]

The manifest can get large if you have many schemas or large schemas. We have plans to decrease the size of the CR, for instance by excluding the schemas.

[jenoOvchi]

The manifest can get large if you have many schemas or large schemas. We have plans to decrease the size of the CR, for instance by excluding the schemas.

Yes - now our schemas are big. It will be great!
What release will this feature be included in?

[RayRoestenburg]

This is not in our plans yet. Are you in contact with our support team?

On Wed, 17 Feb 2021 at 14:17, Evgeniy ***@***.***> wrote: The manifest can get large if you have many schemas or large schemas. We have plans to decrease the size of the CR, for instance by excluding the schemas. Yes - now our schemas are big. It will be great! What release will this feature be included in? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#986 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABGGCNHFAPCNU56WJNDJRDS7O6XPANCNFSM4XWR3IWA> .

-- *Akka Tech Lead, Lightbend, Inc.* ray@lightbend.com <raymond@lightbend.com> @RayRoestenburg <https://twitter.com/RayRoestenburg> <https://www.lightbend.com/>

[jenoOvchi]

This is not in our plans yet. Are you in contact with our support team?
On Wed, 17 Feb 2021 at 14:17, Evgeniy @.***> wrote: The manifest can get large if you have many schemas or large schemas. We have plans to decrease the size of the CR, for instance by excluding the schemas. Yes - now our schemas are big. It will be great! What release will this feature be included in? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#986 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABGGCNHFAPCNU56WJNDJRDS7O6XPANCNFSM4XWR3IWA .
-- Akka Tech Lead, Lightbend, Inc. ray@lightbend.com raymond@lightbend.com @RayRoestenburg https://twitter.com/RayRoestenburg https://www.lightbend.com/

No. On this project we use Open Source Cloudflow.

[andreaTP]

Fixed in 2.0.25 just released, feel free to re-open if you still have issues @jenoOvchi

[jenoOvchi]

Thanks!