[Proposal] Remove Scala from StreamPark — design and migration plan

[shangeyao]

Background

StreamPark was originally built with a Scala-first development model: user applications extend FlinkStreaming / FlinkTable / SparkStreaming / SparkBatch traits; Flink multi-version support is implemented via Scala shims behind FlinkShimsProxy; and most runtime modules (streampark-common, Flink client/K8s/packer, Spark client) are written in Scala.

This creates several long-term costs:

1. Build complexity: scala-maven-plugin, Scalafmt, Scalastyle, silencer, ScalaTest, and mixed Java/Scala compilation increase CI time and contributor onboarding friction.
2. Ecosystem alignment: Apache Flink 2.x removed the Scala DataStream API; continuing Scala-centric APIs fights upstream direction.
3. Console alignment: streampark-console-service is Java/Spring Boot but depends heavily on Scala-compiled libraries (HadoopUtils, FlinkClient, FlinkShimsProxy, packer pipelines, etc.).
4. Maintenance multiplier: Every shims change must be replicated across many Flink version modules, most of which are Scala.

#4432 removed the user-facing dev framework and connector wrappers. Flink SQL jobs now run through SqlClient → versioned StreamTableContext / TableContext → FlinkSqlExecutor. Structural refactors on that path must preserve external behavior.

This issue tracks the design and execution plan to remove Scala from the StreamPark codebase and safely refactor the Flink SQL submission stack.

Implementation PRs: #4458, #4461 merged; #4462–#4464 in review (stacked)
Depends on: #4409 (JDK 11 baseline ✅), #4432 (dev framework removal ✅)
Parent roadmap: #4410

Community workflow (PMC sync): migrate class-by-class; each new Java class should ship with tests (or document why not). Sub-task issues: #4445 #4446 #4447 #4448 #4449 #4450 #4451 #4452 #4453 — link PRs to the matching row below. Opening a separate GitHub issue per class is optional when you want an isolated review thread.

---

Progress Update (2026-07-29)

Overall: Stage 1 is ~75% landed on dev; Stage 2 stacked PRs (#4463, #4464) are in flight.

Milestone	Issue	PR	Status
streampark-common	#4445	#4458	✅ Merged
streampark-flink-core	#4446	—	✅ Closed (N/A) — entire module removed in #4432; no migration needed
streampark-flink-shims	#4447	#4461	✅ Merged
streampark-flink-sqlclient	#4448	#4462	🔄 PR open, CI green, mergeable
streampark-flink-packer	#4449	#4463	🔄 PR open (stacked on #4462)
streampark-flink-proxy	#4450	#4464	🔄 PR open (stacked on #4463)
streampark-flink-kubernetes	#4451	—	⏳ Pending
streampark-flink-client	#4452	—	⏳ Pending
Stage 3 refactor	#4453	—	⏳ Pending (after Stages 1–2)

Scala inventory on dev (post-#4461): ~118 .scala files remaining (down from ~240 baseline). Modules at 0 Scala: streampark-common, streampark-flink-shims. Intentional bridge module: streampark-common-scala-bridge (2 files, retained for downstream Scala interop during transition).

Note on streampark-flink-core: The module no longer exists — it was fully removed in #4432 along with FlinkStreaming / FlinkTable user-dev traits. Equivalent infrastructure (config, SQL runtime, initializers, version contexts) now lives in streampark-flink-shims-base and was migrated in #4461. Issue #4446 has been closed as obsolete.

---

Goals

• Eliminate Scala source code from all StreamPark modules (including tests).
• Remove Scala from the Maven build toolchain (scala-maven-plugin, Scalafmt/Scalastyle, ScalaTest, silencer, mockito-scala).
• Provide Java-first public APIs for user application development and SQL/streaming lifecycle where still needed.
• Preserve functional parity for Console operations (submit/cancel/deploy, SQL validation, K8s tracking, packaging).
• Lock and refactor the Flink SQL submission path without breaking SqlClient entry or deploy-mode behavior.
• Document migration path and deprecation timeline for existing Scala-based user jobs.

---

Current Inventory (2026-07-29, dev branch)

Updated after #4458 (common) and #4461 (shims) merged. Baseline was ~240 files before migration started.

Area	Scala .scala files	Notes
streampark-common	0	✅ Migrated in #4458
streampark-flink-shims (+ base)	0	✅ Migrated in #4461
streampark-flink-sqlclient	1	🔄 PR #4462 ready
streampark-flink-packer	34	🔄 PR #4463 in review
streampark-flink-proxy	1	🔄 PR #4464 in review
streampark-flink-kubernetes	38	Pending (#4451)
streampark-flink-client	27	Pending (#4452)
streampark-spark (remaining)	~15	Client + sqlclient (core traits removed in #4432)
streampark-common-scala-bridge	2	Intentional — Logger/Implicits bridge during transition
streampark-console	0	Pure Java; consumes Scala JARs
Total	~118	Down from ~240

Already removed (not in migration scope)

Module	PR	Notes
streampark-flink-connector (all sub-modules)	#4432	Sink/source wrapper helpers dropped; use Flink native connectors
streampark-spark-connector (all sub-modules)	#4432	Spark connector wrappers dropped
streampark-flink-core (entire module)	#4432	Module deleted; FlinkStreaming, FlinkTable, DataStreamExt removed. Core infra moved to streampark-flink-shims-base → migrated in #4461. #4446 closed as N/A.
streampark-spark-core Scala traits	#4432	SparkStreaming, SparkBatch removed

Key Scala-specific patterns in use

Pattern	Example	Java replacement strategy
object singleton facades	FlinkClient, FlinkShimsProxy, Utils	final class + static methods, or Spring @Component where appropriate
trait lifecycle API	FlinkStreaming, FlinkTable, SparkStreaming	Abstract Java class or interface + template method (new Java API where needed)
implicit conversions	Implicits, EnhancerImplicit, DataStreamExt	Explicit static helper classes; no implicit magic
enumeratum enums	SqlCommandParser command types	Java enum (migrated in common/shims)
ClassTag / reflection	FlinkClient shims dispatch	Plain Java classes + ClassLoader reflection (already partially used)
json4s	K8s REST JSON parsing	Jackson (already in dependency tree)
ScalaTest	packer/k8s/client tests	JUnit 5 + AssertJ (Console standard)

---

Execution Plan — Sub-Tasks

The overall plan is delivered in three stages. Stages 1 and 2 are language migration only — rewrite streampark-common / streampark-flink from Scala to Java following Java architecture and idioms, with no behavior changes. These tasks are a good entry point for new contributors; PMC members should provide guidance.

Important: PRs within Stage 1 and Stage 2 are not parallel. Each PR must land sequentially; later PRs depend on the previous one.

Stage 1 — Flink Core

Principle: no logic changes; migrate modules to Java with readable, idiomatic code. Minimal-risk executable task list:

#	Module	Brief role	Issue	PR	Status
1	streampark-common	Shared foundation — utils, config, filesystem, constants; depended on by all modules	#4445	#4458	✅ Merged
2	streampark-flink-core	Flink core — job runtime context, environment initialization	#4446	—	✅ N/A — module removed (#4432), issue closed
3	streampark-flink-shims	Flink multi-version adapter — shims hide API differences across Flink 1.17–1.20	#4447	#4461	✅ Merged
4	streampark-flink-sqlclient	Flink SQL job entry — SqlClient CLI, SQL parse & execute path	#4448	#4462	🔄 PR open

Stage 2 — Job deployment

#	Module	Brief role	Issue	PR	Status
1	streampark-flink-packer	Build & package user jobs into distributable deployable artifacts	#4449	#4463	🔄 PR open
2	streampark-flink-proxy	Shims proxy — ChildFirstClassLoader, per-Flink-version isolation and dispatch	#4450	#4464	🔄 PR open
3	streampark-flink-kubernetes	Kubernetes integration — deploy, job watchers, ingress, etc.	#4451	—	⏳ Pending
4	streampark-flink-client	Submit client — Flink job submit / cancel / deploy	#4452	—	⏳ Pending

Stage 3 — Architecture upgrade & refactor

Stages 1–2 complete the Scala → Java language conversion. Stage 3 revisits the modules above and performs necessary architectural refactors on that Java baseline. This stage is higher difficulty and should be led by core maintainers (or completed with their direct support).

#	Module	Brief role	Issue	PR	Status
1	Stage 3 scope	Architecture upgrade & refactor on Java baseline (Stage 3)	#4453	—	⏳ Pending

Already landed:

• [Build] Raise StreamPark compile and CI baseline to JDK 11 #4417 — JDK 11 baseline ✅
• [Flink] Remove dev framework and connector wrappers #4432 — Removed connectors, dev framework & streampark-flink-core module ✅
• [Build] Switch to 3.0.0-SNAPSHOT using Maven revision property #4426 — Maven ${revision} ✅
• [Migrate] Convert streampark-common from Scala to Java #4458 — streampark-common Scala → Java ✅
• [Migrate] Migrate streampark-flink-shims from Scala to Java #4461 — streampark-flink-shims Scala → Java ✅

---

Risks & Mitigations

Risk	Impact	Mitigation
Breaking user Scala jobs	High	Migration guide; clear 3.0 release notes
Shims classloader regressions	High	Integration tests per Flink version; no change to cache key semantics
Mode priority / StatementSet confusion	High	Phase 0 tests; explicit SqlSubmissionResult in Phase 5
Large PR review load	Medium	One architecture layer per phase (Phases 1–5)
Credential leakage in SQL logs	Medium	Redaction in Phase 7 only

---

Acceptance Criteria

1. ./mvnw clean install -DskipTests succeeds without scala-maven-plugin.
2. ./mvnw clean install full test suite passes (Java tests only).
3. Console can submit/cancel Flink & Spark apps on representative deploy modes (YARN, K8s session/application).
4. SQL validation and Flink SQL submit work for supported Flink versions.
5. Existing Console Flink SQL apps start without user changes; mode priority covered by tests.
6. JobGraph and Application submit paths verified (Phase 6).
7. No .scala files under src/main or src/test.
8. User migration guide published on streampark.apache.org.
9. Each stage/phase independently revertible.

---

References

• Parent roadmap: [Roadmap] StreamPark 3.0 #4410
• JDK 11 baseline: [Proposal] Migrate StreamPark Console baseline from JDK 8 to JDK 11 #4409 / [Build] Raise StreamPark compile and CI baseline to JDK 11 #4417 ✅
• Dev framework removal: [Flink] Remove StreamPark dev framework and connector wrappers #4431 / [Flink] Remove dev framework and connector wrappers #4432 ✅
• Maven ${revision}: [Build] Switch to 3.0.0-SNAPSHOT using Maven revision property #4425 / [Build] Switch to 3.0.0-SNAPSHOT using Maven revision property #4426 ✅
• Flatten POM gitignore: [Build] Stop tracking flatten-maven-plugin generated .flattened-pom.xml files #4443 / [Build] Gitignore flatten-maven-plugin generated POM files #4444
• FlinkShimsProxy / ChildFirstClassLoader — high sensitivity (see AGENTS.md)

[shangeyao]

Reformatted Sub-Issues as tables with [Module] Description titles, PR links, and completion status. Status reflects current PR branch state (PR #4418 open, #4417 not yet merged).

[shangeyao]

Removed PR column from sub-issue tables and reset all task statuses to Pending. Implementation PR is TBD — prior draft work does not count as final delivery.

[RocMarshal]

Thanks @shangeyao for the proposal.

Based on the outcomes of the recent community sync meeting, this initiative requires expedited support. We have reached a general consensus on the following guidelines:

Migration from Scala to Java will be conducted file-by-file or class-by-class. Each newly introduced Java class must be accompanied by corresponding test cases; if tests are omitted, the rationale must be explicitly stated.

Accordingly, the standard community workflow for migrating a Scala class to Java is outlined below:

• Select a Scala class​ you are interested in migrating.
• Create an issue​ and link it as a sub-task of [Proposal] Remove Scala from StreamPark — design and migration plan #4408. Clearly describe which specific class this issue targets.
• Begin development and adaptation:
  • Package Consistency:​ Please retain the original package path after migration. If retaining the same path is not feasible, provide a brief justification.
  • AI-Assisted Development:​ You are encouraged to leverage AI tools to facilitate development. However, we kindly ask that you perform at least one thorough review of the generated code to ensure correctness and soundness before submission.
• Submit a Pull Request​ and link it back to your sub-task issue.
• Ping the committers / PMC members​ to request a code review.