Skip to content

fix(backends): enable ROCm/HIP GPU offload for ggml audio backends (#10666)#10667

Merged
mudler merged 1 commit into
masterfrom
fix/rocm-hip-tts-backends-10666
Jul 4, 2026
Merged

fix(backends): enable ROCm/HIP GPU offload for ggml audio backends (#10666)#10667
mudler merged 1 commit into
masterfrom
fix/rocm-hip-tts-backends-10666

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

Summary

Fixes #10666. The ROCm variants of the ggml-based audio backends — rocm-qwen3-tts-cpp, rocm-omnivoice-cpp, and the same-family acestep-cpp / vibevoice-cpp — silently ran on CPU instead of the AMD GPU. The reporter's log showed [Load] Talker backend: CPU (CPU threads: 4); upstream qwentts.cpp selects the best device via ggml_backend_init_best(), so a CPU result means the HIP backend was never compiled into the binary.

Two coupled build defects, both invisible without a ROCm build:

  1. Wrong CMake flag (primary). The Makefiles passed -DGGML_HIPBLAS=ON, but the vendored ggml only understands -DGGML_HIP=ON (GGML_HIPBLAS was removed upstream — no references anywhere in the source tree). The flag was a silent no-op: the ggml-hip target was never created and no GPU code was built. The blocks also lacked the ROCm compiler (CC/CXX) and AMDGPU_TARGETS a HIP build requires.
  2. Missing link (secondary). The CMake foreach that links ggml GPU backends into the module listed blas cuda metal vulkan [sycl] but not hip — so even a built ggml-hip would not be linked and its static backend-registration constructor would never run.

CUDA users were unaffected because cublas passes the correct GGML_CUDA=ON and the foreach already links cuda — which is exactly why the two code paths diverged.

Changes

Across qwen3-tts-cpp, omnivoice-cpp, acestep-cpp, vibevoice-cpp (Makefile + CMakeLists):

  • Replace the no-op GGML_HIPBLAS block with the proven llama-cpp hipblas recipe: ROCm clang CC/CXX, AMDGPU_TARGETS (default list includes gfx1201, the reporter's R9700), and -DGGML_HIP=ON.
  • Add hip to each linking foreach so ggml-hip and its propagated ROCm deps (hip::host, roc::rocblas, roc::hipblas) land on the module's link line.

No run.sh change is needed: the HIP backend is statically linked into each CPU-SIMD variant .so and auto-selected at load time.

Testing

  • All four Makefiles parse with BUILD_TYPE=hipblas and resolve to -DGGML_HIP=ON -DAMDGPU_TARGETS=….
  • Confirmed the vendored ggml defines GGML_HIP and creates target ggml-hip; PRIVATE deps of a static lib propagate to consumers, so linking ggml-hip pulls the ROCm runtime automatically.
  • The *_HAVE_HIP compile-defs newly set on acestep/vibevoice are inert (defined but unreferenced) — no behavioral change beyond enabling GPU.

⚠️ I do not have a ROCm toolchain/GPU locally, so final validation is the ROCm CI build plus confirmation on the reporter's gfx1201 box.

Assisted-by: Claude:claude-opus-4-8[1m] [Claude Code]

…10666)

qwen3-tts-cpp, omnivoice-cpp, acestep-cpp and vibevoice-cpp shipped
rocm-* variants that silently ran on CPU ([Load] backend: CPU). Two
coupled defects:

- The Makefiles passed -DGGML_HIPBLAS=ON, but the vendored ggml only
  understands -DGGML_HIP=ON (GGML_HIPBLAS was removed upstream), so the
  ggml-hip backend target was never created and no GPU code was built.
- The CMake foreach that links the ggml GPU backends into the module
  listed blas/cuda/metal/vulkan but not hip, so even a built ggml-hip
  would not have been linked and its static backend registration would
  never run.

CUDA users were unaffected because cublas passes the correct GGML_CUDA=ON
and the foreach already links cuda. Mirror the proven llama-cpp hipblas
block (ROCm clang CC/CXX + AMDGPU_TARGETS) and add hip to each foreach.
Upstream picks the best device via ggml_backend_init_best(), so no
runtime flag is needed once HIP is compiled and linked.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8[1m] [Claude Code]
@mudler
mudler self-requested a review July 3, 2026 20:58
@mudler mudler added the bug Something isn't working label Jul 3, 2026
@mudler
mudler merged commit 38350d3 into master Jul 4, 2026
208 of 214 checks passed
@mudler
mudler deleted the fix/rocm-hip-tts-backends-10666 branch July 4, 2026 07:08
l0caldadmin added a commit to l0caldadmin/LocalAI that referenced this pull request Jul 4, 2026
* fix(gpu-libs): bundle hipBLASLt TensileLibrary data so ROCm backends stop falling back (mudler#10660) (mudler#10672) the 

The ROCm packager copied rocBLAS kernel data (rocblas/library/*.dat) into the
bundled lib/ dir and run.sh pointed ROCBLAS_TENSILE_LIBPATH at it, but the
parallel hipBLASLt data dir (hipblaslt/library/TensileLibrary_lazy_gfx*.dat)
was never packaged and no HIPBLASLT_TENSILE_LIBPATH was set. The bundled
libhipblaslt.so therefore resolved its per-arch kernel data relative to itself,
found nothing, and silently fell back to slow generic kernels, logging:

    rocblaslt error: Cannot read "TensileLibrary_lazy_gfx1201.dat": No such file or directory
    rocblaslt error: Could not load "TensileLibrary_lazy_gfx1201.dat"

Fix, mirroring the existing rocBLAS handling:
- package-gpu-libs.sh: extract the rocblas data-dir copy into a reusable
  copy_rocm_data_dir helper and call it for both rocblas and hipblaslt.
- llama-cpp/turboquant run.sh: export HIPBLASLT_TENSILE_LIBPATH when the
  bundled hipblaslt/library dir exists.

The helper takes an optional ROCM_BASE_DIRS override so the copy is unit
testable without a real ROCm install; add a regression test that runs
package_rocm_libs against a fabricated ROCm tree and asserts both data dirs
are bundled.

Note: this bundles whatever gfx*.dat the build image's ROCm provides. If a
given arch's tensile data is absent from the shipped ROCm, that arch still
needs a ROCm bump; the packaging gap itself is fixed for every supported arch.


Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

* chore: ⬆️ Update ggml-org/llama.cpp to `d4cff114c0084f1fbc9b4c62717eca8fb2ae494a` (mudler#10671)

:arrow_up: Update ggml-org/llama.cpp

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>

* chore: :arrow_up: Update CrispStrobe/CrispASR to `f35185b876fc482fcb2053a81a2697936ed5fcc0` (mudler#10670)

:arrow_up: Update CrispStrobe/CrispASR

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>

* fix(backends): enable ROCm/HIP GPU offload for ggml audio backends (mudler#10666) (mudler#10667)

qwen3-tts-cpp, omnivoice-cpp, acestep-cpp and vibevoice-cpp shipped
rocm-* variants that silently ran on CPU ([Load] backend: CPU). Two
coupled defects:

- The Makefiles passed -DGGML_HIPBLAS=ON, but the vendored ggml only
  understands -DGGML_HIP=ON (GGML_HIPBLAS was removed upstream), so the
  ggml-hip backend target was never created and no GPU code was built.
- The CMake foreach that links the ggml GPU backends into the module
  listed blas/cuda/metal/vulkan but not hip, so even a built ggml-hip
  would not have been linked and its static backend registration would
  never run.

CUDA users were unaffected because cublas passes the correct GGML_CUDA=ON
and the foreach already links cuda. Mirror the proven llama-cpp hipblas
block (ROCm clang CC/CXX + AMDGPU_TARGETS) and add hip to each foreach.
Upstream picks the best device via ggml_backend_init_best(), so no
runtime flag is needed once HIP is compiled and linked.


Assisted-by: Claude:claude-opus-4-8[1m] [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: LocalAI [bot] <139863280+localai-bot@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Backend rocm-qwen3-tts-cpp using CPU instead of GPU HIPBLAS gfx1201

2 participants