Skip to content

fix(config): only inject llama.cpp serving options on the llama.cpp path#10822

Merged
mudler merged 1 commit into
masterfrom
fix/gate-llama-serving-defaults
Jul 14, 2026
Merged

fix(config): only inject llama.cpp serving options on the llama.cpp path#10822
mudler merged 1 commit into
masterfrom
fix/gate-llama-serving-defaults

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

Problem

Starting a non-llama backend model (e.g. longcat-video-avatar-1.5) in distributed mode fails at LoadModel:

unknown model option(s): cache_reuse, parallel

SetDefaults injects the llama.cpp server options cache_reuse (ApplyServingDefaults) and parallel (ApplyHardwareDefaults, re-applied per selected node by the distributed router) onto every model config, backend-blind. Every other backend silently ignores options it doesn't understand, so this was harmless until longcat-video, which is the only backend that strictly validates its options and hard-rejects unknowns.

Fix

Core - gate the injections by backend. New UsesLlamaCppServingOptions(backend) allow-list (llama-cpp + the empty/auto-detect case that resolves to llama.cpp from a GGUF file, mirroring how llamaCppDefaults is registered). It follows the existing UsesLlamaSamplerDefaults precedent for llama-only defaults. Applied at all three sites:

  • cache_reuse in ApplyServingDefaults (single-host)
  • parallel in ApplyHardwareDefaults (single-host)
  • parallel in applyNodeHardwareDefaults (distributed router, per selected node) - backendType threaded through

The typed NBatch field is intentionally left ungated: it's a proto field every backend simply ignores, which is exactly why batch never appeared in the error alongside the two option strings.

Backend - defense in depth. longcat-video now warns-and-ignores unknown model options and request params via a testable select_known_options helper, matching the other LocalAI Python backends, so a future server-injected option can't break loading again.

Tests

TDD throughout (each new behavior watched fail first):

  • core/config: new specs asserting cache_reuse / parallel are not injected for a non-llama backend, and still are for llama-cpp
  • core/services/nodes: new spec asserting the node-tuning path skips parallel for a non-llama backend
  • backend/python/longcat-video: new select_known_options unit tests

All affected suites pass locally (304 config, 336 nodes, 9 longcat util specs); golangci-lint --new-from-merge-base clean.

Assisted-by: Claude:claude-opus-4-8 [Claude Code]

SetDefaults injected the llama.cpp server options cache_reuse
(ApplyServingDefaults) and parallel (ApplyHardwareDefaults, re-applied
per selected node by the distributed router) onto every model config
regardless of backend. Every other backend ignores options it does not
understand, so this was harmless until longcat-video, which strictly
validates its options and fails LoadModel with
"unknown model option(s): cache_reuse, parallel".

Gate both injections behind a new UsesLlamaCppServingOptions allow-list
(llama-cpp plus the empty/auto-detect case that resolves to llama.cpp
from a GGUF file, mirroring how llamaCppDefaults is registered). This
follows the existing UsesLlamaSamplerDefaults precedent for llama-only
defaults. The typed NBatch field is deliberately left alone: it is a
proto field every backend simply ignores, which is why batch never
triggered the error.

Also harden the longcat-video backend to warn-and-ignore unknown model
options and request params through a testable select_known_options
helper, matching the other LocalAI Python backends, so a future
server-injected option cannot break loading again.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
@mudler
mudler merged commit b224c96 into master Jul 14, 2026
30 checks passed
@mudler
mudler deleted the fix/gate-llama-serving-defaults branch July 14, 2026 15:46
richiejp pushed a commit to richiejp/LocalAI that referenced this pull request Jul 16, 2026
…ath (mudler#10822)

SetDefaults injected the llama.cpp server options cache_reuse
(ApplyServingDefaults) and parallel (ApplyHardwareDefaults, re-applied
per selected node by the distributed router) onto every model config
regardless of backend. Every other backend ignores options it does not
understand, so this was harmless until longcat-video, which strictly
validates its options and fails LoadModel with
"unknown model option(s): cache_reuse, parallel".

Gate both injections behind a new UsesLlamaCppServingOptions allow-list
(llama-cpp plus the empty/auto-detect case that resolves to llama.cpp
from a GGUF file, mirroring how llamaCppDefaults is registered). This
follows the existing UsesLlamaSamplerDefaults precedent for llama-only
defaults. The typed NBatch field is deliberately left alone: it is a
proto field every backend simply ignores, which is why batch never
triggered the error.

Also harden the longcat-video backend to warn-and-ignore unknown model
options and request params through a testable select_known_options
helper, matching the other LocalAI Python backends, so a future
server-injected option cannot break loading again.

Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants