fix(config): only inject llama.cpp serving options on the llama.cpp path#10822
Merged
Conversation
SetDefaults injected the llama.cpp server options cache_reuse (ApplyServingDefaults) and parallel (ApplyHardwareDefaults, re-applied per selected node by the distributed router) onto every model config regardless of backend. Every other backend ignores options it does not understand, so this was harmless until longcat-video, which strictly validates its options and fails LoadModel with "unknown model option(s): cache_reuse, parallel". Gate both injections behind a new UsesLlamaCppServingOptions allow-list (llama-cpp plus the empty/auto-detect case that resolves to llama.cpp from a GGUF file, mirroring how llamaCppDefaults is registered). This follows the existing UsesLlamaSamplerDefaults precedent for llama-only defaults. The typed NBatch field is deliberately left alone: it is a proto field every backend simply ignores, which is why batch never triggered the error. Also harden the longcat-video backend to warn-and-ignore unknown model options and request params through a testable select_known_options helper, matching the other LocalAI Python backends, so a future server-injected option cannot break loading again. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
richiejp
pushed a commit
to richiejp/LocalAI
that referenced
this pull request
Jul 16, 2026
…ath (mudler#10822) SetDefaults injected the llama.cpp server options cache_reuse (ApplyServingDefaults) and parallel (ApplyHardwareDefaults, re-applied per selected node by the distributed router) onto every model config regardless of backend. Every other backend ignores options it does not understand, so this was harmless until longcat-video, which strictly validates its options and fails LoadModel with "unknown model option(s): cache_reuse, parallel". Gate both injections behind a new UsesLlamaCppServingOptions allow-list (llama-cpp plus the empty/auto-detect case that resolves to llama.cpp from a GGUF file, mirroring how llamaCppDefaults is registered). This follows the existing UsesLlamaSamplerDefaults precedent for llama-only defaults. The typed NBatch field is deliberately left alone: it is a proto field every backend simply ignores, which is why batch never triggered the error. Also harden the longcat-video backend to warn-and-ignore unknown model options and request params through a testable select_known_options helper, matching the other LocalAI Python backends, so a future server-injected option cannot break loading again. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Signed-off-by: Richard Palethorpe <io@richiejp.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Starting a non-llama backend model (e.g.
longcat-video-avatar-1.5) in distributed mode fails atLoadModel:SetDefaultsinjects the llama.cpp server optionscache_reuse(ApplyServingDefaults) andparallel(ApplyHardwareDefaults, re-applied per selected node by the distributed router) onto every model config, backend-blind. Every other backend silently ignores options it doesn't understand, so this was harmless untillongcat-video, which is the only backend that strictly validates its options and hard-rejects unknowns.Fix
Core - gate the injections by backend. New
UsesLlamaCppServingOptions(backend)allow-list (llama-cpp+ the empty/auto-detect case that resolves to llama.cpp from a GGUF file, mirroring howllamaCppDefaultsis registered). It follows the existingUsesLlamaSamplerDefaultsprecedent for llama-only defaults. Applied at all three sites:cache_reuseinApplyServingDefaults(single-host)parallelinApplyHardwareDefaults(single-host)parallelinapplyNodeHardwareDefaults(distributed router, per selected node) -backendTypethreaded throughThe typed
NBatchfield is intentionally left ungated: it's a proto field every backend simply ignores, which is exactly whybatchnever appeared in the error alongside the two option strings.Backend - defense in depth.
longcat-videonow warns-and-ignores unknown model options and request params via a testableselect_known_optionshelper, matching the other LocalAI Python backends, so a future server-injected option can't break loading again.Tests
TDD throughout (each new behavior watched fail first):
core/config: new specs assertingcache_reuse/parallelare not injected for a non-llama backend, and still are forllama-cppcore/services/nodes: new spec asserting the node-tuning path skipsparallelfor a non-llama backendbackend/python/longcat-video: newselect_known_optionsunit testsAll affected suites pass locally (304 config, 336 nodes, 9 longcat util specs);
golangci-lint --new-from-merge-baseclean.Assisted-by: Claude:claude-opus-4-8 [Claude Code]