fix(ik-llama): port multimodal path to mtmd API and bump to f96eaddb (#10534)

[localai-bot]

Description

PR #10534 bumps IK_LLAMA_VERSION to f96eaddba8bed6a9a5e628bbf6a566775c70b49c (repo ikawrakow/ik_llama.cpp). Between the old pin and that commit, ik_llama landed an upstream change that prunes examples/llava (clip.h/.cpp, llava.h/.cpp). The ik-llama backend's grpc-server.cpp is a hand-rolled, old-style llava server: it built a local myclip library from those now-deleted files and called the removed clip/llava C API (clip_model_load, clip_image_load_from_bytes, llava_image_embed_make_with_clip_img). The bump therefore does not build until the multimodal path is ported.

This PR is the surgical in-place port of that path onto ik_llama's surviving multimodal stack, the mtmd library (examples/mtmd/, public headers mtmd.h + mtmd-helper.h). ik_llama is a fork that keeps the old common API, so the text path is unaffected; only the clip/llava multimodal path needed changing. The backend keeps its own self-contained examples/grpc-server/ (it is not rewritten onto the sibling llama-cpp backend's modular-server architecture, which is at a different refactor point and uses a different mctx access pattern).

Changes

• Makefile: bump IK_LLAMA_VERSION to f96eaddb.
• prepare.sh: drop the clip/llava source-copy + sed block. mtmd is a library target that builds transitively; no source copy is needed.
• CMakeLists.txt: remove the myclip target; link mtmd and add its include dir; build grpc-server as C++17 (mtmd headers require it).
• patches/: drop 0002-clip-ggml-quantize-chunk-user-data.patch (it targeted the now-deleted examples/llava/clip.cpp; the replacement examples/mtmd/clip.cpp never calls ggml_quantize_chunk, so the fix is needed nowhere). Keep 0001-fix-missing-cstdint-include.patch (verified still applies cleanly).
• grpc-server.cpp / utils.hpp: replace the legacy multimodal path with the high-level mtmd API.

Old → new mtmd call-site mapping (verified against the fork @ f96eaddb)

Old (removed)	New (verified signature in examples/mtmd/)
clip_ctx *clp_ctx member	mtmd_context *mctx
clip_model_load(mmproj, 1) (before model load)	mtmd_init_from_file(mmproj, model, mtmd_context_params) moved after llama_init_from_gpt_params (it requires the loaded llama_model*)
clip_n_mmproj_embd dim guard	dropped (mtmd validates the projector internally, returns nullptr on mismatch)
clip_image_u8_init + clip_image_load_from_bytes(buf,len,..)	mtmd_helper_bitmap_init_from_buf(mctx, buf, len)
[img-N] prefix-splitting into per-image prefix_prompt	translate each [img-N], in order, into mtmd_default_marker(); collect bitmaps in marker order; trailing suffix kept on the normal token path
llava_image_embed_make_with_clip_img(...) eager encode	removed (encode happens inside eval)
struct llava_embd_batch + manual per-image llama_decode loop	mtmd_tokenize(mctx, chunks, &input_text, bitmaps, n) then mtmd_helper_eval_chunks(mctx, ctx, chunks, n_past, seq_id, n_batch, /*logits_last=*/false, &new_n_past)
clip_image_u8_free / free(image_embedding) cleanup	mtmd_bitmap_free

slot_image now holds a single mtmd_bitmap * (owned by the slot, freed on reset) instead of the clip image + embedding + token-count + per-image prefix fields.

Notes for Reviewers

The text (non-multimodal) path is unchanged - it still compiles against ik_llama's retained old common API; the only shared-flow edit makes the per-image prefix-token queue empty for the multimodal case.

Multimodal (VLM) inference is NOT yet verified end-to-end. It needs a full docker build of the backend plus an image-model e2e run, which I could not perform here.

What was statically verified (against a clone of ikawrakow/ik_llama.cpp @ f96eaddb):

• Every mtmd symbol called exists with the exact signature used: mtmd_default_marker, mtmd_context_params_default, mtmd_init_from_file, mtmd_helper_bitmap_init_from_buf, mtmd_tokenize, mtmd_helper_eval_chunks, mtmd_bitmap_free, mtmd_input_chunks_init, and the mtmd::input_chunks C++ wrapper.
• The mtmd library target exists and is wired via add_subdirectory(mtmd); its PUBLIC include dir (examples/mtmd) and the nlohmann/json.hpp it pulls (via common's vendor include) resolve for the grpc-server target.
• mtmd.h / mtmd-helper.h carry include guards (included both directly and via utils.hpp).
• All gpt_params fields referenced for the mtmd context (mmproj_use_gpu, n_threads_mtmd, flash_attn, image_min_tokens, image_max_tokens) and the LLAMA_FLASH_ATTN_TYPE_* enum exist in the fork.
• Patch 0001 still applies cleanly; patch 0002's target is gone and its fix is unneeded in the mtmd path.
• No dangling references to myclip, clip_*, llava_*, or the removed slot fields remain.

What still needs the build / runtime:

• First-try compile of the grpc-server.cpp edits against ik_llama's exact struct layouts (no C++ build was run).
• Link of mtmd under the static-gRPC (BUILD_SHARED_LIBS=OFF) build, and each per-microarch variant.
• The riskiest part is the prompt-processing rewrite: the [img-N] → <__media__> marker translation and the mtmd_helper_eval_chunks n_past/seq_id bookkeeping vs. the surrounding slot state. This is behavioral, not mechanical, and is unverified until an image-model e2e run.

Fixes #10534

Signed commits

• Yes, I signed my commits.