Skip to content

Add ordered content parts to UserMessage - #6729

Open
dmitriierokhin wants to merge 7 commits into
spring-projects:mainfrom
dmitriierokhin:pr/user-message-content-parts
Open

Add ordered content parts to UserMessage#6729
dmitriierokhin wants to merge 7 commits into
spring-projects:mainfrom
dmitriierokhin:pr/user-message-content-parts

Conversation

@dmitriierokhin

Copy link
Copy Markdown

Problem

UserMessage holds a single text plus a List<Media> with no ordering relation between
them, so every provider converter can only ever serialize a user turn as all text, then all
media
.

Some multimodal prompts depend on a finer ordering. The case that led me here is a per-page
document prompt: each page contributes a marker, its OCR text, and its page image, and the
image must directly follow that page's own text so the model associates the two. Flattened to
text-then-media, the page↔image association is gone. There is currently no way to express this
through the core message API, even for providers whose wire format is itself an ordered list of
parts.

Solution

Add ContentPart (spring-ai-commons, org.springframework.ai.content) — a sealed
TextPart | MediaPart — and let a UserMessage carry an ordered List<ContentPart>:

UserMessage.builder()
    .contentParts(
        ContentPart.text("--- page 1 ---"), ContentPart.media(page1Image),
        ContentPart.text("--- page 2 ---"), ContentPart.media(page2Image))
    .build();

The design point that keeps this additive: text and media remain faithful projections of
the parts, and vice versa.
They are never a second source of truth.

  • A message built the flat way exposes getContentParts() as the derived [text, media…]
    sequence, so providers need only one code path.
  • A message built from parts exposes getText() (text parts joined with \n) and getMedia()
    (media parts in order).
  • Because the projections stay faithful, providers that cannot express ordering keep working
    with no change at all, and token counting still sees one source of truth. I deliberately did
    not add anything to MediaContent, so JTokkitTokenCountEstimator cannot double-count.
  • AbstractMessage's non-null-text invariant is untouched: a media-only message projects to
    "".

hasInterleavedContent() reports whether flattening would lose information (i.e. anything other
than one leading text followed by media). Providers with a flat text-plus-media shortcut keep
taking it while that is false, so existing requests are byte-identical; only genuinely ordered
content takes the parts path. This is what stops the change rewriting the wire format for every
existing Anthropic and OpenAI user message.

Provider adoption

Module Native shape Change
spring-ai-google-genai List<Part> parts walk; mediaToPart extracted for reuse
spring-ai-anthropic List<ContentBlockParam> parts walk; cache control moved to the last text block
spring-ai-openai ChatCompletionContentPart[] toContentPart(Media) extracted verbatim from the inline lambda, then parts walk
spring-ai-bedrock-converse List<ContentBlock> parts walk
spring-ai-mistral-ai sealed ContentChunk parts walk, hoisted above the non-null-text assertion
spring-ai-ollama flat content + images no code change; comment recording that parts are flattened
spring-ai-deepseek text only no change

Two pre-existing issues were fixed incidentally, because walking the parts replaced the code
that contained them:

  • Bedrock Converse called ContentBlock.fromText(userMessage.getText()) unconditionally, so a
    media-only message produced an empty text block, which Bedrock rejects.
  • Mistral asserted non-null text before looking at media, so a message whose content begins with
    media threw.

Backwards compatibility

Every existing UserMessage constructor, Builder method, getText(), getMedia() contents,
toString(), equals() and hashCode() keep their exact signatures and semantics. For a
message built the flat way getText() returns the identical string, including empty and
whitespace-only values, and getMedia() returns the identical contents in the identical order.
copy()/mutate() round-trip faithfully. No provider requires a change. Nothing is
deprecated.

Three behaviours do change, all intentional and covered by tests:

  1. getMedia() now returns an unmodifiable view. Mutating the returned list was never a
    supported contract, and it would desynchronize the list from the content parts. One test in
    spring-ai-mistral-ai did this (userMessage.getMedia().add(...)) and is updated to use the
    builder.
  2. Builder.text(...)/Builder.media(...) discard previously set content parts. This is
    enforced eagerly in each setter rather than validated in build(), deliberately: components
    that rewrite user text (ChatModelCallAdvisor, StructuredOutputValidationAdvisor,
    Prompt.augmentUserMessage) all do mutate().text(...), and throwing there would break the
    whole advisor chain the moment one interleaved message reached it. Flattening loses ordering
    but keeps all content, which is exactly today's behaviour. Builder.appendText(String) is
    added as the structure-preserving alternative for those callers to migrate to; I left the
    migration itself out of this PR.
  3. Media-only user messages become representable, and their getText() is "". Components
    that treat user text as a query would see an empty query for such a message.

Known limitation

UserMessage still does not override equals/hashCode, so neither media nor part ordering
participates in equality — unchanged from today, and pinned by a test with a comment. Including
parts in equality now would give identity semantics, because Media has no equals/hashCode;
that would make messages round-tripped through the Redis and Neo4j chat-memory repositories stop
comparing equal to their originals. The follow-up is value-based equals on Media first.

Deliberately out of scope

ChatClient's fluent surface. DefaultChatClientUtils only materializes a UserMessage when
StringUtils.hasText(processedUserText), so a blocks-only spec would be silently dropped, and
DefaultPromptUserSpec holds media as a flat list. Threading parts through
PromptUserSpecDefaultChatClientRequestSpecDefaultChatClientUtils is a separate
design question (including how param() rendering should apply per part), and
ChatClientRequestSpec.messages(Message...) already accepts a pre-built UserMessage in the
meantime. Happy to follow up if you'd like it in the same change.

Tests

  • ContentPartTests (new) — construction, null rejection, blank text allowed, record equality
    (and that MediaPart equality is necessarily identity-based given Media has no equals).
  • UserMessageTestsall pre-existing tests pass unmodified, which is the main regression
    signal. Added: projection in both directions, the \n join separator, no media marker leaking
    into the text projection, builder channel exclusivity in both orders, copy()/mutate()
    fidelity including that mutate().text(...) leaves no stale part carrying the old text,
    appendText in both modes, hasInterleavedContent() truth table, immutability of both
    returned lists, and equality semantics pinned as-is.
  • CreateGeminiRequestTests — interleaved parts asserted index-by-index on the resulting
    GeminiRequest (one Content, N parts, alternating text()/fileData().fileUri()), parts
    alongside a SystemMessage, and a case pinning the legacy flat path unchanged.

./mvnw clean package passes locally.

Notes

Happy to split this into a core-only PR plus per-provider follow-ups if that is easier to
review, or to reshape the API — the naming (ContentPart/TextPart/MediaPart) avoids
collisions with the Anthropic and Bedrock SDKs' own ContentBlock/TextBlock types, but I have
no attachment to it.

This was developed with the help of an AI coding agent; I have reviewed every line and am
accountable for it, per CONTRIBUTING.md. The per-commit Co-authored-by trailers record that.

Dmitrii Erokhin and others added 7 commits August 3, 2026 16:38
A UserMessage pairs a single text with a collection of media and
defines no ordering between them, so providers can only serialize it
as text followed by all media. Prompts that need media positioned
relative to text are not representable: a per-page document prompt
where each page's image must directly follow that page's own text
loses the association the model relies on.

Introduce ContentPart, a sealed TextPart/MediaPart pair, and let a
UserMessage carry an ordered list of them. The existing text and
media accessors become faithful projections of the parts and vice
versa, so a message built either way reads correctly through either
accessor, providers that cannot express ordering keep working
unchanged, and token counting still sees a single source of truth.

Builder.text and Builder.media discard any previously set parts
eagerly. That flattens ordered content rather than leaving a stale
projection behind, so components which rewrite user text keep
working; Builder.appendText is added as the structure-preserving
alternative for them to migrate to.

hasInterleavedContent() reports whether flattening would lose
information, letting providers with a flat text-plus-media shortcut
keep taking it for content that is flat-equivalent.

getMedia() now returns an unmodifiable view, since mutating it would
desynchronize it from the content parts.

Signed-off-by: Dmitrii Erokhin <dmitrii.erokhin@webbfontaine.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Gemini's Content carries an ordered Part list, which maps one-to-one
onto a user message's content parts, so an interleaved prompt
survives verbatim.

The media mapping is extracted into mediaToPart so a single
MediaPart can reuse it. For a message built in the flat form the
parts are the text followed by the media, so this produces the same
Part list as before.

Signed-off-by: Dmitrii Erokhin <dmitrii.erokhin@webbfontaine.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Anthropic's content is an ordered ContentBlockParam list, so a user
message's content parts map onto it directly.

The block-building branch is now also entered when flattening the
content would lose information, otherwise a message carrying several
text parts would fall into the flat fast path and be collapsed into
one block. Messages whose content is flat-equivalent still take that
fast path, so their requests are unchanged.

Cache control moves from the first text block to the last one: a
cache prefix has to cover the whole content, and with several text
blocks the first no longer marks its end.

Signed-off-by: Dmitrii Erokhin <dmitrii.erokhin@webbfontaine.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Bedrock's Converse content is an ordered ContentBlock list, so a
user message's content parts map onto it directly.

Walking the parts also stops an empty text block being sent for a
message whose content is media only, which the previous
unconditional ContentBlock.fromText produced and which Bedrock
rejects.

Signed-off-by: Dmitrii Erokhin <dmitrii.erokhin@webbfontaine.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Mistral's content is an ordered ContentChunk list, so a user
message's content parts map onto it directly.

The parts are handled before the assertion that the message text is
non-null, since a message whose content starts with media has no
leading text to assert on. mapToImageUrlChunks is removed, its only
caller being the stream-concat this replaces.

Signed-off-by: Dmitrii Erokhin <dmitrii.erokhin@webbfontaine.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
OpenAI's content is an ordered ChatCompletionContentPart array, so a
user message's content parts map onto it directly.

The media mapping is extracted verbatim from the inline lambda into
toContentPart, returning null for media it cannot represent so the
caller skips it exactly as the lambda did. The content-parts array
is now also built when flattening the content would lose
information; a message with no media whose content is
flat-equivalent still takes the plain string path, and system
messages are untouched.

Signed-off-by: Dmitrii Erokhin <dmitrii.erokhin@webbfontaine.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Ollama's message format is a text plus a separate image list and
cannot express an ordering between them, so ordered content parts
are flattened to text-then-images by the projections. Record that at
the conversion site; no behaviour changes.

Signed-off-by: Dmitrii Erokhin <dmitrii.erokhin@webbfontaine.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants