openai api compatibility: ChatCompletionContentPartText array

[edk4971]

LocalAI version:
deb430f

Environment, CPU architecture, OS, and Version:
docker on ubuntu-server, running inside proxmox vm

Describe the bug
I'm using mealie's openai tool to parse recipes. it passes a content array in the user prompt, which results in:
rpc error: code = InvalidArgument desc = unsupported content[].type

openai api spec allows for an array of ChatCompletionContentPartText.

To Reproduce
[ { "role": "system", "content": "Parse ingredient strings into components. You will receive a list of one or more ingredients. It is critical that you return the same number of ingredients as given and in the same order.\n\nWhen parsing:\n- If uncertain about quantity, unit, or food, put the entire string in the note field\n- Respect grammar rules for multiple languages\n- Interpret singular/plural/grammatical variations\n- Text in parentheses = notes (e.g., \"3 potatoes (roughly chopped)\" → note: \"roughly chopped\")\n- Correct common typos (e.g., \"potatos\" → \"potatoes\")\n- Recognize units: tablespoon, teaspoon, gram, tsp, tbsp, oz, sprig, can, bundle, bunch, unit, cube, package, pinch\n- For ranges (e.g., \"3-5\", \"1 to 2\"), use the lower number\n- All text must appear somewhere, or otherwise be accounted for; if converting \"2 dozen\" → \"24\", don't put \"dozen\" elsewhere. If you're unsure, put extra text in the notes\n\n\n###\nBelow is a list of units found in the units database. While parsing, you should reference this list when determining which part of the input is the unit. You may find a unit in the input that does not exist in this list. This should not prevent you from parsing that text as a unit.\n---\n\n[\"servings\",\"g\",\"clove\",\"bunches\",\"dashes\",\"quart\",\"stalk\",\"tablespoons\",\"splash\",\"gram\",\"cloves\",\"milligram\",\"cup\",\"pinch\",\"bunch\",\"gal\",\"tbsp\",\"fluid ounce\",\"ounces\",\"tsp\",\"ml\",\"quarts\",\"liters\",\"oz\",\"sprig\",\"cups\",\"l\",\"gallon\",\"gallons\",\"milligrams\",\"fl oz\",\"mg\",\"teaspoons\",\"c\",\"pt\",\"liter\",\"pint\",\"pounds\",\"grams\",\"pack\",\"kilograms\",\"lbs\",\"head\",\"pints\",\"packs\",\"pound\",\"milliliters\",\"heads\",\"qt\",\"kg\",\"teaspoon\",\"milliliter\",\"fluid ounces\",\"cans\",\"kilogram\",\"serving\",\"dash\",\"ounce\",\"can\",\"pinches\",\"sprigs\",\"lb\",\"tablespoon\",\"splashes\"]\n" }, { "role": "user", "content": [ { "text": "[\"1/4 cup brown sugar, packed\",\"1/4 cup reduced sodium soy sauce\",\"2 teaspoons sesame oil\",\"1/2 teaspoon crushed red-pepper flakes, or more to taste\",\"1/4 teaspoon ground ginger\",\"1 tablespoon vegetable oil\",\"3 cloves garlic, minced\",\"1 pound ground beef\",\"2 green onions, thinly sliced\",\"1/4 teaspoon sesame seeds\",\"1 cup brown rice\"]", "type": "text" } ] } ]

Expected behavior
LLM returns a parsed array of ingredients

Logs

Additional context

[edk4971]

AI recommended fix (I have zero go talent so I have no idea if this actually works):

in: core/http/endpoints/openai/types/message_item.go

package types

import (
	"encoding/json"
	"fmt"
	"strings"
)

// Add or update the Custom Unmarshaler for the Message content
func (m *Message) UnmarshalJSON(data []byte) error {
	type Alias Message
	aux := &struct {
		Content json.RawMessage `json:"content"`
		*Alias
	}{
		Alias: (*Alias)(m),
	}

	if err := json.Unmarshal(data, &aux); err != nil {
		return err
	}

	// 1. Try parsing as a standard string first
	var stringContent string
	if err := json.Unmarshal(aux.Content, &stringContent); err == nil {
		m.Content = stringContent
		return nil
	}

	// 2. If it fails, try parsing as an array of Content Parts (Multimodal Schema)
	var arrayContent []struct {
		Type string `json:"type"`
		Text string `json:"text,omitempty"`
		// (Image URL struct fields would go here, omitted for text-flattening brevity)
	}

	if err := json.Unmarshal(aux.Content, &arrayContent); err == nil {
		var flattenedText strings.Builder
		for _, part := range arrayContent {
			// Flatten text parts safely
			if part.Type == "text" {
				flattenedText.WriteString(part.Text)
				flattenedText.WriteString("\n")
			}
			// If backend is strictly text-only, you can either ignore "image_url" 
			// or return an explicit error here if an image is actually passed.
		}
		m.Content = strings.TrimSpace(flattenedText.String())
		return nil
	}

	return fmt.Errorf("unsupported content format: must be string or array of parts")
}