[FE feat] Mustache support

api/ee/tests/manual/evaluations/sdk/test_loop.py → api/ee/tests/manual/evaluations/sdk/test_sdk_evaluation.py

@@ -89,7 +89,7 @@ async def my_random_evaluator(capital: str):
     description="Use an LLM to judge if the previous answer is correct",
     #
     correct_answer_key="capital",
-    model="openai/gpt-4o-mini",
+    model="gpt-5.4-mini",
     prompt_template=[
         {
             "role": "system",

api/oss/src/resources/evaluators/evaluators.py

@@ -69,7 +69,8 @@
                         },
                         "strict": True,
                     },
-                    "version": "4",
+                    "version": "5",
+                    "template_format": "mustache",
                 },
             },
             {
@@ -114,7 +115,8 @@
                         },
                         "strict": True,
                     },
-                    "version": "4",
+                    "version": "5",
+                    "template_format": "mustache",
                 },
             },
             {
@@ -158,7 +160,8 @@
                         },
                         "strict": True,
                     },
-                    "version": "4",
+                    "version": "5",
+                    "template_format": "mustache",
                 },
             },
             {
@@ -203,7 +206,8 @@
                         },
                         "strict": True,
                     },
-                    "version": "4",
+                    "version": "5",
+                    "template_format": "mustache",
                 },
             },
         ],
@@ -284,10 +288,17 @@
                 "x-ag-ui-advanced": False,
                 "description": "Select a response format to structure how your evaluation results are returned.",
             },
+            "template_format": {
+                "label": "Template Format",
+                "type": "hidden",
+                "default": "mustache",
+                "description": "The template format used to render the prompt",  # ignored by the FE
+                "x-ag-ui-advanced": False,
+            },
             "version": {
                 "label": "Version",
                 "type": "hidden",
-                "default": "4",
+                "default": "5",
                 "description": "The version of the evaluator",  # ignore by the FE
                 "x-ag-ui-advanced": False,
             },

api/pyproject.toml

@@ -16,7 +16,7 @@ dependencies = [
     "httpx>=0.28,<0.29",
     "structlog>=25,<26",
     "python-dotenv>=1,<2",
-    "python-multipart>=0.0.29,<0.0.30",
+    "python-multipart>=0.0.30,<0.0.31",
     "cryptography>=48,<49",
     "sqlalchemy>=2,<3",
     "sqlalchemy-utils>=0.42,<0.43",

docs/design/prompt-runtime-unification/README.md

@@ -47,9 +47,7 @@ The current system makes those answers difficult because runtime behavior and fr
 All three services should share the same building blocks but currently don't:
 
 - **Provider/model resolution.** Chat and completion use workflow provider settings; the judge manually extracts a fixed provider-key set and therefore cannot reliably use custom or self-hosted models configured in the UI.
-- **Rendering.** Each service has different rendering behavior:
-  - `PromptTemplate.format` raises on Jinja errors; `_format_with_template` returns the original content with a warning.
-  - Chat and completion recursively render `llm_config.response_format`. The judge builds `response_format` from `response_type` / `json_schema` and does not render variables inside `json_schema`.
+- **Rendering.** The services now share more renderer infrastructure than before, but they still differ in configuration shape and runtime entrypoint. Chat and completion render from `parameters.prompt`, while the judge still uses its flat evaluator config and separate handler path.
 - **Config.** The judge does not allow configuring `temperature`. It currently sends a hard-coded `temperature=0.01`, which some models reject as an unsupported optional parameter.
 - **Output.** Completion, chat, and judge return different output shapes. Shared runtime code should stop before handler-specific output normalization.
 
@@ -97,7 +95,7 @@ Example.
 {"profile": {"name": "Ada"}}
 ```
 
-`profile` is a JSON object. The runtime can access `{{profile.name}}`, `{{profile.tags.0}}`, or `{{profile}}` (renders as compact JSON text).
+`profile` is a JSON object. The runtime can access `{{profile.name}}`, `{{$.profile.tags[0]}}`, or `{{profile}}` (renders as compact JSON text).
 
 ```json
 {"profile": "{\"name\":\"Ada\"}"}
@@ -123,7 +121,7 @@ Example.
 
 Three substitution formats plus one full templating engine:
 
-- **`mustache`** — `{{variable}}` substitution. **The default for new apps.** `{{a.b}}` always means nested access (`a` then property `b`). Supports JSONPath (`{{$.profile.name}}`) and JSON Pointer (`{{/profile/name}}`). JSON objects and arrays render as compact JSON text when inserted as whole values into a string template. We use the name "mustache" for recognizability; we do not implement the full mustache spec (no sections or partials) — just variable substitution with our path enhancements.
+- **`mustache`** — Mustache rendering for new apps. **The default for new apps.** Tags that start with `{{$` are resolved as JSONPath expressions against the render context and substituted into the rendered output last, as inert data (never re-parsed) — they are shielded from the Mustache engine, not fed back through it. Ordinary Mustache behavior handles plain tags such as `{{name}}`, dotted names such as `{{profile.name}}`, and standard Mustache sections. Partials are not supported in Agenta runtime and must fail clearly if present. JSON objects and arrays render as compact JSON text when inserted as whole values into a string template.
 
 - **`curly`** — `{{variable}}` substitution. **Deprecated.** Same syntax as `mustache`, but the resolver applies literal-key-first lookup: if a top-level key is literally named `foo.bar`, `{{foo.bar}}` returns that key's value before nested traversal is attempted. This is what keeps old apps that have variables with literal dots in their names working. Not surfaced in the playground for new apps; existing apps keep their declared format.
 
@@ -269,13 +267,9 @@ Builds on WP-B1.
 
 Builds on WP-B1.
 
-- Add `mustache` as a new template-format option in the runtime. Semantics: `{{variable}}` substitution; `{{a.b}}` is nested access only (no literal-key-first); JSONPath and JSON Pointer supported.
-- `mustache` becomes the runtime default for new apps. Existing apps continue to use the format they declared.
-- **Brace escaping.** `mustache` ships with an explicit escape mechanism so users can include literal `{{` / `}}` in prompts (e.g., few-shot examples that show LLM output formatting). `curly` and `fstring` keep their current escape semantics:
-  - `fstring` already escapes via `{{` → `{` and `}}` → `}` (Python `str.format` rule).
-  - `jinja2` already supports `{% raw %}…{% endraw %}` blocks.
-  - `curly` has no escape today. Adding one to `curly` is non-trivial because the existing `\{\{\s*(.*?)\s*\}\}` regex captures the inner braces of `{{{{x}}}}` as part of the variable name. We document the gap in the rendering appendix and recommend `jinja2` for prompts that need literal braces; the cleanest place to land a real fix is `mustache`, which is greenfield. Whether to also retrofit an escape into `curly` is an open question for WP-B3 (decision factor: how many existing apps need literal `{{` in their prompts).
-- See [appendix-rendering-edge-cases.md](appendix-rendering-edge-cases.md) for the current escape behavior of every mode and the expected behavior for `mustache`.
+- Add `mustache` as a new template-format option in the runtime. Semantics: tags that start with `{{$` are resolved as JSONPath expressions and substituted as inert data after rendering (never re-parsed); all other tags render through a Mustache engine. Partials are unsupported and fail clearly.
+- `mustache` becomes the default rendering format for newly created apps / prompt configs. Existing apps continue to use the format they declared.
+- See [appendix-rendering-edge-cases.md](appendix-rendering-edge-cases.md) for the current escape behavior of every mode.
 
 ### Frontend
 
@@ -292,8 +286,8 @@ The same UX pattern lives in playground inputs, the testset editor, observabilit
 Depends on WP-B3 (`mustache` available) and WP-F1 (type switching available).
 
 - Send testcase data as native JSON when the stored type is JSON. No `JSON.stringify` in request construction.
-- In `mustache` mode, `{{a.b}}` is treated as a single variable `a` (object) with `.b` as a property accessor. The playground's variable discovery creates one variable `a` of type object that the user fills with JSON.
-- `curly` mode is hidden from new-app creation flows. Existing apps still on `curly` continue to work — the literal-key-first behavior is preserved for them.
+- In `mustache` mode, `{{$.a.b}}` is the explicit JSONPath escape hatch. Plain Mustache tags such as `{{a}}` or `{{profile.name}}` follow ordinary Mustache rendering behavior. The playground's variable discovery for nested JSON is still a frontend concern.
+- The frontend honors the WP-B3 `mustache` default and hides `curly` from the format list unless the current old app already has `curly` selected. Existing apps still on `curly` continue to work — the literal-key-first behavior is preserved for them.
 
 #### WP-F3 — Variable discovery (autocomplete)
 
@@ -306,19 +300,18 @@ A nicety on top of WP-F2.
 
 #### WP-D1 — Documentation and SDK examples
 
-- Publish prompt templating docs covering `mustache` (default for new apps), `curly` (legacy compat), `fstring`, and `jinja2`. Spell out the dot-semantics distinction between `mustache` and `curly`.
+- Publish prompt templating docs covering `mustache` (default for new apps), `curly` (legacy compat), `fstring`, and `jinja2`. Spell out the JSONPath shield-and-substitute rule for `mustache` and the literal-key-first distinction of `curly`.
 - Publish the variable matrix by service and interface.
 - Add SDK / local examples for completion, chat, and LLM-as-a-judge.
-- Document escaping rules, JSON vs. stringified JSON, JSONPath, and JSON Pointer.
+- Document escaping rules, JSON vs. stringified JSON, and JSONPath. Document JSON Pointer only for legacy `curly` semantics if it remains supported there.
 
 Reference template examples:
 
 ```text
 Hello {{name}}
 Profile JSON: {{profile}}
 Profile name: {{profile.name}}
-First tag: {{profile.tags.0}}
-JSON Pointer: {{/profile/name}}
+First tag: {{$.profile.tags[0]}}
 JSONPath: {{$.profile.name}}
 ```
 
@@ -344,16 +337,17 @@ The work packages have a natural order:
 2. **WP-B2** stands up message and JSON-return rendering on the helper, and aligns Jinja error behavior.
 3. **WP-B3** adds `mustache` as a new format.
 4. **WP-F1** lands the same JSON/string switching pattern in playground / testset / observability / evaluation.
-5. **WP-F2** flips playground execution to send native JSON and switches new apps to `mustache`. Curly stays for legacy apps only.
+5. **WP-F2** flips playground execution to send native JSON and hides `curly` from new-app format selection. Curly stays visible for legacy apps that already use it.
 6. **WP-F3** adds autocomplete on top.
 7. **WP-D1** publishes the templating docs, variable matrix, and SDK examples.
 
 ## Test Plan
 
 ### Backend runtime
 
-- The low-level rendering helper resolves top-level, nested, JSONPath, and JSON Pointer references in `mustache` and `curly` modes.
-- `curly` preserves literal-key-first behavior on dotted keys; `mustache` does not (treats them as nested).
+- The low-level rendering helper keeps current `curly` behavior and adds `mustache` support.
+- In `mustache` mode, only tags that start with `{{$` are resolved through JSONPath (shielded from the engine and substituted last as inert data); other tags render through the Mustache engine.
+- `curly` preserves literal-key-first behavior on dotted keys.
 - Whole-object insertion renders as compact JSON text in both modes.
 - `fstring` and `jinja2` behavior is preserved.
 - Stringified JSON inputs are treated as strings (no auto-parse).
@@ -383,7 +377,7 @@ The work packages have a natural order:
 
 ### Documentation
 
-- Examples cover string, JSON object, array, nested access, JSONPath, JSON Pointer, messages, and brace escaping in each format.
+- Examples cover string, JSON object, array, JSONPath, Mustache dotted names, messages, and brace escaping in each format.
 - Variable matrix covers completion, chat, judge, and the four interfaces (Direct API/SDK, evaluator playground, evaluation service offline, evaluation service online).
 
 ## Future Directions
@@ -428,6 +422,7 @@ Each work package gets its own subfolder with research, plan, implementation not
 
 - [`wp-b1-runtime-foundation/`](wp-b1-runtime-foundation/README.md) — judge backend patch (provider/secret resolution + temperature removal) and the low-level rendering helper extraction.
 - [`wp-b2-rendering-unification/`](wp-b2-rendering-unification/README.md) - shared message rendering, JSON-return rendering, judge `json_schema` rendering, and Jinja error alignment.
+- [`wp-b3-mustache-rendering/`](wp-b3-mustache-rendering/README.md) - `mustache` runtime format, `{{$...}}` JSONPath shield-and-substitute resolution, `mystace` rendering, and clear partial failures.
 
 Subfolders for the remaining work packages will be added as each is picked up.