docs(api): evaluators endpoint docstrings + concept page - #4178
Conversation
Adds the Evaluators concept page under the REST API guide and annotates every evaluators endpoint and request/response model with docstrings plus Field(description=...). - New: docs/docs/reference/api-guide/07-evaluators.mdx - Adds handler docstrings to EvaluatorsRouter and SimpleEvaluatorsRouter (catalog, artifact, variant, revision, deploy, resolve, simple CRUD, templates) - Adds Field descriptions to every request/response schema in api/oss/src/apis/fastapi/evaluators/models.py No behavior changes; no openapi_extra; no marketing words. Cross-links to Versioning and Query Pattern guides.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Resolves Support-mixin conflicts in api/oss/src/apis/fastapi/evaluators/models.py by keeping the docstrings and Field(description=...) descriptions while adopting the umbrella's Support base class on 16 response envelopes. Reverts auto-generated docs/docs/reference/api/ changes per write-docs §8. The auto-regen is handled separately, downstream of the openapi.json update.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
| sidebar_position: 7 | ||
| --- | ||
|
|
||
| An **evaluator** is a versioned resource that scores a single tuple of inputs, outputs, and (optionally) ground truth. Evaluations attach one or more evaluators to a set of testcases or traces and run them against application revisions; each score lands in the evaluation's result table. |
There was a problem hiding this comment.
I mean that is too dry and wrong. the evaluator does not care about ground truth, it just takes testcase which could include. I'd start with a high level thing.
It's a runnable workflow that is versioned and is used to score the outputs of other workflows and traces.
Evaluators can be run in the playground, in evaluation, online evaluation, or called as they are.
When called in the playound, evaluation, online evaluation, the system give them as input etc..
When called evaluator create traces with type annotation (provide attribute name). These traces are linked to the .. .
|
|
||
| An **evaluator** is a versioned resource that scores a single tuple of inputs, outputs, and (optionally) ground truth. Evaluations attach one or more evaluators to a set of testcases or traces and run them against application revisions; each score lands in the evaluation's result table. | ||
|
|
||
| Evaluators share the artifact / variant / revision model used by applications, workflows, and testsets. See [Versioning](/reference/api-guide/versioning) for commit semantics, `include_archived`, and how revision IDs stay stable. The rest of this page covers only what is specific to evaluators. |
There was a problem hiding this comment.
evaluators are a type of workflow btw
|
|
||
| Evaluators share the artifact / variant / revision model used by applications, workflows, and testsets. See [Versioning](/reference/api-guide/versioning) for commit semantics, `include_archived`, and how revision IDs stay stable. The rest of this page covers only what is specific to evaluators. | ||
|
|
||
| ## Scoring contract |
There was a problem hiding this comment.
Why is this section called scoring contracts while it's talking about the body of the evaluator
|
|
||
| ## Scoring contract | ||
|
|
||
| An evaluator revision commits three things: |
There was a problem hiding this comment.
a better way to say it the evaluator body or data being something and that something is commited as a revision
|
|
||
| | Field | What it describes | | ||
| |-------|------------------| | ||
| | `data.uri` | The handler to invoke — either a built-in (`agenta:builtin:auto_exact_match:v0`) or a custom HTTPS URL for an LLM-as-judge or code-based evaluator. | |
There was a problem hiding this comment.
explain the handler that contains the business logic. Can be either .. not simply either
| |-------|------------------| | ||
| | `data.uri` | The handler to invoke — either a built-in (`agenta:builtin:auto_exact_match:v0`) or a custom HTTPS URL for an LLM-as-judge or code-based evaluator. | | ||
| | `data.schemas` | JSON Schemas for `parameters` (configuration), `inputs` (what the evaluator reads), and `outputs` (what it returns). | | ||
| | `data.parameters` | The configured values for the parameters described in the schema. | |
There was a problem hiding this comment.
not clear put described in the schema maybe under ()
| | `data.schemas` | JSON Schemas for `parameters` (configuration), `inputs` (what the evaluator reads), and `outputs` (what it returns). | | ||
| | `data.parameters` | The configured values for the parameters described in the schema. | | ||
|
|
||
| When the evaluator runs, the service at `data.url` receives the invocation payload (inputs, optional ground truth, and the pinned `parameters`) and returns an object that matches `schemas.outputs`. Outputs are typically a single numeric score or a boolean correctness flag, but a schema with multiple fields is allowed. The built-in `auto_exact_match` evaluator, for example, returns `{"success": true | false}`. |
There was a problem hiding this comment.
this should eb under a subsection. maybe interface ? or how it runs smth like that . then the decsribtion should be easier to understand
An evaluator is invoked by sending invocation paulot ()[there is not ground truth, please check the interface, it's testcase, trace...] to the handler in data.url which can be a url or ...
The evlauator return a dictionary of feedback which can be score, boolean, strings, arrays..
For instance.
Other than returning , the evaluator write a trace that has as inputs, as outputs
|
|
||
| ## Presets and the catalog | ||
|
|
||
| The catalog ships with built-in **templates** that describe what an evaluator can do (key, JSON schemas, handler URI). A **preset** is a named set of parameter values pre-filled against a template — for example, the "Quality Rating" preset against the `feedback` template. |
There was a problem hiding this comment.
a catalog contains the list of built-in evaluators described as template each having the schema, the URI, keys.
Anbd then the descripotuiojn
|
|
||
| ## Simple vs structured endpoints | ||
|
|
||
| The `/simple/evaluators/` surface collapses the artifact, variant, and latest revision into one flat record (including the handler URL and merged schemas). Use it when you want the "current evaluator" without tracking lineage. Use `/evaluators/`, `/evaluators/variants/`, and `/evaluators/revisions/` when you need to commit, fork, or retrieve a specific revision. See [Simple Endpoints](/reference/api-guide/simple-endpoints) for the general pattern. |
There was a problem hiding this comment.
uncelear. simplify abd show examples
|
|
||
| ## Relationship to evaluations | ||
|
|
||
| Evaluators are what [evaluations](/reference/api-guide/evaluations) attach to. An evaluation pins one or more `evaluator_revision_id`s at the moment the run is configured; committing a new revision on the variant does not retroactively change the pinned run. Use the deployment endpoint below to swap the "latest" revision for an environment without creating a new run. |
There was a problem hiding this comment.
That is not how you explain it.
You say evaluations are configured with evaluators which are run against the test set . Each evaluation has a field or more evaluation that . This means that evaluation are linked to a specific evaluator version, if a new revision is committed does not retroactively change.
why are you mentioning the deployment endpont, does not mention
|
|
||
| Evaluators are what [evaluations](/reference/api-guide/evaluations) attach to. An evaluation pins one or more `evaluator_revision_id`s at the moment the run is configured; committing a new revision on the variant does not retroactively change the pinned run. Use the deployment endpoint below to swap the "latest" revision for an environment without creating a new run. | ||
|
|
||
| ## Deployment |
|
|
||
| `POST /evaluators/revisions/deploy` pins an evaluator revision into an environment revision under a named key. Consumers that dereference the key through the environment see the deployed revision; historical traces remain pinned to their original revision IDs. | ||
|
|
||
| ## Worked example |
There was a problem hiding this comment.
say example, why worked example
| Evaluators, variants, and revisions are soft-deleted. Use `POST /evaluators/{id}/archive` and `POST /evaluators/{id}/unarchive` to flip `deleted_at`; the same pattern applies at the variant and revision level. Archiving an evaluator hides it from `/query` responses unless the request sets `include_archived: true`. See [Versioning](/reference/api-guide/versioning#archive-and-unarchive). | ||
|
|
||
| :::note | ||
| Evaluator artifacts and workflow artifacts share the underlying persistence layer. An evaluator's `id` also serves as its `workflow_id`, which is why revision payloads include both keys. Treat them as the same UUID; the workflow aliases exist for interoperability with the generic workflow endpoints. |
There was a problem hiding this comment.
This probably would go in the beginning whn you explain tha tevaluators are just workflwos wrapper / aliasaround them. and what it means (simply) as in same uuid
Address all 14 review comments on PR #4178. Restructure around the user's mental model: evaluators are a kind of workflow that scores other workflows and traces, runnable in four contexts, and producing annotation-type traces. Key changes: - Lead the intro with the high-level definition and the workflow-alias fact (same UUID), instead of burying it in a footnote. - Drop the "ground truth" framing. The handler receives inputs sourced from a testcase or trace, plus pinned parameters. - Rename "Scoring contract" -> "Body" (it describes data, not a contract). - Split out "How it runs" as its own section: invocation payload, return shape (numeric / boolean / string / array), annotation trace write. - Reframe the catalog as a list of templates with named fields, and present presets as one field of a template, not a separate entity in the prose. - Drop the deployment section per review. - Reframe "Relationship to evaluations": evaluations pin specific revision IDs; new commits do not retroactively change pinned runs. - Rename "Worked example" -> "Example". - Style pass per Williams: characters as subjects, concrete verbs, no nominalizations in subject position, no em dashes.
dosubot
Bot
added
size:XL
ddb0d1b
into
docs/age-3734-api-reference-epic
Railway Preview Environment
|
1 participant
Summary
docs/docs/reference/api-guide/07-evaluators.mdx(~600 words). Covers what an evaluator is, the scoring contract (handler URI + JSON schemas + parameters), presets (metadata, not entities), catalog endpoints, relation to evaluations (pinned revision IDs), simple vs structured, deployment, and a worked example (create from builtin + commit + retrieve).api/oss/src/apis/fastapi/evaluators/router.py(bothEvaluatorsRouterandSimpleEvaluatorsRouter, plus catalog and templates).Field(description=...)on every request/response field inapi/oss/src/apis/fastapi/evaluators/models.py.openapi_extra, no invented fields.Endpoint coverage
30 endpoints documented across:
Verified against
https://eu.cloud.agenta.aiwith the provided API key: catalog types, catalog templates (list + fetch), catalog presets (list), evaluators/query, evaluators/{id}, revisions/query, revisions/retrieve, revisions/log, revisions/resolve, variants/query, simple/evaluators/query, simple/evaluators/templates, POST simple/evaluators/, POST revisions/commit, archive + unarchive. Shapes match what the docs describe.Related
tmp-docs-analysis/plan.md(PR-4 in the plan).tmp-docs-analysis/concept-map/runnables.md(customer docs use "evaluator"; the "runnable" term does not appear).Test plan
uvx ruff format api/oss/src/apis/fastapi/evaluators/— 1 file reformatted, rest unchanged.uvx ruff check api/oss/src/apis/fastapi/evaluators/— passes.ast.parseon edited files — OK.Notes for reviewer
--no-verifybecausepre-commitis not installed in the sandbox. No secrets were touched; hooks will run on merge. Ruff was run manually in place of the hook.fork_evaluator_variantreturns an empty envelope for both payload shapes I tried ({evaluator_variant_id}and{evaluator_variant: {src: {id}}}). The handler has a pre-existing# TODO: FIX MEmarker. Not filing a new Linear issue; leaving the docstring generic until the fork surface is finalized. Worth a look as a follow-up.edit_*docstrings note that renaming is temporarily disabled — this is enforced in the router (RENAME_EVALUATORS_DISABLED_MESSAGE).workflow_id = evaluator_idcontinuity is called out in a note at the bottom of the concept page.Draft intentionally. Please do not mark ready until reviewed.