Update, 2026-07-03: Evaluating your own AI system's output — the limitation described below — is now supported via
sourceType: "external_submission"and a dedicated submission endpoint. See Evaluate your own AI's output for the full writeup, or the AI Output Evaluation docs for the field reference.
A pipeline can return "manipulated, 92% confidence" with an explanation that's technically correct and still nearly useless to the editor who has to act on it — too jargon-heavy, missing the specific evidence that actually matters, or burying the one detail that would change the decision. Confidence scores measure the model's internal certainty. They say nothing about whether a human reading the explanation would actually understand or trust it. AI Result Evaluation asks a crowd panel that exact question, for any completed verification or Language Technology run.
Requesting an evaluation is a single POST against a completed run — sourceType, sourceId, and optionally how many responses you want. There's no separate task template to build: the endpoint auto-generates a disposable SurveyJS template showing the run's pipeline, verdict, and explanation, wraps it in a crowd job, and dispatches it. Behind the scenes this reuses the exact same crowd job engine that powers every other survey on the platform — evaluation requests aren't a special case architecturally, just a specific auto-generated survey.
Reviewers rate each result on four dimensions, one to five each: clarity (is the explanation easy to understand), evidence sufficiency (is the reasoning backed by something specific), actionability (is it clear what to do with this result), and bias risk (how likely is the result to be skewed or unbalanced). We built a dedicated SurveyJS widget for this rather than reusing a generic rating component, because bias risk in particular needed its own framing — a high score there means high risk, which is the opposite direction from the other three dimensions, and that asymmetry needs to be obvious in the UI or reviewers rate it backwards.
Those four scores combine into a single transparency score per response: clarity, evidence sufficiency, and actionability count positively, bias risk counts negatively. Individual responses aggregate into the batch's overall score once enough responses arrive — three by default, configurable per request.
Every rating writes to an append-only log keyed by the specific AI component it evaluated — the durable signal behind actually improving a pipeline's prompts or model choice over time. We deliberately didn't build automatic retraining or prompt-tuning on top of this; a reviewer can flag a result for retraining and leave free-text notes, but acting on that signal is a decision for your team, or ours, to make manually. An automated feedback loop that silently rewrites prompts based on crowd ratings is a much bigger claim than we're ready to make, and probably a bigger claim than anyone should make without a human looking at what's actually changing.
The data model identifies what's being evaluated by a sourceType and sourceId pair rather than assuming it always comes from a Crowdee pipeline. Today only our own verification and Language Technology runs are wired up as valid sources — evaluating a customer's own AI system's outputs isn't supported yet, since sourceId has to resolve to an existing internal run. But the schema was built this way specifically so that adding an external-submission source later is a new enum value and a new lookup branch, not a rewrite.
45 credits by default — 15 credits per crowd response, three responses minimum. Requesting more responses for a higher-confidence aggregate costs proportionally more. See the AI Output Evaluation documentation for the full request API and rating breakdown.
Share this article:
