AI Result Evaluation answers "is this specific result any good," on demand. It doesn't answer "has our model quietly gotten worse since last month," because nobody's going to remember to ask that question at the right moment for every pipeline running in production. Continuous Monitoring is that same evaluation, scheduled instead of requested — the thing that actually catches quality regression before a customer does.
This is a small detail with an outsized amount of engineering behind it: nothing on the Crowdee platform had ever run on a recurring schedule before. Every existing job — verification runs, crowd tasks, enrichment — is triggered once and finishes. Building monitoring meant introducing recurring scheduling for the first time, registering a cron-pattern tick per monitoring schedule (hourly, daily, or weekly) and tearing it down cleanly when a schedule is paused or deleted. It's a small surface area, but it's new infrastructure, not a variation on something we already had.
A schedule's tick looks at completed runs in scope — organization- or project-wide, matching the schedule's source type — from the preceding period that it hasn't already sampled, checking against its own history so the same run never gets evaluated twice by the same schedule. If nothing matches, the tick records an empty period and moves on; an empty period isn't a failure, it's just quiet. If something matches, it dispatches a normal AI Result Evaluation batch against the most recent eligible run, using the exact same path a manual evaluation request would use. Once the crowd panel reaches quorum, the aggregate transparency score gets written back to the monitoring run.
We deliberately didn't ship a fixed quality threshold you configure once and forget. A "score below 70 is bad" rule doesn't generalize — a pipeline that normally scores around 60 isn't drifting at 58, and one that normally scores 95 is absolutely drifting at 80. Instead, each scored run gets compared against that specific schedule's own trailing average of prior completed runs, and gets flagged when it deviates meaningfully from its own history. The first run a schedule ever samples has no baseline to compare against, so it's never flagged — drift is inherently a second-run-onward signal, and that's fine, because you can't detect a change from a single data point anyway.
Here's the part we didn't smooth over: right now, every tick samples exactly one run, even though the schedule configuration already accepts a sampling strategy (all, a random percentage, or a fixed count per period) and a sample count. Those fields are stored and validated, but only the single most recent eligible run is ever pulled per tick. We chose to ship the scheduling and drift-detection infrastructure correctly rather than delay it until multi-sample support was done, and we'd rather say that plainly in the docs than let you discover it by noticing your sample counts don't match what you configured. If you need tighter coverage today, a shorter cadence — hourly instead of daily — is the workaround until multi-sample lands.
An empty tick is free. A tick that dispatches an evaluation costs the same as a manual AI Result Evaluation — 45 credits by default — since it's the exact same underlying operation, just triggered on a timer instead of a request. See the Continuous Monitoring documentation for schedule fields and the full drift-detection formula.
Share this article:
