AI generatedThe Data Platform is the module of Crowdee where you create, clean, extend, and manage large multi-modal datasets. It is the one place you upload files, run them through the cleaning and language-tech pipeline catalog, deduplicate, and version the result — so the cleaned dataset that runs today is reproducible, every time, from the exact inputs you started with.
The Data Platform sits between the Crowd Platform, where workers generate labels, and the AI Platform, where a dataset becomes training data for a finetuned model. It is the layer that takes raw files — your own archives, a content-gathering output, a third-party dump — and turns them into a structured asset that downstream pipelines can consume. Every downstream surface on Crowdee that needs a real dataset reads from the Data Platform, not from a one-off file upload.
This is different from the verification products. Verification is a delivered outcome: you bring a piece of content, we send back a verdict. The Data Platform is the durable substrate that makes recurring verification, recurring labelling, and recurring model training possible at all. If your need is "I have a pile of files and I want them clean, deduplicated, versioned, and ready to feed into a job that runs every week," the Data Platform is the surface that was built for that.
A dataset is a named, versioned collection of items. You create a dataset, upload a first batch of files — image, audio, video, text, or document — and the platform stores them. Every upload, every cleaning run, every pipeline output creates a new version of the dataset. Versions are append-only: you never lose a previous state, even when the next run supersedes it.
Each item carries an immutable hash and a parent-version reference. If you upload the same file twice (or it appears in a later batch), the platform detects it. If a cleaning run recomputes a derived version, the new version still links back to the inputs that produced it. This is the same model that sits behind reproducible ML pipelines in research and production: you can always answer the question "what did this dataset contain three months ago, and what ran on it then?" by walking the version graph.
Versions are the unit of consumption. When the Crowd Platform runs a job, it points at a specific version of an input dataset. When the AI Platform pulls training data, it points at a specific version. When you re-run a verification pipeline against last quarter's data, you point at last quarter's version. The version reference is what makes the whole platform auditable end-to-end.
Every file in a dataset can be processed through the pipeline catalog. The catalog is organised into two families. Cleaning pipelines do data hygiene: silence trimming on audio, PII redaction on text, format conversion between document formats, and similar transformations that turn a raw upload into a usable file. Language-tech pipelines do extraction: Whisper-based transcription of audio, vision-model OCR of scanned documents, language identification, named-entity detection, and LLM-based translation.
A pipeline run takes a dataset version and produces a new dataset version. Trimming audio produces a version with shorter duration and the same item count. Transcribing audio produces a version where each item now has a transcript column. Translating text produces a version where the source text is paired with a target-language version. Each output is itself a real, queryable dataset, not a side file attached to the original.
Pipelines can be chained. A run that trims audio of a podcast, transcribes it, identifies the language, and extracts entity mentions is a normal pipeline composition, not a special case. The catalog exposes the building blocks; the per-project run configuration picks the order. This is the same primitive operationally as an ETL pipeline, but the storage layer is the Data Platform's versioned dataset, not a generic blob store.
Two items in a dataset that look the same should not be paid for twice. The Data Platform hashes every item on upload and on every pipeline output, and surfaces near-duplicates and exact-duplicates as a review queue. You can configure automatic deduplication (drop items whose hash matches an existing one) or human-reviewed deduplication (promote them to a reviewer queue that asks the operator to decide).
The review queue is the same dashboard primitive you use elsewhere on Crowdee: it shows pending items, the operator accepts or rejects, and the result is persisted. A weekly review of pending duplicates takes the same shape as a weekly review of pending crowd answers. The reason this matters: deduplication is the single biggest cost-control lever in a labelling program, and the operator UI should make it as easy as accepting a crowd answer.
Quality flags are surfaced alongside duplicates. Items that failed a pipeline run, items below a configured length or duration threshold, items whose MIME type doesn't match the expected one — all of these appear in the review queue the same way. The operator decides what to keep; the platform keeps an audit trail of every decision.
A finished dataset is consumed by every downstream surface on Crowdee. A versioned dataset is the input data for a Crowd Platform job — the worker panel is fed the items from the version you point at, and the answers are stitched back into a new version on completion. A versioned dataset is the training data for an AI Platform finetune run — the same hashes, the same version reference, the same audit trail. A versioned dataset is the file source for a verification pipeline run — the same content, the same provenance, the same reproducible re-run when the question comes back.
The Data Platform is the answer to "where do my files live, and how do I know what was done to them." Files in, cleaned, deduplicated, versioned, versioned again, and now consumable by the rest of Crowdee. The same platforms that handle the verification work and the finetuning work pull their inputs from here, so the dataset you curate this month is the dataset you can re-run on next quarter — and the audit trail follows it.
To see what a finished dataset looks like from the operator side, take a look at the Data Platform overview. To discuss how the dataset fits your pipeline, book a demo call — we'll walk through your actual data and the pipelines you'd run on it.
