llms.txt is a proposed convention for providing a structured, plain-text summary of a website or API, intended to be consumed by large language models and AI-powered developer tools. The concept is analogous to robots.txt for web crawlers: a well-known path that a tool can fetch to understand what a site offers, without scraping or parsing complex formats.
For APIs specifically, llms.txt describes the available endpoints, their purpose, the key fields they accept and return, and any authentication requirements — written in a way that reads naturally in an LLM's context window. We publish one for the Crowdee API. It's public and requires no authentication, because its entire purpose is to be fetched by tools that may not yet have credentials.
A full API reference is the right format for precise, machine-readable contracts. It handles every edge case: nullable fields, discriminated unions, enum constraints, nested arrays of objects. Code generators rely on this precision. But that expressiveness comes at a cost: a realistic API reference is tens of thousands of tokens. Feeding the entire reference to an LLM every time it needs to make an API call is expensive and often unnecessary — most tasks only need a fraction of the total surface area.
There is also a readability gap. A formal API spec is written for tools, not for text prediction. An LLM can parse and understand it, but its output quality improves when the documentation is written in the kind of clear, structured prose that resembles its training data. A llms.txt file bridges that gap: it is precise enough to generate correct code, but structured as natural language rather than a nested schema.
The two formats are not in competition. The full reference — available at docs.crowdee.ai — is the source of truth for tooling; llms.txt is the fast-path for LLMs that need context without the full contract.
Our llms.txt covers the complete Crowdee API surface in a condensed form. Each section corresponds to a feature area: authentication, projects, datasets, verification pipelines, LT pipelines, crowdsourcing, and worker routes. For each area, the file lists the available operations, a one-sentence description of what each one does, the key inputs it accepts, and the shape of what it returns.
The file also covers authentication: it explains the supported methods and how to use each one. A developer or an AI assistant reading the file top to bottom has everything needed to write a working integration for the most common workflows — creating a project, uploading files, running a pipeline, and reading results.
We keep the llms.txt synchronized with the API manually, reviewing it whenever we add or change functionality. This is a deliberate choice: unlike a fully auto-generated reference, the llms.txt benefits from the editorial judgment that goes into deciding which details matter most to a model working from limited context.
The most direct use is fetching the file as context before asking an LLM to write integration code. In any tool that supports context injection — Claude, Cursor, Copilot, or similar — a developer can point the model at Crowdee's llms.txt as a context source and then ask questions about the API or request code generation. The model receives a complete, accurate API summary without requiring the developer to copy-paste from documentation pages.
The same file is useful for agentic workflows. An agent that needs to call the Crowdee API can fetch it as a first step and use the content to plan which operations to call and in what order. This is complementary to the MCP server: the MCP server provides callable tools, while llms.txt provides the context that helps a model decide which tools to call and how to use them effectively. We cover the MCP server in detail in our post on the Crowdee MCP server.
We think of our documentation as serving a hierarchy of consumer needs:
| Format | Consumer | Use case |
|---|---|---|
| Full API reference | Tooling (code generators, linters, SDK generators) | Precise, complete contract |
| Interactive docs at docs.crowdee.ai | Human developers | Exploration and testing |
llms.txt | LLMs and AI coding assistants | Context-efficient API summary |
None of these replaces the others. A developer building a client library will use the full reference. A developer debugging an unfamiliar endpoint will open the interactive docs. An AI assistant generating integration code will fetch llms.txt. All three derive from the same underlying API definitions, so they stay consistent without manual synchronization. Together they ensure that whether the consumer is a code generator, a human, or a language model, it gets the format that works best for it.
Share this article:
