Training data is where 60 to 80 percent of an LLM project's actual cost lives, but procurement in the category is closer to consulting than to software. Ten vendors split across four lanes, none of them with transparent pricing for the work that matters, and a 4-to-8-week sales cycle on every enterprise contract. This is the buyer-side procurement guide: how the lanes split, which vendor wins which lane, and the questions to ask sales before the pilot.
Key takeaways
- Four lanes — High-end RLHF (Scale AI, Surge AI, Invisible Tech), large-scale workforce (Appen, Toloka, Sama), platform-led (Labelbox, iMerit), and synthetic (Snorkel AI, Defined AI). Pick lane before vendor.
- What changes in 2026 — RLHF and post-training labelling have moved from specialty to table-stakes. Vendors without a credible post-training story are losing share fast.
- What buyers underweight — Quality-control model and rework cost. Cheap labels with high rework end up more expensive than premium labels with low rework.
- What buyers overweight — Headline pricing and worker count. Both are easy to inflate; neither correlates strongly with actual delivery quality.
The four lanes
Vendors marketed against each other often serve different jobs. The lane split is the first cut to make; it eliminates two-thirds of the comparison work that buyers waste time on.
1. High-end RLHF and frontier-model labelling
The premium lane. Vendors here serve frontier labs, post-training programmes, and any project where label quality directly drives model quality. Scale AI, Surge AI, and Invisible Tech dominate. The economics are sales-led, the engagements run six figures and up, and the workforce is curated rather than open. If your project will be benchmarked publicly or read by competitors, this lane is the right answer.
2. Large-scale workforce labelling
Generalist annotation at scale. Appen is the veteran; Toloka brings a transparent marketplace model and international coverage; Sama brings ethical workforce practices and African talent base. The cost per label is lower, the workforce is larger, and the QA varies more than in the premium lane. Right for high-volume classical labelling, multilingual data collection, and pre-training corpus work.
3. Platform-led labelling
Tooling-first vendors that let you bring your own workforce, a partner workforce, or a managed workforce within their platform. Labelbox is the canonical example; iMerit blends platform with managed in-house workforce. Right when you have annotator capacity in-house or via a BPO partner, and you need the workflow tooling, QA frameworks, and integration with your training pipeline.
4. Synthetic and programmatic data
The lane that moved fastest in 2024 and 2025. Snorkel AI is the most established; Defined AI specialises in speech and voice. The promise is to generate labelled data programmatically (through weak supervision, foundation-model-graded labelling, or full synthetic generation) at a fraction of the manual cost. As of 2026, the lane is production-primary for narrow domains (image and text classification, code generation post-training) and still experimental for the harder cases (complex RLHF on subjective tasks).
RLHF is now table-stakes
Through 2023 and 2024, RLHF was a specialty offering that separated the leaders from the long tail. By 2026 it has become table-stakes for any vendor serious about LLM work. The structural advantage is held by the vendors who built infrastructure for it before 2024: Scale, Surge, Invisible Tech, and to a lesser degree iMerit on regulated-domain RLHF. Vendors who pivoted in late, or who treat RLHF as a special-engagement add-on rather than a core product, lose competitive deals fast.
Two operational implications. First, ask vendors specifically about multi-rater adjudication and expert escalation paths for ambiguous cases; both are markers of mature RLHF infrastructure. Second, ask for case-study evidence on the specific kind of RLHF you need (preference data on reasoning tasks looks nothing like preference data on creative writing; vendors strong at one are not automatically strong at the other).
Scored comparison
The scoring rubric: lane positioning, strongest capability, worker selection model, built-in QA depth, RLHF preference data maturity, constitutional AI and red-teaming capability, pricing transparency, and minimum engagement size. Eight axes across ten vendors.
| Feature | Scale AI | Surge AI | Appen | Toloka | Labelbox | iMerit | Sama | Invisible Tech | Defined AI | Snorkel AI |
|---|---|---|---|---|---|---|---|---|---|---|
| Lane and positioning | ||||||||||
| Primary lane | High-end RLHF and frontier-model labelling | RLHF specialist; high-end expert annotation | Large-scale workforce, generalist | Large-scale workforce, marketplace model | Platform-led; bring-your-own-workforce | Platform + managed workforce hybrid | Ethical workforce; large-scale generalist | RLHF + complex knowledge work | Speech and voice data specialist | Programmatic and synthetic data platform |
| Strongest at | RLHF, autonomous vehicles, complex multimodal | RLHF preference data, expert raters | Scale on classical labelling tasks | International workforce coverage | Tooling and quality workflows | Domain-specific labelling (medical, finance) | Ethical workforce, East African roots, North American + European delivery centres | Expert-only RLHF, very complex tasks | Multilingual speech datasets | Programmatic data generation; weak supervision |
| Quality model | ||||||||||
| Worker selection model | Vetted pool + experts via Outlier/SEAL | Curated expert network | Large global workforce | Open marketplace with tiering | Customer-managed or partner-supplied | Vetted in-house pool | Vetted, salaried workforce | Highly vetted expert network | Native speakers, vetted | Not applicable (programmatic) |
| Built-in QA | Multi-pass, automated + human review | Multi-rater + expert adjudication | Standard multi-pass | Per-tier; varies | Configurable workflows | Strong in regulated domains | Strong managed QA | Highest-touch QA in the category | Domain-specialist QA | Programmatic + model-graded QA |
| RLHF and post-training | ||||||||||
| RLHF preference data | Mature offering | Core product | Available; less specialised | Available | Via partner workforce | Specialist domain RLHF | Available | Core product | Not the focus | Programmatic preference synthesis |
| Constitutional AI / red-teaming | Yes | Yes | Custom engagements | Custom engagements | Not native | Yes for regulated industries | Custom engagements | Yes; expert red-teaming | Out of scope | Via synthetic adversarials |
| Procurement and pricing | ||||||||||
| Pricing transparency | Sales-led; varies by engagement | Sales-led | Sales-led | Public marketplace pricing | Public tiers + sales-led for scale | Sales-led | Sales-led | Sales-led | Sales-led | Platform tiers + custom |
| Minimum engagement | Enterprise | Mid-market to enterprise | Mid-market to enterprise | Self-serve from small batches | Self-serve to enterprise | Mid-market | Mid-market to enterprise | Enterprise | Mid-market | Self-serve to enterprise |
The verdict by lane
Same data, organised by lane and recommendation. The right answer almost always involves more than one vendor; the four lanes solve different problems.
Recommended for high-end RLHF and frontier-model work
- Scale AI. The default for any frontier-lab-adjacent project. Outlier and SEAL workforce, mature RLHF pipeline, the deepest customer evidence in the category. Tax: enterprise-only engagement, sales cycle measured in months, pricing only disclosed under NDA.
- Surge AI. The strongest pure-RLHF specialist. Curated expert network, multi-rater adjudication, and a quality ceiling that competes with Scale at lower headline cost. Tax: less generalist; if you need both RLHF and classical labelling, Scale or Invisible fit better.
- Invisible Tech. The highest-touch option in the category. Expert-only network for the genuinely hard tasks: complex reasoning RLHF, expert-domain red-teaming. Tax: priced accordingly; not the right vendor for routine work.
Recommended for large-scale workforce labelling
- Appen. The veteran. Massive global workforce, broad capability surface, mature processes. Tax: less differentiated than the specialists; project management quality varies by engagement.
- Toloka. Strongest for international and multilingual work, with public marketplace pricing that makes budgeting tractable. Tax: open-marketplace model means QA setup falls more on the buyer.
- Sama. Differentiated on ethical workforce practices. Roots in East Africa with expanded operations in Montreal and Europe. Strong managed-QA and a Fortune-50 enterprise track record. Tax: smaller crowd headcount than Appen or Toloka; rush-project capacity ceiling shows up at the very high end.
Recommended for platform-led and specialised lanes
- Labelbox. The platform pick. Best tooling for teams that want to bring their own workforce or partner workforce, with configurable QA workflows. Tax: you supply the labour or pick a partner; pure-software model leaves judgement to the buyer.
- iMerit. Best for domain-specific labelling, particularly medical and financial services. In-house pool, strong regulated-industry QA. Tax: narrower than Scale or Appen outside the strong domains.
- Snorkel AI. The programmatic-data pick. Weak supervision, synthetic data generation, model-graded QA. Right when the manual-labelling model breaks economically or when synthetic augmentation is the primary supply.
- Defined AI. The speech and voice specialist. Multilingual speech datasets, native-speaker network. Right for any project where audio data quality is the binding constraint.
The five-stage procurement playbook
The mechanics that separate working procurement from the deck-led version most teams settle for.
- Write the evaluation rubric before the first sales call. Specify the task, the languages, the quality bar, the volume, the timeline, and the QA model you expect. Without this, every vendor will anchor the conversation to their strengths and the procurement runs aground inside three weeks.
- Shortlist three vendors per lane. Not five and not one. Three forces you to commit to actual differentiation, and three is the number that gives you negotiating leverage on the eventual production contract.
- Run a paid pilot of identical scope across the shortlist. Paid is the operative word. Free pilots run on the vendor's discretion; paid pilots run on yours. Use a blinded reference set so the result is comparable. Budget 3 to 4 weeks for execution and 1 to 2 weeks for analysis.
- Score against your rubric and the blinded reference. The headline metric is agreement rate. The under-measured metrics are rework percentage and edge-case consistency. The combination of all three predicts production quality better than any single number.
- Negotiate the production contract with the winning vendor, including the off-ramp. SLA, rework terms, capacity commitments, and an explicit off-ramp clause. The off-ramp is the most-skipped step in this category and the one that hurts most when it is missing. You do not want to discover you cannot switch vendors mid-project because of an exclusivity clause buried in the schedule of work.
When to combine vendors
Most production AI programmes end up running two or three vendors in parallel. The combinations that work in practice:
- Scale or Surge for RLHF + Toloka or Appen for classical labelling. The high-end-plus-large-scale combination. Premium spend on the parts that drive quality; workforce-scale spend on the volume work.
- Snorkel AI for synthetic data + a workforce vendor for edge cases. Programmatic primary supply with human annotation for the hard tail. Often cuts overall cost by 40 percent or more on suitable lanes.
- Labelbox for tooling + a BPO or in-house workforce. The platform-led shape. Right when you already have annotator capacity and want to upgrade the tooling rather than the people.
- iMerit for regulated-industry expert work + Sama or Appen for the routine. Domain-specialist plus volume. Common in medical, financial-services, and legal AI programmes.
CTO POV and field evidence
Related reads
Frequently asked questions
What is an AI training data provider?
A company that supplies the labelled data, preference judgements, or synthetic datasets used to train and post-train AI models. The category covers four lanes in 2026: high-end RLHF and expert annotation, large-scale workforce labelling, platform-led tooling that supports customer-owned workforces, and synthetic or programmatic data generation. Most production AI projects use a combination.
How do Scale AI and Surge AI compare?
Both are premium RLHF specialists. Scale is broader, with capabilities across autonomous vehicles, multimodal, and frontier-model work alongside RLHF. Surge is the purer RLHF play, with a curated expert network and a multi-rater quality model that competes head-on with Scale at sometimes lower headline cost. The choice depends on the breadth of the engagement. If you need only RLHF preference data and you want the strongest specialist, Surge is the default. If you need RLHF plus classical labelling plus complex multimodal work, Scale is the easier procurement.
What is RLHF and why does it matter?
Reinforcement Learning from Human Feedback. The post-training technique where humans rate model outputs and the ratings train a reward model that fine-tunes the base model. RLHF is the technique behind ChatGPT, Claude, Gemini, and most production LLM products. Quality of the human preference data is the largest predictor of final-model quality, which is why the vendors in this category command premium pricing. As of 2026, RLHF capability is table-stakes for any provider serious about LLM work; vendors without it are losing the projects that move the market.
How do I evaluate quality before signing a contract?
Run a paid pilot with a fixed, blinded evaluation set. Send the same task to two or three shortlisted vendors; score the returned labels against a known ground truth or against expert review you control. Measure rework rate, agreement with your reference, and the consistency of edge-case handling. The CTO POV essay on prommer.net walks through the evaluation rubric in detail with the specific questions to ask sales and the metrics to capture. Do not skip the paid pilot; sales decks are not predictive of delivery quality.
Is synthetic data ready to replace human labelling?
For narrow domains, yes. For frontier RLHF and complex reasoning preference data, not yet. The 2024 to 2025 progression in programmatic and synthetic data, led by Snorkel AI and a handful of newer specialists, moved several lanes from "experimental" to "production primary supplier." Image classification, text classification, and code-generation post-training are the lanes where synthetic is genuinely competitive on quality and dramatically cheaper. RLHF on subjective reasoning tasks still benefits from human raters; the gap is narrowing but is still real.
Why is pricing so opaque in this category?
Three reasons. Engagements are bespoke (task complexity, language coverage, QA depth all vary widely). Pricing is competitive intelligence that vendors guard. And large customer agreements include volume discounts, prepay rebates, and SLA premiums that change effective per-unit cost by 40 percent or more. Toloka, Labelbox, and Snorkel publish baseline pricing; the rest are sales-led. Plan for a 4-to-8-week procurement cycle on enterprise contracts.
How do I structure the procurement process?
Five stages. Define the evaluation rubric before the first sales call (otherwise vendors anchor the criteria to their strengths). Shortlist three vendors per lane based on capability fit and customer-reference quality. Run a paid pilot of identical scope across the shortlist. Score the results against your rubric and the blinded reference. Negotiate the production contract with the winning vendor, including SLA, rework terms, and an off-ramp clause. The off-ramp is the most-skipped step and the one that hurts most when it is missing.
What changes when you need expert-domain annotators (medical, legal, financial)?
The vendor list narrows. iMerit and Invisible Tech are the strongest for expert-domain work; Scale has it in the catalogue but at premium pricing. Sama and Appen can deliver some expert work but the depth varies. Cost per label rises by 5-to-20x against generalist tasks, which changes the procurement math. The right answer is often a hybrid: synthetic data or programmatic generation for the routine cases, expert annotators for the edge cases that drive model performance.
What is an AI training data provider?
A company that supplies the labelled data, preference judgements, or synthetic datasets used to train and post-train AI models. The category covers four lanes in 2026: high-end RLHF and expert annotation, large-scale workforce labelling, platform-led tooling that supports customer-owned workforces, and synthetic or programmatic data generation. Most production AI projects use a combination.
How do Scale AI and Surge AI compare?
Both are premium RLHF specialists. Scale is broader, with capabilities across autonomous vehicles, multimodal, and frontier-model work alongside RLHF. Surge is the purer RLHF play, with a curated expert network and a multi-rater quality model that competes head-on with Scale at sometimes lower headline cost. The choice depends on the breadth of the engagement. If you need only RLHF preference data and you want the strongest specialist, Surge is the default. If you need RLHF plus classical labelling plus complex multimodal work, Scale is the easier procurement.
What is RLHF and why does it matter?
Reinforcement Learning from Human Feedback. The post-training technique where humans rate model outputs and the ratings train a reward model that fine-tunes the base model. RLHF is the technique behind ChatGPT, Claude, Gemini, and most production LLM products. Quality of the human preference data is the largest predictor of final-model quality, which is why the vendors in this category command premium pricing. As of 2026, RLHF capability is table-stakes for any provider serious about LLM work; vendors without it are losing the projects that move the market.
How do I evaluate quality before signing a contract?
Run a paid pilot with a fixed, blinded evaluation set. Send the same task to two or three shortlisted vendors; score the returned labels against a known ground truth or against expert review you control. Measure rework rate, agreement with your reference, and the consistency of edge-case handling. The CTO POV essay on prommer.net walks through the evaluation rubric in detail with the specific questions to ask sales and the metrics to capture. Do not skip the paid pilot; sales decks are not predictive of delivery quality.
Is synthetic data ready to replace human labelling?
For narrow domains, yes. For frontier RLHF and complex reasoning preference data, not yet. The 2024 to 2025 progression in programmatic and synthetic data, led by Snorkel AI and a handful of newer specialists, moved several lanes from "experimental" to "production primary supplier." Image classification, text classification, and code-generation post-training are the lanes where synthetic is genuinely competitive on quality and dramatically cheaper. RLHF on subjective reasoning tasks still benefits from human raters; the gap is narrowing but is still real.
Why is pricing so opaque in this category?
Three reasons. Engagements are bespoke (task complexity, language coverage, QA depth all vary widely). Pricing is competitive intelligence that vendors guard. And large customer agreements include volume discounts, prepay rebates, and SLA premiums that change effective per-unit cost by 40 percent or more. Toloka, Labelbox, and Snorkel publish baseline pricing; the rest are sales-led. Plan for a 4-to-8-week procurement cycle on enterprise contracts.
How do I structure the procurement process?
Five stages. Define the evaluation rubric before the first sales call (otherwise vendors anchor the criteria to their strengths). Shortlist three vendors per lane based on capability fit and customer-reference quality. Run a paid pilot of identical scope across the shortlist. Score the results against your rubric and the blinded reference. Negotiate the production contract with the winning vendor, including SLA, rework terms, and an off-ramp clause. The off-ramp is the most-skipped step and the one that hurts most when it is missing.
What changes when you need expert-domain annotators (medical, legal, financial)?
The vendor list narrows. iMerit and Invisible Tech are the strongest for expert-domain work; Scale has it in the catalogue but at premium pricing. Sama and Appen can deliver some expert work but the depth varies. Cost per label rises by 5-to-20x against generalist tasks, which changes the procurement math. The right answer is often a hybrid: synthetic data or programmatic generation for the routine cases, expert annotators for the edge cases that drive model performance.
Ready to Find the Right AI Tools?
Browse our data-driven rankings to find the best AI tools for your team.