field season open · q2 2026
Survey 047 · Quarter 2 / 2026

Evaluate modelson your own ground.Publish the report.

Stratum is a layered evaluation platform for production AI. Sample your traffic, score it on your rubric, and produce a benchmark you can hand to a regulator — or to next quarter's you.

Begin a surveyRead the method
Reproducible·Sealed reports·Drift watch·Air-gap ready
survey · 047 / customer-support
SEALED
model
qual
cost
latency
opus-4.7
0.94
$0.084
1.4s
sonnet-4.6
0.91
$0.012
612ms
haiku-4
0.84
$0.0008
182ms
gpt-5
0.93
$0.071
1.6s
llama-4-70b
0.86
$0.0002
240ms
240 examples · 5 models · seed 4127↓ report.pdf
Helvetia Cartographic▪Field Notes Co.▪Linnea Bakery▪Foundry 47▪Northwind Labs▪Substrate▪Marlow Workwear▪Mercury OS▪Plover Studio▪Verge Editions▪Helvetia Cartographic▪Field Notes Co.▪Linnea Bakery▪Foundry 47▪Northwind Labs▪Substrate▪Marlow Workwear▪Mercury OS▪Plover Studio▪Verge Editions▪
Section II — Apparatus

Four layers. One survey.

Sampling
billing question412
refund request218
feature ask184
complaint96
praise47

Stratified samples of live traffic. The bench reflects what your users actually ask, weighted by frequency.

Scoring
rubric · v 3.4
accuracy human-graded · 240
tone model-graded · 1k
brevity deterministic
safety classifier · 0.94
inter-rater κ = 0.84
Drift
alert
Survey log
#047
customer-support · q2 2026
sealed
04 / 24
#046
code-review · monthly drift
sealed
04 / 17
#045
summarisation · regulator audit
sealed
04 / 09
#044
intent-classify · weekly
draft
04 / 02
Sealed report
survey 047 · sealed 04/24
opus-4.7
recommended default
qual
0.94
cost
$0.084
lat
1.4s
Section III — In the field

Three teams. Three surveys.

Choose a model, watch for drift, satisfy an auditor. Three jobs. One apparatus, one method, one report.

I
Pre-launch

Choose a model on your own work, not the launch chart.

Stratum runs the candidate models against your evaluation set, in your environment, on your data. The leaderboard is yours, scoped to your problem.

0
% of teams change their default after one survey
II
Drift watch

Production quality, sampled and surfaced weekly.

We sample live traffic, replay through alternative models, and chart the drift. Regression alerts you before your customer support queue does.

0
× faster regression detection
III
Audit

A reproducible benchmark you can hand to a regulator.

Each survey produces a sealed report with seeds, prompts, model versions, and scores. The same report runs again next quarter. The numbers move only when the data does.

0
% of audits closed in a single review
Section IV — Comparison

Marketing benchmarks are not your benchmark.

Capability
Stratum
Public leaderboard
Eval set source
Your traffic, sampled
Public benchmark
Reproducibility
Sealed report w/ seeds
Spreadsheet
Drift detection
Weekly replay + alert
Customer complaint
Cost surfacing
Per-question, per-model
Aggregate
Comparison axis
Quality · cost · latency
Quality only
field reports

“Stratum told us our 'best' model was costing twelve cents more per query for a 0.4 quality bump no user could see. We switched the same week.”

Maja Nilsen
Head of Mapping, Helvetia Cartographic
0
surveys published
0%
reproducibility
0%
cut from model bills
0×
drift detection
Section V — Subscriptions

Pay for surveys. Not for opinions.

Field
$0free for solo

For independent researchers and side projects.

  • 1,000 evaluations / mo
  • 3 models
  • Public surveys
  • Community Discord
Begin a survey
Survey
$124per seat / month

For teams choosing models with evidence.

  • 120k evaluations / mo
  • Unlimited models
  • Drift watch + alerts
  • Sealed reports
  • API + webhook
  • Email + chat support
Start a 14-day survey
Atlas
—annual contract

For institutions with audit obligations.

  • Unlimited evaluations
  • Air-gapped runner
  • Region-locked storage
  • Regulator-ready exports
  • Survey methodologist
  • 99.94% SLA
Talk to us
Section VI — Method notes

Direct answers to recurring questions.

You bring your rubric. We run model-graded scoring with a calibration set you label yourself, and produce inter-rater agreement against a held-out human-labelled batch. Numbers come with confidence intervals.

From the field

all entries
Apr 24
method
Inter-rater agreement now exported alongside every quality score.
Apr 13
drift
Drift watch alerts thread Slack and Linear, with the offending sample attached.
Apr 02
atlas
Air-gapped runner ships as a single Helm chart for on-prem clusters.
Mar 19
report
Sealed report PDFs now include a deterministic re-run command.

Survey today. Decide tomorrow.

One thousand evaluations on the house. Sealed report at the end. Bring your own data.

Begin a survey Talk to methodologist