Aircraft are tested in instrumented rigs long before a pilot flies one. Digital creative mostly skips the wind tunnel — it goes from generation straight to paid traffic. CreativeGate is the wind tunnel: a simulation-first evaluation funnel that scores any creative artifact before money moves, plus the discipline that keeps a wind tunnel honest — continuous calibration of every simulated measurement against real-world outcomes.
One evaluation, wire by wire
what actually happens between "Run gate" and a verdict on this dashboard
01client → servicePOST /evaluate with text or an uploaded artifact reference. A job row is written (queued) and the id returned immediately — the caller polls /jobs/{id} or registers a webhook.
02service → engineA background worker marks the job running, loads the evaluation profile (declarative YAML: which rungs, thresholds, fusion policy, seed) and the named ground-truth set, and builds the funnel.
03engine → rungsRungs execute cheapest-first, each receiving a context (config, ground truth, seed, provider slots, calibration lookup). A failed threshold eliminates the artifact — more expensive rungs never run, and the exact failing evidence rides on the verdict.
04rungs → harnessEvery scoring rung's raw prediction is recorded against the artifact — this is what makes future recalibration possible. Validity is stamped onto each result by the framework from calibration records; rungs cannot self-assert trustworthiness.
05engine → fusionSurviving scores fuse with weights = each rung's measured correlation for this segment/modality. The confidence band widens with ensemble disagreement, calibration weakness (1−r²), and thin coverage. If nothing is calibrated, the verdict says DIRECTIONAL ONLY — loudly.
06engine → verdictThe verdict stores score ± band, the full evidence ledger, per-rung validity statements, flags (including HUMAN_REVIEW_REQUIRED when a known-good anchor scores below the floor), the cheapest-next-test recommendation, runtime, config hash and versions.
07service → clientsJob flips to done; webhook fires if registered. Every open dashboard tab picks the run up on its next poll — no reload.
⟲reality → harnessDays or weeks later, real outcomes land via POST /ground-truth. The harness re-scores what every cheap rung had predicted for those artifacts, updates per-cell correlations, raises drift alarms, and adjusts fusion weights — with an append-only audit trail. The system learns how much to trust itself.
The integration contract — JSON over HTTPS
three calls: submit → collect → report outcomes · every field means something
POST /evaluate — submit an artifact
One JSON body. Binary creatives (.png/.jpg) go through POST /artifacts first (multipart upload), then reference the returned id here instead of text.
{
"artifact_id": "summer-hero-01", // optional — auto-generated if omitted. Use YOUR id: real
// outcomes are joined back by this key later
"modality": "text", // "text" | "image"
"text": "Meet your new favorite running shoes. Free shipping. Shop now.",
"segment": "gen-z-us", // audience segment — scopes which calibration records apply;
// trust earned on one segment never transfers silently
"platform": "meta", // free-form platform tag
"profile": "banner-web-v1", // which criteria set judges this artifact class
// (gate rules, thresholds, rubric, outcome corpus). Default if omitted
"metadata": {"campaign": "q3-launch"}, // free-form passthrough, stored with the verdict
"webhook_url": "https://you.example/cb" // optional — POSTed {job_id, status, verdict_id} on completion
}
// → immediately returns {"job_id": "job-a1b2c3", "status": "queued"}
GET /jobs/{job_id} → /verdict/{verdict_id} — collect the verdict
Poll the job (evaluations take ~0.3–2s; states are a durable machine: queued → running → done | error | interrupted) or wait for the webhook, then fetch the verdict — the output contract:
{
"id": "v-7fc716df", "artifact_id": "summer-hero-01",
"eliminated": false, // true ⇒ a rung killed it; more expensive rungs never ran
"eliminated_by": null, // which rung, e.g. "deterministic_gate" — evidence says exactly why
"score": 0.842, // calibration-WEIGHTED fusion, 0..1 — weights are measured
// correlations with real outcomes, never opinions
"band": [0.46, 1.0], // the honesty interval — wide = thin or weakly-validated evidence.
// Gate launches on the LOWER edge, not the point score
"confidence_note": "Calibration-weighted fusion of performance_predictor
(weight 0.98), judge_ensemble (weight 0.42)...", // or "DIRECTIONAL ONLY" when nothing is calibrated
"flags": [], // e.g. HUMAN_REVIEW_REQUIRED (known-good ad scored below floor),
// degraded_provider (keyless comparator served)
"validity_summary": "performance_predictor: Validated for segment='gen-z-us',
modality='text': Spearman r=0.91 vs ctr (n=144)...", // the verdict citing its credentials —
// and volunteering where it has none
"next_test": { // what limits confidence + the cheapest way to fix it
"kind": "human_pretest", "estimated_cost_usd": 400, "rationale": "..." },
"rung_results": [ /* every rung: score, passed, confidence, validity, evidence */ ],
"evidence": [ /* the full ledger: every check, comparison, prediction — nothing unexplained */ ],
"config_hash": "41fc701c0c06c031", // exact criteria that produced this number — replayable
"seed": 7, "runtime_ms": 701
}
- How a machine reads it, in four rules:
eliminated → reject, evidence says why · compare band[0] (not score) against your launch bar · HUMAN_REVIEW_REQUIRED in flags → route to a person, never auto-decide · DIRECTIONAL ONLY in the note → advisory signal, don't gate on it.
POST /ground-truth — report reality (the call most integrators forget)
{"records": [{"artifact_ref": "summer-hero-01", // the id you evaluated with — this is the join key
"modality": "text", "segment": "gen-z-us",
"outcomes": {"ctr": 0.043}}]} // what actually happened when it ran
// → recalibrates every affected rung: correlations re-measured, drift alarms raised,
// fusion weights adjusted. This call is what keeps every future verdict honest.
How agents and automated systems will talk to it — MCP PLANNED · v0.2
MCP is not a second pipeline — it wraps the same engine as native tools so agent frameworks (Claude agents, LangGraph nodes, n8n) call the gate without HTTP glue. An autonomous creative pipeline replaces its internal critic with these five tools; the REST contract above does the identical job today.
T1evaluate_artifacttext or artifact ref + profile + segment → verdict summary in one round-trip (eliminated, score, band, flags, next_test) — no polling; agents want a single call.
T2get_verdictfull document — evidence ledger and validity statements — when the agent must explain or defend a decision.
T3report_outcomeartifact ref + metric + value → recalibration summary. The agent closes the trust loop itself.
T4get_rung_validity"how much should I trust you?" — the calibration report as a tool, so an agent can check credentials before acting.
T5plan_live_test(v0.3) funnel survivors + budget constraints → experiment spec with funnel scores as bandit priors.
⟲the agent loopgenerate variants → evaluate_artifact each → eliminated ⇒ regenerate using the evidence as repair instructions ("banned phrase 'guaranteed results'") · survivor ⇒ rank by band-lower-edge · flags ⇒ escalate to a human → launch top-N through the agent's own platform tooling (never ours) → days later report_outcome per variant → the gate is sharper on the agent's next run.
Roadmap
wire, don't stub — a few rungs working end-to-end with real calibration beat all rungs half-built
v0.1SHIPPEDThe vertical slice: stable schemas, funnel engine, three rungs end-to-end, calibration harness verified against a synthetic world with known physics, CLI, API, this dashboard. Hardened: durable jobs, token auth, artifact uploads, model-cache persistence, live polling.
v0.2NEXTMCP server (the engine as native agent tools); attention/saliency rung for images with a keyless contrast-and-center fallback; event-driven recalibration; verdict replay from stored config hashes.
v0.3PLANNEDLive-Test Planner: bandit specs seeded with funnel scores as priors, sequential designs with SRM sanity gates, micro-flight/geo-split designs. Off-policy evaluation over propensity logs. Synthetic persona panel with "directional-only until calibrated" enforcement.
v1.0PLANNEDVideo modality (frame-sampled); proxy-metric ladder validated per segment; multi-tenant profiles; Postgres; hardening.
Deliberately out of scope, forever: serving traffic, managing spend, an ad server, user management beyond a token, a heavyweight frontend. The funnel plans and scores; it never moves money.