Replication and Lab Limits

TLDR: Treat lab effects as hypotheses, and use field validation to decide. Prefer effects with strong real‑world support. Beware of headline claims without denominators and windows.

Replication context

• Social/behavioral sciences show mixed replication rates; many lab findings fail under natural contexts and incentives.
• Loss aversion is context‑dependent: robust for large losses; broad “losses loom larger” claims are contested.
• Publication bias and p‑hacking inflate apparent effects; practitioners must validate locally.

Practitioner guidance

• Start from behaviors and contexts, not a catalog of “effects.”
• Run small field pilots with Δ‑B and TTFB instrumentation before scale.
• Treat “defaults,” “priming,” and “magic words” with skepticism unless proven in your segment/context.
• Use Evidence Ledger entries to ground claims and limit overreach (see BS‑0003, BS‑0004).

Why lab → field often breaks

• Context shifts: stakes, social setting, timing, and device all change behavior.
• Interaction effects: multiple components interact (ability × motivation × environment); single‑factor lab manipulations rarely hold alone.
• Measurement drift: proxies (clicks, toggles) do not equal target behaviors.

What travels well

• Fit‑first method: validate problem seeking, then behavior feasibility/desirability.
• Friction removal: TTFB reductions for the same behavior in the same context.
• Clear value loops: immediate reinforcement for first successful behavior.

---

See also: Why Nudges Fail, Nudge Limitations, Measurement Standards.