Skip to main content

Replication and Lab Limits

Jason Hreha· Updated February 1, 2026

TLDR: Treat lab effects as hypotheses, and use field validation to decide. Prefer effects with strong real-world support. Beware of headline claims without denominators and windows.

Replication context #

  • Social/behavioral sciences show mixed replication rates; many lab findings fail under natural contexts and incentives.

  • Some classic paradigms have failed high-powered, preregistered multilab replication (e.g., induced compliance). BS-0051

  • Some highly cited “behavioral ethics” interventions have been retracted and should not be treated as strategy-grade evidence. BS-0053

  • Some canonical narratives are being re-evaluated via archival work; treat “classic” texts as sources to be checked, not as proofs. BS-0052

  • Not all subfields are equally fragile: preregistered replications of personality-outcome links show comparatively high reproducibility (with expected shrinkage). BS-0047

  • Loss aversion is context-dependent: robust for large losses; broad “losses loom larger” claims are contested.
  • Publication bias and p-hacking inflate apparent effects; practitioners must validate locally.

Practitioner guidance #

  • Start from behaviors and contexts, not a catalog of “effects.”
  • Run small field pilots with Δ-B and TTFB instrumentation before scale.
  • Treat “defaults,” “priming,” and “magic words” with skepticism unless proven in your segment/context.
  • Use Evidence Ledger entries to ground claims and limit overreach (see BS-0003, BS-0004).

Why lab → field often breaks #

  • Context shifts: stakes, social setting, timing, and device all change behavior.
  • Interaction effects: multiple components interact (ability × motivation × environment); single-factor lab manipulations rarely hold alone.
  • Measurement drift: proxies (clicks, toggles) do not equal target behaviors.

What travels well #

  • Fit-first method: validate problem seeking, then behavior feasibility/desirability.
  • Friction removal: TTFB reductions for the same behavior in the same context.
  • Clear value loops: immediate reinforcement for first successful behavior.

See also #