Quick Sheet
- Main claim: Democracies fall into self-reinforcing policy failure loops; adopting an “experimenting society” (per Donald T. Campbell) can improve outcomes.
- Core supports: Campbell’s 1971 lecture; Campbell’s Law; examples of partisan/populist behavior and centrist timidity; virtues of honesty, humility, iteration.
- What’s strong: Clear framing; well-articulated values (curiosity, rigor, iteration); acknowledges metric gaming and policy churn.
- What’s weak: Limited comparative/quantitative evidence; few concrete implementation steps (funding, governance, ethics); broad political generalizations.
- Net: Persuasive op-ed that motivates experimentation; needs a governance playbook to be operational.
C — Clarify
- Claims
- Democracies exhibit a loop of “unforced errors” → desperate gambles → further errors.
- Campbell’s “experimenting society” (vigorous trials, multidimensional evaluation, willingness to discard failures) offers a better path.
- Campbell’s Law warns that high-stakes metrics distort behavior; therefore evaluation must be rigorous and multi-dimensional.
- Populists dismiss expertise; center-left actors are risk-averse—both impede experimentation.
- Definitions
- Experimenting society: Institutionalized trial-and-learn approach to recurrent policy problems with willingness to pivot.
- Multidimensional evaluation: Blending outcomes (effect sizes, costs), implementation fidelity, equity, spillovers, and unintended consequences.
- Recurrent problems: Persistent issues (education achievement, crime recidivism, congestion) resistant to one-off reforms.
- Scope: Op-ed level analysis; not a systematic review; illustrative examples from UK/US politics.
- Assumptions: (a) RCT-style methods scale to governance; (b) political actors can tolerate humility/uncertainty; (c) evaluation capacity exists or can be built.
O — Organize
| Element | What the article provides | Gaps / what’s missing |
|---|---|---|
| Conceptual foundation | Campbell’s lecture; Campbell’s Law; rationale for experimentation and humility. | Comparative frameworks (e.g., adaptive regulation, prediction markets) and when to use which. |
| Evidence base | Historical references; general observations on politics and policy failures. | Meta-analyses or case series quantifying policy experiments’ ROI; replication records; failure rates. |
| Causal risks | Metric gaming per Campbell’s Law. | Experiment ethics, external validity, spillovers, politicization of control groups, and equity impacts. |
| Implementation | Value list (active, honest, practical); call for nimbleness. | Concrete operating model: budget lines, legal authorities, staffing, procurement, pre-registration, open data. |
| Counter-positions | Notes populism/centrism barriers. | Engagement with critiques of RCT-centrism (context dependence; complexity; dignity of subjects; alternatives to trials). |
D — Discover
- What evidence would most change the conclusion?
- Cross-country panel showing jurisdictions with dedicated policy-lab capacity (RCTs, quasi-experiments, A/B regs) achieve better fiscal ROI/quality-of-service vs. peers.
- Audits showing minimal/major metric gaming despite high-stakes use of indicators.
- Conversely, evidence that experimentation commonly stalls or backfires due to politics/ethics/scale—undercutting the thesis.
- Critical unknowns: political time-horizons; data interoperability; civil-service skills; safeguards for equity and dignity; when experiments are inappropriate (e.g., rights-based services).
- Falsifiable next steps (pilotable):
- Stand up a Policy Experiment Charter in one agency with 1% set-aside for experiments; pre-register all trials; publish nulls.
- Adopt multidimensional scorecards (effect, cost, equity, durability, public trust) to avoid single-metric gaming.
- Create a bipartisan Sunset & Switch rule: if pre-specified thresholds aren’t met, pivot to the runner-up design.
- Mandate Evidence ADR: independent methods board to arbitrate feasibility/ethics quickly.
E — Evaluate
| Dimension | Score (1–5) | Notes |
|---|---|---|
| Clarity & framing | 5 | Compelling “experimenting society” narrative; defines virtues and pitfalls (Campbell’s Law). |
| Evidence quality | 3 | Authoritative references but light on comparative data and concrete case syntheses. |
| Balance / fairness | 3 | Critiques both populists and centrists; still uses broad political generalizations. |
| Actionability | 3 | Values are actionable if paired with a policy-lab playbook; not provided here. |
| Rigor against gaming | 4 | Correctly flags metric gaming; recommends multidimensional evaluation in principle. |
| Overall | 3.6 / 5 | Strong concept piece; incomplete operational guidance. |
Practical Takeaways (OS “What to do Monday”)
- Create a small, protected Policy Lab with legal cover and 1% budget carve-out.
- Require pre-specification (targets, stopping rules) and publish nulls to reduce cherry-picking.
- Use multidimensional scorecards to counter Campbell’s Law.
- Adopt Sunset & Switch rules to ensure failed pilots are retired.
- Invest in data plumbing (ID linkage, privacy, dashboards) so experiments can be run cheaply and ethically.