THE TEST THAT PASSES IS THE BEGINNING OF THE PROBLEM.

A failed experiment is wonderfully convenient.

Something didn't work.

You inspect the wreckage, find out why, change something, and try again.

A successful experiment is much more annoying.

Now you have to figure out what actually succeeded.

I've been spending an unreasonable amount of time lately doing exactly that: taking results I was pleased with and trying to make them go away.

Not reproducing them.

Not improving them.

Destroying them.

Because a result can be completely real while your explanation for it is completely wrong.

And those are two very different things.

A GOOD NUMBER DOESN'T TELL YOU WHY IT EXISTS

There is a seductive moment in experimental work when the number comes back looking exactly the way you hoped it would.

Maybe performance improves.

Maybe an intervention produces the predicted change.

Maybe a structure appears precisely where your model said it should.

Excellent.

Unfortunately, you have learned considerably less than it feels like you have.

You know that something happened.

You do not necessarily know why.

There may be another variable carrying the effect. A preprocessing decision may have quietly encoded information you thought you had removed. Two quantities you treated as independent may be coupled. Your partition may have created the pattern. Magnitude may be doing work you attributed to direction. Identity may be doing work you attributed to structure.

Or the thing you think you measured may actually be real.

The irritating part is that all of these possibilities can produce a beautiful result.

So the next experiment should not be another opportunity for your explanation to succeed.

It should be an opportunity for it to die.

I HAVE BECOME VERY RUDE TO MY OWN HYPOTHESES

This has produced a recurring pattern in my work.

A test passes.

For approximately thirty seconds, this is good news.

Then the questions begin.

Could magnitude be carrying the result?

Remove magnitude.

It survives.

Fine.

Could the effect depend on which identities happened to carry particular orientations?

Rearrange them while preserving the relevant counts.

It survives.

Could the apparent localization be an artifact of how the population was divided?

Attack that.

It survives.

Could reconstruction itself be introducing the effect?

Build a control for that.

Could the control be preserving something it was supposed to destroy?

Attack the control.

Could the new test be exploiting some accidental property of the previous one?

Unfortunately, yes.

Build another test.

At some point the process begins to resemble an argument with an extremely stubborn object.

This is good.

THE PURPOSE OF A CONTROL IS NOT TO LOOK RESPONSIBLE

There is a version of experimental rigor that can become strangely ceremonial.

You include a control because experiments are supposed to have controls.

You report it.

It behaves itself.

Everyone goes home.

But a useful control should represent a plausible competing explanation.

It should be dangerous.

If your interpretation says, “This happened because of X,” the interesting question is not merely whether X correlates with the result.

The interesting question is whether you can preserve everything else that reasonably could have produced the result, destroy X specifically, and make the effect disappear.

Then reverse the logic.

Destroy something you believe should not matter while preserving X.

Does the effect survive?

Change the representation.

Does it survive?

Change the partition.

Does it survive?

Normalize away an obvious signal.

Does it survive?

Construct an alternative explanation and give that explanation the best possible chance to win.

Does it?

This is where experiments become useful arguments rather than demonstrations.

You are no longer collecting examples of your idea behaving correctly.

You are removing places for the explanation to hide.

SURVIVAL CHANGES THE QUESTION

There is another reason this process matters.

Every time a result survives a serious attack, the next question becomes more precise.

The first hypothesis is usually broad.

Something matters.

Fine.

What part?

You localize it.

Now: what property of that part matters?

Magnitude?

Direction?

Identity?

Relationship?

Interaction?

Some conditional structure that only exists in combination with something else?

Each experiment removes part of the explanatory space.

That means a surviving result can actually become harder to explain as the work improves.

This initially feels backward.

You started with several plausible explanations.

You tested one.

Gone.

Tested another.

Gone.

Built a stronger control.

Gone.

The phenomenon remains.

Now you know considerably more about what it isn't, while the thing itself is sitting there with fewer convenient stories available to explain it.

That is progress.

Not particularly emotionally satisfying progress.

But progress.

THIS IS ALSO HOW YOU FIND OUT YOUR FAVORITE IDEA WAS WRONG

There is an obvious danger in designing experiments around something you've built yourself.

You know what you want to be true.

Worse, you know the system well enough to construct extremely sophisticated reasons why it probably is true.

Intelligence does not solve confirmation bias particularly well.

Sometimes it just gives confirmation bias better equipment.

So I try to make the experiments adversarial before I know their outcome.

Define what counts.

Define what doesn't.

Define the populations.

Define the exclusions.

Define the failure conditions.

Freeze the procedure.

Then run it.

This removes one particularly tempting form of creativity from the experiment:

the creativity that appears immediately after seeing a result you don't like.

There are legitimate reasons to revise an experiment after running it. Bugs exist. Bad assumptions exist. Sometimes the test simply doesn't measure what you thought it measured.

But if every unexpected result causes the rules to move, eventually you aren't testing the hypothesis.

You're negotiating with the data.

I would rather lose.

A clean failure is information.

A victory whose rules changed halfway through is decoration.

SOMETIMES THE CONTROL BECOMES THE EXPERIMENT

One of the stranger consequences of this process is that controls tend to reproduce.

You build one to eliminate an alternative explanation.

It works.

Then you realize the control itself contains an assumption.

So you test the assumption.

That test reveals another possible confound.

Now that gets a control.

Soon the original experiment is several generations behind you and you are investigating a question that didn't exist when you started.

This can look ridiculous from the outside.

Occasionally it looks ridiculous from the inside too.

But there is an important distinction between complexity caused by refusing to let go of an idea and complexity caused by systematically eliminating explanations.

The first produces a fortress.

The second produces a narrower claim.

I want the narrower claim.

“Something interesting seems to happen” is easy.

“This particular phenomenon survives these specific competing explanations under these specific conditions” is less exciting to put on a T-shirt.

It is also much more useful.

THE BEST RESULT ISN'T ALWAYS THE BIGGEST ONE

This process changes your relationship with impressive numbers.

A spectacular result with six obvious alternative explanations is interesting.

A smaller result that survives six attempts to destroy it may be considerably more interesting.

Magnitude attracts attention.

Survival earns confidence.

Those aren't the same property.

This is especially important when a result is unusually good.

The more surprising the result, the less interested I become in immediately celebrating it.

I want to know what I accidentally measured.

I want to know whether information leaked across a boundary.

I want to know whether the representation smuggled the answer into the test.

I want to know whether some boring statistical feature is impersonating the mechanism I care about.

And if all of those attacks fail?

Good.

Now the result has earned the right to become interesting.

Not true forever.

Not proven beyond challenge.

Interesting.

That's enough.

A RESULT SHOULD HAVE SCARS

There is a temptation to present experimental work as a clean progression.

Hypothesis.

Experiment.

Result.

Conclusion.

The actual process is often much uglier.

Hypothesis.

Experiment.

Interesting result.

Suspicion.

Control.

New suspicion.

Different control.

Failure.

Bug.

Correction.

Result survives.

Annoyance.

More elaborate attempt to kill it.

Result survives again.

A completely different explanation occurs to you at an unreasonable hour.

Back to work.

By the time I trust a result, I don't want it looking pristine.

I want to know what happened when we removed its easiest explanation.

I want to know what happened when we attacked its second-easiest explanation.

I want to know what assumptions it depends on and which ones it doesn't.

I want scars.

Because the goal isn't to produce a result that looks convincing.

The goal is to discover how much punishment the explanation can take before it stops being the explanation.

I still like successful experiments.

I'm not completely deranged.

I just don't trust them very much.

A result tells you that something happened.

The interesting work begins when you start taking away all the reasons it could have happened, and it refuses to disappear.