# The work that isn't code

An analysis is finished when somebody has acted on it. Everything between the question and that moment is work, and none of it is code: choosing which question is worth two weeks, deciding in advance what would change your mind, knowing when a rough number is good enough to send, stating it so a busy person can act, and going back later to find out whether you were right.

Most of that has no session of its own. Nothing here is graded, and all of it shows up in work that is.

## Which chair you are in

Some of you will commission an analysis and sign off on it. Some of you will produce it. Most people do both inside the same year, often in the same week. Each section below ends with what the habit looks like from each side of that handoff.

## 1. Deciding which question is worth answering

This choice has the largest consequences and gets the least attention. The [project brief](/guides/project-brief/) is where the course makes you face it. What the brief does not tell you is how to rank candidate questions, and the moves below do.

**Turn the vague thing into a number.** Douglas Hubbard's *How to Measure Anything* opens with three steps for anything that sounds unmeasurable. If it matters, it must be detectable somehow. If it is detectable, there is more of it or less of it. If there is more or less of it, there is a number. Run that on a request like *are we too dependent on certain suppliers*. Dependence is detectable, or nobody would be worried about it. It comes in amounts. So a number exists, and the work is choosing which one.

**Rank by what you would do differently.** For each candidate, write down the decision the answer feeds, then write what you would do at each plausible answer. When the action is the same across the whole range, the question is worth nothing, however interesting it is. Hubbard's teams saw this pattern often enough to name it the measurement inversion: what organizations measure most tends to matter least, and the variables that would change a decision are the ones nobody has measured.

| | |
|---|---|
| **Commissioning** | State what you would do differently before you ask for the work. Nobody can infer it from the request. |
| **Producing** | Ask for it, and keep asking until the answer is specific. A request whose owner cannot state what they would do differently is a request to be talked out of. |

## 2. Before you start work

Put these on paper before the first query. Both get harder to write honestly once you have seen the data.

**Your criteria.** Cassie Kozyrkov, Google's first chief decision scientist, puts the order plainly: frame the decision, set the criteria, and only then open the data. Take those in the other order and you lose the ability to be wrong, because whatever the data shows will read as support for what you already believed. So write down what you would do if no new data arrived, which is your default action, and what result would move you off it, as a number with a direction. The project brief asks for the second one. The first is what makes it answerable, because a threshold with no action behind it is one you will revise the moment the data disappoints you, and you will not notice yourself doing it.

**Your reading of the request.** Most requests arrive vague and stay vague if you wait. *Why are users dropping* is not a question yet, and the person who asked it usually cannot make it one on the spot. Waiting is the expensive option. What works is a proposal: write the interpretation you intend to pursue in one paragraph, name the metric and the window and the comparison, name the decision you think it feeds, and send it back for correction. Being corrected on a paragraph costs a day. Being corrected on two weeks of work costs two weeks.

In the session *Signing off on somebody else's work*, you work through a request where every part is missing, using a five-part specification: the metric, its grain, the window, the comparison, and the decision it informs.

| | |
|---|---|
| **Commissioning** | Answer the proposal. Silence reads as agreement, and it is the most expensive thing you can send. |
| **Producing** | Send it early enough that being wrong about it is still cheap. |

## 3. Knowing when a rough number is finished

Time, data quality and access are always short, and the decision has a date on it. A rough number before the meeting is worth more than a precise one after it. That trade is real, and it is easy to make dishonestly by shipping the rough number without stating it is rough.

**State the precision you have.** "Somewhere between 30 and 45 percent, and I would not defend a tighter range" is more useful than 37.4 percent, because it tells the reader which decisions the number can carry and which it cannot. A number reported to four digits of precision it does not have is a claim about your work, and the claim is false.

**Run a premortem before you send it.** Gary Klein's version, published in *Harvard Business Review* in 2007: assume it is six weeks from now and the number turned out to be wrong, then write down why. Assuming the failure has already happened is what makes the reasons available. The research behind it, by Deborah Mitchell, Jay Russo and Nancy Pennington, found that imagining an outcome has already occurred improves people's ability to identify its causes by about 30 percent.

It takes five minutes, and it is worth the most when you have the least time, which is exactly when it gets skipped.

| | |
|---|---|
| **Commissioning** | State the deadline and the precision the decision actually needs. Most decisions need less than the analyst assumes, and nobody tells them so. |
| **Producing** | Give the range you would defend, and state what another week would buy. |

## 4. Delivering it so somebody can act

Barbara Minto developed the pyramid principle at McKinsey in the years before 1973, and its core instruction is one line: put the answer at the top. The governing thought comes first, the two or three arguments holding it up sit underneath, and the evidence sits under those. A reader who stops after the first paragraph should still have enough to act on.

So a memo goes in this order:

1. What to do, in one sentence.
2. The number it rests on.
3. What would change it.
4. How you know the number is right, which is the section every deliverable that states a number carries.

The other shape, where the reader follows your investigation in the order you happened to do it, is a story about your week. Nobody reads it to the end.

**Put it where the reader will open it.** Some people read a paragraph in a message and never open an attachment. Some want the deck. Some want the notebook and will re-run it themselves. The content stays the same and the container changes, so ask which one lands before you spend an evening on the wrong one.

**Disclose the limitation before somebody finds it.** Trust survives a number that turned out wrong and was flagged as uncertain. Trust does not survive a caveat the reader finds on their own, because after that every number of yours gets checked.

| | |
|---|---|
| **Commissioning** | Ask what the analyst is least sure of. The answer tells you more than the headline number does. |
| **Producing** | Lead with the answer, then with the assumption you had to make to produce it. |

## 5. After it is delivered

Handing over the memo is the middle of the job. Whether you learn anything from it depends on these questions.

**Was the decision made, and did your work have anything to do with it?** An analysis that changed nothing is worth knowing about while you can still ask why. Sometimes it arrived after the decision. Sometimes it answered a question the decision did not turn on. Both are fixable, and neither is visible from inside the work.

**Did the number hold?** Three months on, is the pattern still there.

The habit that lets you answer the second one is a **decision journal**. Asked once what single change would most improve an investor's performance, Daniel Kahneman said: keep one. Michael Mauboussin's version is to write down what you decided, why you decided it, what you expect to happen, and how confident you are, before the outcome is known. Philip Tetlock's forecasting research points the same way. The people whose judgment improved were the ones who wrote predictions down and went back to score them.

Two lines in the repository where the analysis lives are enough:

> Expect the western reorder rate to stay above twice the eastern rate through Q1. Confidence 60 percent. Check on 2027-03-31.

Your memory will not do this for you. It edits the prediction to match the outcome, and it does it without telling you.

| | |
|---|---|
| **Commissioning** | Tell the analyst what you decided and which part of their work you used. Almost nobody does this, and it is the only way they find out. |
| **Producing** | Ask. Then check your own written prediction, so the answer means something. |

Part 6 of the project brief, the one you leave blank when you hand the brief in, is where this starts. The final memo answers it, and the walkthrough questions are the first time somebody else scores your confidence against your evidence.

## Where these come from

There are five sources here, each for one thing.

- Douglas Hubbard, *How to Measure Anything*. The clarification chain and the measurement inversion. Read the first three chapters and stop.
- Cassie Kozyrkov, "What Great Data Analysts Do and Why Every Organization Needs Them", *Harvard Business Review*, December 2018. What an analyst is for, and why getting to a useful question fast beats going deep on the wrong one. The criteria-before-data order comes from her writing on decision intelligence elsewhere.
- Gary Klein, "Performing a Project Premortem", *Harvard Business Review*, 2007. Two pages. The whole method is in them.
- Barbara Minto, *The Pyramid Principle*. Answer-first structure. The first part is the useful part.
- Emily Robinson and Jacqueline Nolis, *Build a Career in Data Science*. Chapter 12, "Working with stakeholders", and chapter 13, "When your data science project fails", which is the territory of sections 2 and 5 above.