When developing code lots of people talk about optimizing your “inner loop.” The cycle of writing code and knowing if what you wrote works as intended.
For many decisions we make—especially product decisions—there is no inner loop. The feedback cycle for knowing if we chose the right thing can be weeks, months, or years. The further away in time you move from the decision the harder it is to know the reasons for success.
One thing that is important to do with long feedback loops is to collect relative data from the moment you make a decision until the time you know the decision outcome. You can’t attribute success to a single decision, and you can never accurately compare two simultaneous decisions.
Layering data from multiple decisions can retroactively shrink feedback loops to discover patterns. Thankfully, people are very good pattern matchers and even with long feedback loops the patterns you find can help you make better decisions with less data in the future.