Data-Informed Decisions

What it is

Hypothesizing, experimenting, observing, learning, creating and building on theories.

Why it's important

• There is no way to know what will work until you actually try it out
• The quickest way to the best solution is experimentation and iteration
• The best solutions are based on (scientifically / quantitatively) proven principles -- theories

When you need it

• Research -- to define hypotheses and set up experiments and observations that lead to theories that we can design around
• Design -- create concepts to test out, observe, and iterate, reform/refine the theories
• Monitoring -- after a solution is deployed, observe, learn, iterate, reform/refine theories

How to do it

• Research. Conduct primary and secondary research, and quantify (to relative significance) all of the results of your research in a synthesis.
• Hypothesis. After you've conducted enough research to venture a guess, share that with others for validation; this will be in the form of a concept (could be mocks or a clickable prototype), charter or specification.
• Experiments. Either validation sessions (with mocks / concepts) or the release to users of a solution developed based on your specification(s), along with monitoring to learn / validate.
• Share findings. Return to your research participants and key stakeholders to report on the quantitative / qualitative findings and to share next steps.
• Develop a theory. With the data from the experiment and the feedback from the sharing sessions, calibrate the validated learnings / findings and document them in the form of a documented and well-laid-out theory (i.e. "this persona behaves this way and prefers to use these kinds of tools in this way and doing so yields this kind of ROI"), which becomes a product principle.
• Build on a theory. Conduct new research to find problems and design solutions in the space of your theories, validating them in a similar way.
• Iterate. Decide whether to pivot or persevere.
• Be objective. It's not all roses. Make sure to balance the case for with the case against.

Resources

• The Lean Startup
• Solving Product
• About Face
• Trustworthy Online Controlled Experiments

The competency

You are able to postulate meaningful hypotheses and create a method for invalidating them. You carefully gather, store, and analyze data through observation. You routinely challenge perceptions and keep a healthy skepticism until hypotheses can be validated. You work hard to eliminate bias by triangulating the truth through several tests of the hypothesis. Your work leads you to the development of theories backed by experimentation and data.

Outputs and deliverables

Data backed and well-referenced/triangulated findings through experimentation and observation. Well-documented findings referenced in charters, specifications, and reports.

• Regular research with monthly (at least) syntheses shared to the team
• Validation data (customer feedback on product, usability tests, perception tests, etc.)
• Assessments of monitored releases on judgment days

Outcomes

• Product roadmap/specs based on scientific data
• Validated learnings