Building a data informed Decisions Practice

data informed decisions means letting data guide your thinking while retaining human judgment. Data-driven decisions means letting data make decisions for you. The distinction matters enormously.

I’ve seen teams paralyzed by data, refusing to ship until metrics proved it would work. I’ve also seen teams ignore clear signals in their data because gut feel said otherwise. Neither extreme works.

The teams that build great products use data to inform hypotheses, validate assumptions, and measure outcomes. But they still apply judgment about what data means and what actions to take. That balance is what we’re after.

Getting Started: The Foundation

Before you can build a data informed practice, you need the basics in place. Not sophisticated analytics platforms—just the fundamentals.

Instrument the Critical Paths

You can’t make data informed decisions without data. Start by tracking the metrics that matter most to your business.

For most products, that means: acquisition (how users find you), activation (first meaningful value), engagement (ongoing usage), retention (coming back), and revenue (monetization). Track these before anything else.

Amplitude built their business on the insight that most teams don’t actually know their activation metrics. What action predicts that a user will stick around? If you can’t answer that with data, you’re guessing.

I use this test: can you draw your funnel with actual numbers? What percentage of visitors sign up? What percentage activate? What percentage engage regularly? If these are fuzzy, start there.

Define Success Metrics Early

Before building features, define how you’ll measure success. Not in retrospect—before you start.

This forces clarity. If you can’t articulate what success looks like numerically, you probably don’t understand the problem well enough to solve it.

I’ve made the mistake of shipping features and then scrambling to figure out if they worked. Don’t do this. Decide success criteria before building.

Build Data Literacy in Your Team

Not everyone needs to be a data scientist, but everyone should be comfortable with basic analysis. Understanding conversion funnels, retention cohorts, and statistical significance isn’t optional for modern product teams.

I run monthly data workshops where we analyze recent launches together. What did we expect? What happened? Why? This builds intuition for what data means and how to interpret it.

The goal isn’t turning designers into analysts. It’s ensuring everyone can critically evaluate data rather than just accepting whatever numbers are presented.

Creating Your Process

Having data is step one. Using it systematically is where most teams struggle.

Make Data Part of Regular Rituals

Don’t make data analysis a separate activity. Embed it in how you already work.

Before sprint planning, review metrics for features you shipped last sprint. What worked? What didn’t? Why? This turns data review from “someone should look at that sometime” to “we always do this.”

Run Experiments, Not Just Features

When you’re uncertain, structure work as experiments with clear hypotheses and success criteria.

“We believe improving onboarding will increase activation” is a hypothesis you can test. “We should improve onboarding” is just an opinion. The difference is measurability.

I don’t A/B test everything. But for significant changes where the outcome is uncertain, experiments beat guessing.

Track Leading and Lagging Indicators

Revenue is a lagging indicator—it tells you what already happened. Engagement is a leading indicator—it predicts what will happen.

Track both. Lagging indicators tell you if you’re succeeding. Leading indicators tell you if you’re heading in the right direction before it’s obvious in outcomes.

Duolingo tracks lesson completion as a leading indicator of retention. If completion rates drop, they know retention will follow before seeing it in the retention cohorts.

I create dashboards showing both types of indicators. This helps teams see problems coming rather than only reacting after they’re visible in business metrics.

Hold Data Review Sessions

Weekly or bi-weekly, sit down with your team and look at the data together. Not to report—to discuss and learn.

What’s surprising? What contradicts our assumptions? What patterns are we seeing? These conversations build shared understanding that individuals analyzing data alone don’t get.

Common Pitfalls That Derail Teams

Let me spare you some mistakes I’ve seen or made.

Analysis Paralysis

Waiting for perfect data before making decisions is a great way to do nothing. At some point, you need to make a call based on the data you have.

I use this guideline: if the decision is reversible, don’t wait for certainty. Ship, measure, iterate. If the decision is expensive to reverse, invest more in validation first.

Cherry-Picking Data

It’s easy to find data supporting what you already believe. The discipline is honestly evaluating contradictory data too.

I’ve was a PM in teams that dismiss clear signals that a feature wasn’t working because they were emotionally invested in shipping it. “The data must be wrong” is sometimes true but more often is self-deception.

Build a culture where bad news about product performance is welcomed, not shot. If people fear sharing negative data, you’ll only see positive data—which means you’re flying blind.

Ignoring Qualitative Signals

Not everything meaningful shows up in metrics. User frustration, confusion, and delight often require qualitative research to understand.

I combine quantitative data with regular user interviews. The quant tells me what’s broken. The qual tells me why and what to do about it.

Mistaking Correlation for Causation

Just because two metrics move together doesn’t mean one causes the other.

I once saw a team celebrate that a feature correlated with higher retention. Turned out power users discovered the feature. It didn’t improve retention—retained users found it. Cause and effect were backwards.

Run experiments when causation matters. Correlations generate hypotheses. Experiments test them.

Maintaining the Practice Long-Term

Building a data informed culture is ongoing work, not a one-time setup.

Make Data Accessible

If data lives in SQL databases only engineers can query, most people won’t use it. Build dashboards and tools that make key metrics accessible to everyone.

Amplitude, Mixpanel, and similar tools exist precisely because making data accessible improves decision-making. The easier it is to answer questions with data, the more questions people ask.

I create “starter dashboards” for common questions. How’s activation trending? Where do users drop off? What features do retained users use? Answer these questions visually so people don’t need to know SQL.

Celebrate data informed Decisions

When someone uses data well to inform a decision, make it visible. Share the analysis. Explain the reasoning. This reinforces the behavior you want.

Conversely, when gut feel decisions go poorly, retrospect on what data could have informed that choice better. Not to blame—to learn.

Update Your Approach as You Learn

Your data needs evolve. Metrics that mattered at 100 users might not matter at 10,000. The questions you can answer with more users and data should expand over time.

I review our analytics setup about quarterly. What questions can’t we answer? What metrics are we tracking but not using? What new capabilities do we need?

This prevents analytics debt—tracking things out of habit that nobody looks at, while ignoring questions that actually matter.

Key Takeaways

Here’s what matters for data informed decisions:

• Start with metrics that matter: Track acquisition, activation, engagement, retention, and revenue before getting fancy. These fundamentals drive business outcomes.
• Define success before building: Decide how you’ll measure success before shipping features, not after. This forces clarity and enables real evaluation.
• Balance data with judgment: Let data inform decisions while applying human judgment about context and meaning. Data tells you what happened, not always why or what to do.
• Make data accessible and regular: Embed data review in existing rituals and make metrics easy to access. If looking at data is hard, people won’t do it.

Making This Work

Start small. Pick three metrics that matter most to your product right now. Make sure you’re tracking them accurately. Review them weekly with your team.

When you make your next product decision, write down what data informed it and what you expect will happen. Then measure what actually happens. The gap between prediction and reality is where you learn.

Building a data informed practice isn’t about fancy tools or complex analyses. It’s about consistently using data to inform decisions, measuring outcomes, and learning from the results. Start doing that, and sophistication will follow naturally.

Have questions or thoughts? Get in touch - I’d love to hear from you!