The Art of Failing: How Experimentation Culture Transforms Engineering Teams

I was sitting around the monthly product meeting table when Bron stepped up to present to the group. What followed was a masterclass in how not to launch a feature. Months of development work, multiple iterations, and the users had reacted so poorly to the experiment that you could practically see the product owners near him slowly edging away, as if trying to escape the blast radius of what was surely coming from senior leadership.

Our CEO at the time leaned forward at the end of the presentation. The room held its breath. Then he said one simple sentence: “What did we learn?”

The Winners and Losers Mindset

Here’s the thing — we’re all conditioned from childhood to worship at the altar of winning. Winners get gold medals. Winners get trophies. Winners get promoted. As Ricky Bobby so eloquently put it in Talladega Nights: “If you ain’t first, you’re last.” This binary thinking becomes so deeply embedded that we carry it straight into our engineering cultures, where it becomes toxic.

You know this mindset. We’ve all seen it. The team that spent six months building the “revolutionary” feature that nobody wanted. The architect who doubled down on a failing design because admitting it was wrong felt like career suicide. The PM who kept pushing a doomed experiment because stopping meant “giving up.”

This is the sunk cost fallacy in action — the irrational belief that because we’ve already invested time, money, or effort into something, we must continue investing to justify the initial investment. In engineering, this manifests as “we’ve already written 10,000 lines of code, we can’t throw it away now” or “the team has been working on this for months, we have to ship something.” But here’s the brutal truth: those six months are gone whether you ship a feature users hate or you kill it and build something they love.

As I explored in my previous post about cultural anti-patterns, this “lines must go up” mentality creates environments where we optimise for the appearance of success rather than actual learning.

The Experimentation Antidote

But here’s where experimentation culture becomes your secret weapon. When you build systems that expect failure — whether that’s through A/B testing platforms, feature flags, or even just proper data collection — you create something revolutionary: an environment where it’s safe to fail.

Think about it this way. When your A/B test shows that your brilliant new checkout flow actually decreases conversions, the data isn’t attacking you personally. It’s simply telling you that your assumptions about customer behaviour were incorrect. And that’s valuable intelligence, not a personal failure.

“I have not failed. I’ve just found 10,000 ways that won’t work,” Thomas Edison famously said. He understood something we often forget in software engineering: failure is data, and data is how we get smarter.

When we fail in our experiments, it means our collective understanding of our customers was incomplete. Sometimes we need slight adjustments. Sometimes we need to throw everything out and start fresh. Both outcomes are good, because this feedback loop leads to better experiments and ultimately better products for our customers.

Practical Steps to Cultural Change

You don’t need a full-blown experimentation platform to start shifting this mindset. Here are some approaches that work:

Start with Blameless Post-Mortems. These are structured reviews of incidents that focus on understanding what happened and how to prevent it, rather than who was at fault. The goal is to create a timeline of events, identify contributing factors, and establish action items for improvement — all while maintaining psychological safety for everyone involved. Production incidents are usually the most emotionally charged failures in our industry. If you can tackle blame culture there, you’ll find the horizon brightens considerably for everything else. Companies like Google, Etsy, and Netflix have written extensively about their blameless post-mortem practices, and there are excellent resources available through the SRE community and DevOps movement.

Celebrate Intelligent Failures. Not all failures are created equal. The failure that comes from running a well-designed experiment with clear hypotheses deserves different treatment than the failure that comes from ignoring best practices. Learn to distinguish between the two.

Change Your Language. Instead of “this failed,” try “this didn’t work as expected.” Instead of “we lost,” try “we learned.” It sounds like corporate speak, but language shapes thinking more than we realise.

Learning from Other Industries

Some companies have taken this much further than most of us in tech:

Intuit runs “Best Failure” trophy ceremonies where teams present their biggest flops over food and music. The purpose isn’t to humiliate — it’s to encourage risk-taking and share lessons learned across the organisation. This helped them create a “fail fast, learn fast” culture that now runs hundreds of experiments per month.

Eli Lilly, the pharmaceutical giant, throws scientific “failure parties” to honour teams whose drug candidates failed but advanced scientific understanding. In an industry where failure rates exceed 90%, they’ve learned to normalise setbacks while salvaging valuable data. Some of their “failed” molecules have been repurposed into billion-dollar products.

W. L. Gore & Associates (yes, the Gore-Tex people) host what they call “Failure Nights” where associates recount their misfires over beer and champagne. Their philosophy is simple: more calculated risks lead to more breakthroughs.

The Uncomfortable Truth

Here’s what nobody wants to admit: most of our features will fail. Most of our assumptions about users are wrong. Most of our brilliant technical solutions will need significant revision. This isn’t a bug in the system — it’s a feature.

As Winston Churchill observed, “Success is going from failure to failure without losing your enthusiasm.” In software engineering, success is going from failed experiment to failed experiment while continuously improving your understanding of what customers actually want.

Making the Shift

So how do we get from “if you ain’t first, you’re last” to “what did we learn?” It starts with leadership asking different questions. Instead of “Why did this fail?” try “What assumptions did we test?” Instead of “Who’s responsible for this?” try “What would we do differently next time?”

We need to measure learning velocity, not just delivery velocity. We need to celebrate teams that kill their own projects when the data shows they’re heading in the wrong direction. We need to recognise that the team that runs ten experiments and learns from nine failures is more valuable than the team that ships one feature without testing it.

The next time you’re in that product meeting, watching a colleague present what looks like a disaster, remember my CEO’s response. Don’t ask “How could you let this happen?” Ask “What did we learn?”

Because in the end, the companies that learn fastest will always outpace the companies that fail slowest.

What’s your experience with failure culture in engineering teams? Have you seen teams transform from blame to learning? I’d love to hear your stories — the failures and the successes alike.