The Great Merge Queue Evolution: A Tale of Complexity, Necessity, and Coming Full Circle
“The real problem is not whether machines think but whether men do.” — B.F. Skinner
“The real problem is not whether machines think but whether men do.” — B.F. Skinner
When I joined Agoda back in 2016, I discovered something that would have made any modern DevOps engineer’s eye twitch: a manual merge queue managed through HipChat messages. Yes, HipChat — that archaeological artifact from the pre-Slack era when we all pretended alternatives to IRC were revolutionary.
But here’s the thing about that “primitive” system: it worked. Sort of.
The Honest Engineers Era
Picture this: 50 engineers working on a single repository, with nothing but trust and a shared HipChat channel standing between order and chaos. The process was beautifully simple — drop your name in the queue, wait your turn, merge your changes, and push to master. No CI pipeline, no automated testing, just good old-fashioned manual testing and developer honor code.
“Trust, but verify.” — Ronald Reagan
Reagan was talking about nuclear disarmament, but he might as well have been describing our early merge strategy. Except we had the trust part down pat — it was the verification that needed work.
The surprising truth? Everyone was honest. In a repo with 50 active contributors, I never once caught someone jumping the queue or pushing untested code. Sometimes the simplest systems work because they rely on the best part of human nature.
The First Automation: Slack Bot Glory Days
By 2016, I’d set up TeamCity and GitHub, and quickly built a Slack bot to automate our manual queue. The bot was a thing of beauty:
Join the queue with a simple commandAutomatic merging from masterParallel processing of the top 3 PRs (hedging against flaky tests)Real-time feedback in SlackQueue position preservation during merge conflicts*“For every complex problem there is an answer that is clear, simple, and wrong.”* — H.L. Mencken
Mencken clearly never met our Slack bot. Sometimes the simple answer is actually right — at least until it isn’t.
Hitting the Wall: The Mathematics of Merge Constraints
Success bred its own problems. We were pushing 30 merge requests per day, with a 40-minute pipeline. Basic math became our enemy:
24 hours × 60 minutes ÷ 40 minutes = 36 theoretical max merges per day
When your end-to-end tests decided to throw tantrums and stretch builds to 60 minutes, that number dropped to a painful 24. We were hitting the hard ceiling of synchronous merging.
“You can’t use up creativity. The more you use, the more you have.” — Maya Angelou
But you absolutely can use up pipeline capacity. Unlike creativity, merge queues have very real, very mathematical limits.
The Parallel Revolution: BuildFlow
Enter BuildFlow, our internal “merge management tool” that would make even the most seasoned DevOps engineer either applaud or run screaming.
The concept was elegantly complex: group queued PRs into “Release Groups” of 2, stack them progressively (RG2 contains RG1 + new PRs, RG3 contains RG2 + newer PRs, etc.), run all pipelines in parallel, and deploy the highest-numbered group that passes.
With 9 PRs in the queue, we’d create 5 Release Groups. If RG4 failed, RG3 became deployable and 6 PRs could go to prod in the 40min it took to run 5 pipelines in parallel. The beauty? All 20 queued PRs or more could theoretically be ready to deploy in a single pipeline run time.
The risk? Pipeline stability became everything. Below 50% stability, and you’d lose more than you gained. We monitored this religiously with scheduled master builds — our canary in the coal mine.
The GitLab Merge Train Era
Years later, we migrated to GitLab just after they launched Merge Train — their take on our BuildFlow concept, but with less configurability and some questionable optimization choices.
Merge Train’s fatal flaw? It reruns your pipeline even when there’s no queue, optimizing for high-volume scenarios at the expense of low-volume efficiency. Green pipeline? Doesn’t matter — run it again.
Full Circle: The Return to Simplicity
As we broke down our monolithic systems into smaller, more manageable pieces, something beautiful happened: we no longer needed the complexity. Merge Train was slowing us down rather than speeding us up.
So we built MergeFlow — essentially our old Slack bot reborn. Sequential merging, simple queue management, merge from master when conflicts arise, rerun the pipeline, repeat. Back to basics, but with the wisdom earned from our parallel merging adventure.
“Everything should be made as simple as possible, but not simpler.” — Albert Einstein
Einstein’s razor cuts both ways. Our BuildFlow wasn’t too complex for its time — it was exactly as complex as our problem demanded. But when the problem changed, so should the solution.
The Lesson: Match Your Tools to Your Actual Problems
The real insight from this decade-long journey isn’t about merge strategies or pipeline optimization. It’s about the danger of solving tomorrow’s problems with today’s solutions, or worse, solving yesterday’s problems with tomorrow’s tools.
Manual queue (2015): Perfect for low volume, high trustSlack bot (2016): Ideal for medium volume, structured processBuildFlow (2017–2019): Essential for high volume, monolithic architectureMerge Train (2020–2022): Good concept, poor implementation for our use caseMergeFlow (2023-present): Right-sized for distributed architecture*“A fool with a tool is still a fool.”* — Grady Booch
Booch was talking about software design, but the principle applies to every technical decision we make. The most sophisticated merge queue in the world won’t help you if it’s solving the wrong problem.
The Meta-Message
We’ve come full circle, from simple to complex and back to simple again. But this isn’t a story about the failure of innovation — it’s about the success of adaptation.
The manual HipChat queue wasn’t inferior to BuildFlow; it was appropriate for its context. BuildFlow wasn’t over-engineered; it was exactly what we needed when we needed it. And MergeFlow isn’t a step backward; it’s a step forward informed by everything we learned.
“The curious paradox is that when I accept myself just as I am, then I can change.” — Carl Rogers
Rogers was talking about psychology, but he captured something essential about technology evolution: sometimes you have to embrace complexity to eventually find simplicity on the other side.
The next time you’re tempted to implement the latest and greatest merge strategy, deployment tool, or architectural pattern, ask yourself: what problem am I actually trying to solve? Not the problem you might have someday, not the problem that sounds impressive in tech talks, but the actual problem you have right now.
Because sometimes the best solution is the one that matches your actual needs, not your imagined ones.
Got your own tales of technical evolution? Share them over a beer — just keep it away from the servers.