{"id":23,"date":"2026-07-21T10:20:00","date_gmt":"2026-07-21T10:20:00","guid":{"rendered":"https:\/\/colorlibhub.com\/unapp-blog\/2026\/07\/21\/six-hours-on-a-train-and-nothing-to-show-for-it\/"},"modified":"2026-09-18T08:46:40","modified_gmt":"2026-09-18T08:46:40","slug":"three-weeks-on-one-bug","status":"publish","type":"post","link":"https:\/\/colorlibhub.com\/unapp-blog\/2026\/07\/21\/three-weeks-on-one-bug\/","title":{"rendered":"Three weeks on one bug"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">The fix was one line. Finding it took three weeks.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The bug was easy to describe: about one order in two thousand reached the warehouse twice. Not charged twice, which would have been spotted quickly, just shipped twice. Customers who received two parcels mostly kept quiet about it, which is why it had been happening for months before anyone noticed.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How we found out at all<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Nobody reported it. A warehouse supervisor mentioned, in passing, on a call about something else, that they were seeing &#8220;the odd double&#8221; and had assumed it was how the system worked. Four months of it, absorbed by people who thought it was normal.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That has stayed with me more than the bug. Every system has a layer of humans quietly working around its defects, and they will not tell you unless you go and stand next to them.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Week one: the obvious theory<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The obvious theory was a retry. Something times out, something else tries again, and the second attempt succeeds as well as the first. We added logging around every retry in the ordering service and waited.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The logs showed no retries on any of the duplicated orders. We had spent a week proving what the bug was not, which at the time felt like spending a week on nothing.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Week two: the clever theory<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The clever theory was a race between two workers picking up the same job from the queue. It was clever enough that two of us became quite attached to it, which should have been the warning. We wrote a test that reproduced a race very like it, fixed that race, and deployed on a Thursday.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The duplicates carried on through the weekend. We had found a real bug, just not this one, and I had spent three days arguing for a theory because it was elegant rather than because the evidence supported it.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Week three: the spreadsheet<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">In the end we did the dull thing. Every duplicated order from the previous month went into a spreadsheet, one row each, with every timestamp we could find from every system it touched: order created, payment captured, queue enqueued, worker started, warehouse acknowledged, parcel labelled. Eleven columns, forty-two rows. Filling it in took two days and most of a shared afternoon.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">On the third day a pattern appeared that nobody had been looking for: every duplicate had been placed within a few seconds of a deploy. Once we saw it, the rest took an hour. The warehouse integration kept its own list of pending orders in memory and, when it shut down, sent them all again &#8220;to be safe&#8221;. The line that did this had been added two years earlier in a hurry, by someone fixing a different problem, with a comment above it that said &#8220;temporary&#8221;.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We deleted it. The duplicates stopped.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What the three weeks bought<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">It is easy to look at a one-line fix and conclude the time was wasted. I used to think that. I do not now, for a few reasons:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>We fixed a genuine race condition along the way, which would have hurt us later.<\/li>\n\n\n<li>We now log every deploy alongside the order events, which has already explained two other problems.<\/li>\n\n\n<li>Four people understand the warehouse integration properly, where before nobody did.<\/li>\n\n\n<li>The supervisor now rings us about &#8220;odd&#8221; things, because someone took the last one seriously.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">What the three weeks did not buy was anything to show at the fortnightly demo. That is the real difficulty with work like this. Its value is in things that now do not happen, and nobody puts those on a slide.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What we told the customers<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Four months of duplicates meant a few hundred parcels that should not have been sent. We wrote to everyone affected, said what had happened in two paragraphs, and told them to keep the second parcel; asking people to return something we had sent by mistake would have cost more in postage and goodwill than the goods were worth.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Eleven people wrote back. Nine were friendly, two had assumed for months that the extra parcel was a gift with a large order, and one asked, reasonably, how we intended to notice next time. That question became the deploy markers in the order log.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What I look for now<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Since then I have kept a short list of signals that a bug is the kind that rewards the boring approach rather than another theory:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>It is rare but regular. One in two thousand, every week, is a mechanism, not bad luck.<\/li>\n\n\n<li>It crosses a boundary between systems, especially one where two teams each assume the other is handling the edge case.<\/li>\n\n\n<li>It has survived at least one confident fix.<\/li>\n\n\n<li>The people closest to it have already invented a workaround.<\/li>\n\n\n<li>Nobody can say what the system does in the failing case without going and reading the code.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">When three of those are true, I stop theorising and start tabulating on the first day rather than the fifteenth. It is not a rule, and a theory that can be tested in an hour is always worth the hour. What the list protects against is the theory that takes three days to test and is loved by the person testing it.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The lesson I keep relearning<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The fastest route to that bug was the slow one. The spreadsheet was available on day one. We skipped it because it looked like drudgery and the theories looked like progress, and theories feel like competence in a way that filling in columns never does.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Next time, I would like to think I will start with the spreadsheet. On past form, I will start with a theory, notice by Wednesday, and be a fortnight quicker than I was this time. That is the actual shape of getting better at this: not doing it right, but noticing sooner.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The fix was one line. Finding it took three weeks, two wrong theories and a spreadsheet nobody wanted to fill in. I used to think weeks like that were wasted. I no longer do.<\/p>\n","protected":false},"author":117,"featured_media":52,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8],"tags":[],"class_list":["post-23","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-moving-fast"],"_links":{"self":[{"href":"https:\/\/colorlibhub.com\/unapp-blog\/wp-json\/wp\/v2\/posts\/23","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/colorlibhub.com\/unapp-blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/colorlibhub.com\/unapp-blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/colorlibhub.com\/unapp-blog\/wp-json\/wp\/v2\/users\/117"}],"replies":[{"embeddable":true,"href":"https:\/\/colorlibhub.com\/unapp-blog\/wp-json\/wp\/v2\/comments?post=23"}],"version-history":[{"count":2,"href":"https:\/\/colorlibhub.com\/unapp-blog\/wp-json\/wp\/v2\/posts\/23\/revisions"}],"predecessor-version":[{"id":66,"href":"https:\/\/colorlibhub.com\/unapp-blog\/wp-json\/wp\/v2\/posts\/23\/revisions\/66"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/colorlibhub.com\/unapp-blog\/wp-json\/wp\/v2\/media\/52"}],"wp:attachment":[{"href":"https:\/\/colorlibhub.com\/unapp-blog\/wp-json\/wp\/v2\/media?parent=23"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/colorlibhub.com\/unapp-blog\/wp-json\/wp\/v2\/categories?post=23"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/colorlibhub.com\/unapp-blog\/wp-json\/wp\/v2\/tags?post=23"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}