Every codebase old enough to have opinions eventually produces a meeting where someone says the quiet thing: we should start again. Ours arrived on a Tuesday in February, from an engineer who was right about all of the symptoms.
Deploys took forty minutes and failed about one time in five. Two people understood the billing module and both of them dreaded it. A change to the customer record touched nineteen files. New engineers took a month to ship anything, and the honest reason was not that they were slow, it was that the system could not be held in one head.
So we agreed to plan a rewrite properly, and gave it a fortnight.
The fortnight
The rule for the fortnight was that we would not write any new code. We would write down what the new system would have to do, in enough detail that a stranger could estimate it. Three of us, two weeks, one document.
It ran to twenty-two pages. Most of it was not architecture. It was the list of behaviours the current system has that nobody had written down anywhere: the invoice rounding rule that exists because of one customer’s finance department, the retry that has to be idempotent because a payment provider sends duplicate callbacks, the overnight job that must not run during the two hours when the warehouse is closing its day, the seven email templates that three teams edit.
By the end of the first week the document had a shape, and the shape was not flattering to the rewrite. Page after page described behaviour we would have to reimplement exactly, at no benefit to anyone, before we could ship anything at all.
What the document actually showed
In the second week we did something simple: we went through six months of incidents, support escalations and pull requests that had taken more than a week, and marked which part of the system each one touched.
Sixty-one per cent of the pain came from two modules, billing and the customer record. Another quarter came from the deployment process, which was not code at all. The remaining fifteen per cent was spread thinly across everything else, which meant that everything else was, whatever it felt like, basically fine.
A rewrite would have replaced all of it. We would have spent, on our own estimate, eighteen months to two years reproducing the fifteen per cent in order to fix the eighty-five.
What we did instead
We took the two modules and the deployment process and treated each one as a project, in the order that would let us feel the difference soonest.
Deployment went first, because it was the cheapest and the whole team paid its cost daily. Six weeks of work took the deploy from forty minutes to six, and the failure rate from one in five to something we stopped counting.
The customer record came second. We did not rewrite it; we narrowed it. Three years of accumulated optional fields were traced to the code that read them, and two thirds of them turned out to be read by nothing. Deleting those took a fortnight and removed most of the nineteen-file problem by itself.
Billing came last and took three months, because billing always does. That one was a real rewrite, of one module, behind an interface we had already made explicit while writing the document. We ran the old and new implementations side by side for six weeks, compared every invoice they produced, and found eleven disagreements, nine of which were bugs in the old one.
Six months, start to finish, with the product still shipping the whole way through.
What it cost
The fortnight itself was expensive: three senior people, no features, at a moment when the roadmap was already tight. I would spend it again. Not planning a rewrite costs far more than planning one and then declining it.
It also disappointed people. A rewrite is exciting, and “we are going to narrow the customer record” is not. The engineer who first asked for the rewrite told me a year later that the work we did was better, and that it had not felt better at the time.
And we did not fix everything. The reporting code is still ugly. Two of the seven email templates still live somewhere strange. Nobody has been harmed by either, which is the test we now apply.
What it did to the team
I expected relief and got something closer to deflation. A rewrite is a clean story with a beginning and an end; six months of narrowing a customer record is not a story at all. For about a month the team was flat.
What turned it round was making the pain visible. We put the deploy time on a screen in the room and watched it fall, week by week, from forty minutes to six. Nobody has ever been excited by a specification document, but people will cheerfully argue about a number that is going the right way.
The document did not go in a drawer
The twenty-two pages are still the most useful thing we wrote that year. New engineers read them in their first fortnight, because they are the only description of what the system actually promises. Two of the odder behaviours in it turned out to be nobody’s requirement at all, and we removed them. And when the billing rewrite came, the specification was already written, argued over and agreed.
If the fortnight had produced nothing but that document, it would still have paid for itself.
The rule I took from it
When someone proposes a rewrite, the answer is neither yes nor no. It is: write down what the new thing would have to do, and mark where last year’s pain actually came from. Two weeks of writing usually finds a six-month version of the same project.
Occasionally it will not. Sometimes the honest conclusion is that the foundation really is wrong, and then you have twenty-two pages of hard-won specification to start from, which is the best possible way to begin a rewrite. Either way the fortnight pays.
Leave a Reply