When a simple algorithm beats ML

The brief was simple: stop pull requests from sitting unreviewed because nobody knew who should look at them. On a team of about thirty engineers pushing something like fifty PRs a week, "who reviews this?" got answered by whoever felt guilty, or not at all. PRs aged, context evaporated, and the obvious move in 2022 was to throw a model at it. I didn't—and I want to explain why the boring choice was the right one.

A routing problem wearing a prediction problem's clothes

The framing that gets you into trouble is "predict the best reviewer." Say it that way and you've described a supervised-learning problem: gather features about PRs and people, collect labels, train something to rank candidates. Now you need training data, and the label—the "correct" reviewer—is exactly the thing you don't have, because the whole reason you're building this is that reviewer assignment is currently ad hoc. You'd be manufacturing ground truth for a question nobody had been answering consistently in the first place.

But you don't actually need to predict anything. "Who should review this change?" has a defensible answer sitting in plain sight: the people who have worked on the code this change touches. That's not a prediction. It's a lookup. Reviewer assignment is a routing problem—match a diff to the people with standing in the files it modifies—and routing problems are the kind you solve with arithmetic, not learning.

Git already records everything you need. Every commit says who changed which files, and when. The signal isn't hidden in some latent space you have to train a model to recover; it's right there in git log.

The algorithm

For each pull request, I started from its diff—the files it changes and how many lines change in each. Not every file carries the same weight. A change that's eighty percent in session.py and one line in a docstring is really a change to session.py, and the recommendation should reflect that. So each changed file gets a weight from its line count, normalized across the PR's files so a dominant file counts for more and an incidental one-liner counts for little.

Within each file, I looked back over the last year of commits and counted how many each person had made. Raw counts aren't comparable across files—some see a flurry of activity, some a trickle—so instead of using them directly I converted each file's counts to a standard score: how many standard deviations above or below that file's average contributor each person sits. Someone who's committed far more than the file's norm scores high; someone at or below it scores near zero or negative, and negatives get clipped to zero so a light contributor never counts against anyone.

A person's overall score is the sum, across every file in the PR, of that file's weight times their standard score on it. Drop the PR's own author, take the top three, and that's the whole engine.

Recency is where the design has an opinion. Knowledge goes stale: someone who owned a file last week can still review it well; someone who last touched it a year ago has mostly forgotten it. What shipped handled this bluntly—a hard cutoff, where only commits from the last year counted and everything older was invisible. The obvious refinement is to make recency continuous instead of a cliff: weight each contribution by an exponential decay—a seven-day half-life, say, so last week's commit outweighs last quarter's by a wide margin—which models how fast context evaporates far more faithfully. But the hard window was simpler, it shipped, and it was good enough. Choosing the blunt version on purpose—and being able to see exactly what it gives up—is the kind of decision a legible design lets you make in the first place.

The unglamorous eighty percent: matching people to people

Here's the part nobody warns you about. Git commits identify authors by whatever name and email they had configured locally, possibly years ago. The system that assigns reviewers—Bitbucket, in our case—identifies people by account. These two do not agree. The same human is Jane Doe <jane@laptop.local> in one commit, jdoe <jane.doe@company.com> in another, and a Bitbucket account under a display name that matches neither. If you can't bridge git authors to review-system accounts, none of the elegant scoring matters, because you can't actually @-mention anyone.

I ended up writing a small token-based matcher. Take a name or email, strip the diacritics (via Unicode character names—you look up LATIN SMALL LETTER E WITH ACUTE, chop off the WITH …, and look the base letter back up), drop the email domain, split on non-alphanumerics, lowercase, throw away single characters, and you're left with a set of tokens. Do that for every Bitbucket account and for each git author. Then match an author to an account if they share a token that's globally unique—one belonging to exactly one person across the whole workspace. Authors who can't be matched log a warning and get dropped rather than guessed at.

It isn't glamorous, and it has holes—a TODO in the code still reminds me to handle unique combinations of otherwise-common tokens. But it's the kind of problem that decides whether a tool ships or dies in a demo, and it's exactly the kind of problem an ML approach wouldn't have saved me from. I'd have had to solve identity matching just to build a training set.

Advisory, not blocking

The engine ran in Bitbucket Pipelines on every PR and left its recommendations as a comment: "The following people are recommended as reviewers, based on past contributions to modified files," and a numbered list. Deliberately a comment, not an enforced gate. Files with a formal CODEOWNERS entry got those owners added as official reviewers through the API; the algorithm's statistical picks stayed advisory. There was a [skip reviewers] escape hatch in the commit message, the pipeline step was non-blocking, and the bot updated its existing comment instead of stacking new ones when a PR changed.

That restraint was a trust decision. A tool that suggests and explains earns its way into a workflow. A tool that blocks on a heuristic gets ripped out the first Friday afternoon it stands between someone and a merge.

Why not ML—actually

I want to be fair to the machine-learning version, because I did consider it. You could learn the file-weighting and the scoring thresholds from historical review data. You could add features—review latency, who tends to give thorough reviews, expertise beyond raw commit counts. There's a real model to be built here.

I rejected it, and the reasons compound.

The data is already high-quality and structured. Git history isn't a noisy proxy for contribution; it is contribution, recorded exactly. When your signal is already clean and structured, the marginal value of a model that learns to approximate it is small.

Interpretability is the feature. When the bot recommends you, you can see precisely why: you're among the top contributors to the files this PR touches over the last year. That sentence is auditable by the person receiving it—they can agree or dismiss it on the spot. A model that says "0.87, trust me" invites exactly the skepticism that gets a tool ignored. A recommendation an engineer can audit beats a better one they can't, and on a developer-tools team "the engineers quietly stopped trusting it" is precisely how these things fail.

No labels, no training set, no maintenance. There is no ground-truth "correct reviewer" to train against, and no cold-start story for a fresh repo or a new hire. A model is also a thing you own forever: retraining, drift, the day it starts recommending someone who left. The arithmetic version has none of that surface area. It's a few hundred lines of Python: a normalization, a standard score, and a weighted sum.

The honest tradeoff, the one I wrote down at the time: ML might tune the weights slightly better than I did by hand. But that marginal improvement buys real complexity—a pipeline, a training set I'd have to manufacture, a model to maintain—for a tool whose entire job is to nudge the right person toward a PR. Not worth it.

Where ML would have earned its place

This isn't an argument against machine learning; it's an argument for matching the tool to the problem. ML would have earned its keep if the question had been different. If I'd been trying to predict review quality, or how long a review would take, or whether a change was risky enough to demand a senior reviewer—those are genuine prediction problems, with fuzzy signals and no closed-form answer, where a model learns things arithmetic can't express. If the git history had been sparse or misleading—a young repo, or a big rewrite that orphaned all the history—the lookup would have had nothing to stand on, and you'd have to generalize from features instead.

But that wasn't the problem. The problem was routing a diff to the people with the most standing in it, over data that recorded exactly that. When the answer is already written down, the job is to read it clearly, not to predict it cleverly. Reaching for the model first would have been reaching past the answer to build a machine that guesses at it.

The legible solution isn't a compromise you settle for when you can't afford the fancy one. Sometimes it's just correct.