The bug your test cannot see

Here is a function. It has one job: give me the three highest scores.

function topThree(scores) {
  scores.sort((a, b) => b - a)
  return scores.slice(0, 3)
}

It works. topThree([72, 98, 64, 91, 87, 55]) returns [98, 91, 87], which is correct. It throws nothing. It has a test, and the test is green.

Now watch what happens when real code calls it.

const names  = ["Ada", "Bo", "Cy", "Di", "Eve", "Fay"]
const scores = [  72,   98,   64,   91,   87,    55  ]

const pair  = (n, i) => n + " " + scores[i]
const board = () => names.map(pair).join("   ")

console.log("before:   ", board())
const top = topThree(scores)
console.log("after:    ", board())

before:    Ada 72   Bo 98   Cy 64   Di 91   Eve 87   Fay 55
after:     Ada 98   Bo 91   Cy 87   Di 72   Eve 64   Fay 55

Ada scored 72. Ada now has 98. Every name in that list is sitting next to somebody else’s score, and the function that did it also returned exactly the right answer.

Why the test could not have caught this

The obvious reaction is that the test was lazy. Write a better one.

That reaction is wrong in a way I think is worth analyzing, because it is the difference between “we got sloppy” and “this class of bug needs a different kind of attention.”

Array.prototype.sort sorts in place. It rearranges the array you handed it and then returns that same array. So the topThree function shown above has two effects: the value it returns, and the change it makes to its argument. The original test only ever looked at the first one.

A test of the return value cannot observe a side effect. Not because it was written badly, but because a return value is not where the side effect resides.

You could write an assertion that catches it. expect(scores).toEqual([72, 98, 64, 91, 87, 55]) after the call would go red immediately. But you only write that assertion if you already suspected the function touched its argument, and if you already suspected that, you would have fixed the function instead.

That is what makes this expensive. A bug that throws tells you where it is. A bug that fails a test tells you where it is. This one returns the right answer, passes review, and shows up three weeks later as a support ticket about the wrong name on a leaderboard.

The near miss that catches experienced people

The fix is to sort a copy instead of the original.

return [...scores].sort((a, b) => b - a).slice(0, 3)

Here is the version I find genuinely interesting, because it looks like the fix and is not:

return [...scores.sort((a, b) => b - a)].slice(0, 3)

Same characters, different order. The spread is on the outside, so .sort() runs first, on the caller’s array, and by the time the copy happens the damage is already done. It returns the correct answer and mutates, exactly like the original.

I have seen people who have been writing JS for a decade write that one, and its not a knowledge gap. It is that the two versions look identical at a glance and only one of them is about ordering of operations.

This is roughly why lodash orderBy exists

If you have ever wondered what a utility library is buying you when the language already has sort, this is a large part of the answer. _.orderBy(scores, [], ['desc']) sorts a copy and hands that back. The array you passed in is never touched, and that guarantee is the product.

The naming does not help anyone. sort, reverse, splice, push and pop all change the array you call them on. map, filter, slice and concat all give you a new one. Nothing in those names really tells you which is which, so it comes down to memory, and memory is exactly what fails at the worst times.

Why I turned this into the demo on my landing page

I launched CodeTrain three weeks ago. It is an AI tutor with one rule: it never writes your code. You type every line, it plans the steps, runs what you wrote and grades it.

The thing I got wrong was assuming that if people heard about it, the ones who wanted it would sign up. So I did the whole distribution playbook. What I actually had was a funnel where people arrived, hit an email field, and had no way to find out whether the thing was any good before handing it over.

So the lesson above is now the demo, and it runs on the landing page with no account and no install.

It is a real lesson, not a video of one. Your code executes in a real Web Worker in your own tab. The checks are real assertions against what your function actually returned and what it did to its argument, including one case you cannot see, because a fix that only works on the numbers in front of you is not a fix. Break the slice and the check goes red. Delete the function and you get a real missing-entry error.

The one thing that is not live is the tutor’s replies. Those are written in advance, the page says so before you start, and every reply is labeled as such. A preview that implies a live model when there is not one is a lie about the product, and I would rather lose the visitor than start there.

The bit I nearly shipped backwards

The first version of that demo had a problem I did not see until the fourth or fifth read.

Step one showed a passing check next to broken code, and the tutor said something like “the check did not catch it.” I meant it as a statement about tests in general. What it actually reads as, on my own product page, is that CodeTrain wrote a check that misses bugs.

The fix for the demo lesson was to reword the claim. The check in step one belongs to the codebase in the example, not to CodeTrain, and it is now labelled that way on screen for clarity. What CodeTrain checks is step two, and there are three assertions there including the one about the argument. The original test asked one question. The tutor asks three, and one of them is the question that matters.

Try it

The lesson is at codetrain.ai/#try. Two steps, about a minute, no account.

Two steps is a real lesson at the short end, incidentally. The tutor builds between two and six depending on how big the topic is. The difference in the real thing is that it reads whatever you actually wrote, so the questions come from your code instead of from a list I authored ahead of time. Ten lessons a month on the free tier, no card, and you can point it at any public repository and get a lesson built from the code that is really in it.