How I built a tutor that refuses to write your code

Two weeks ago I launched CodeTrain, an AI tutor with one rule: it never writes your code. You type every line, it plans the steps, runs what you wrote, and grades it.

People probably assume the rule is a prompt. Something like “do not write code for the user,” pasted at the top of a system message, and done. That was the first version. It lasted about an hour.

What follows is what the rule actually cost to build: a grading contract instead of a chat reply, two separate model calls that must never be merged, two execution paths so the free tier costs almost nothing to run, and a short list of things that broke in front of real users.

A refusal is not prompted, its contracted

Here is the failure mode nobody warns you about. Ask a model to teach rather than solve, and it agrees enthusiastically. Then the learner gets stuck, and the model helps. It writes “you could try something like this” and drops in four lines. Technically it did not actually solve the exercise. Practically the lesson is over..

The fix to this was to stop asking for prose and start requiring a decision. Every grading turn returns JSON with a fixed shape, or it is treated as a failure:

{
  "verdict": "advance" | "retry" | "comment",
  "feedback_md": "<concise Socratic markdown>",
  "checks": [{"label": "prints exactly Hello", "pass": true}],
  "learned": ["<short concept the learner demonstrated>"]
}

That single change did more for the product than any amount of instruction tuning in the prompt. A model writing prose has infinite room to be helpful in the wrong direction. A model that has to pick advance or retry, and list the criteria it checked with a boolean next to each, has to commit to a judgment about the learner’s code.

The checks array is the part learners actually respond to. Each item is one concrete criterion with pass or fail, and the prompt requires failing criteria to be included rather than quietly dropped. You see exactly which two of five things your code did, which is a very different experience from a paragraph explaining that you are on the right track.

Keep the two calls apart

Lesson authoring and grading are separate calls, with separate prompts, and merging them is the single worst thing you can do to a tutor like this.

I know because I tried it. One call, full context, plan the next step and grade the last one at the same time. It saves a round trip and it is meaningfully cheaper. It also produces a tutor that could write the answer into the question. When the same call that just saw a struggling learner also composes the next step, the step it writes gets suspiciously specific. “Now add the except KeyError branch that returns an empty list” is just the solution with a task label on it.

Separated, the authoring call never sees the failure and cannot overfit to it. That constraint is now enforced in the codebase itself: control-plane bookkeeping stays in the router, the tutor engine stays synchronised with the agent’s copy of the prompt, and a captured-prompt test fails the build if the two drift apart.

Two execution paths, and why the free tier does none of the work

A tutor that grades code has to run code. The obvious design puts a container behind every Run button, and it is the reason a lot of similar products have no free tier, or one with a queue in front of it.

Python and JavaScript run entirely in the learner’s browser. Python goes through Pyodide in a dedicated worker, JavaScript runs in its own worker, and both stay pooled so the second Run does not pay startup again. A prewarm kicks off when the lesson opens, which usually finishes before anyone types their first line. Runs are capped at 20 seconds, which is generous for a teaching step and short enough that a runaway loop does not hang the tab.

On the free tier, the Python and JavaScript a learner writes never reaches a server, which is the only reason the tier can exist without a card and without a queue.

— free tier economics

That holds for the two languages most lessons use. bash, ruby, perl and php genuinely need a real interpreter and a filesystem, so those go to an isolated server sandbox with per-minute and per-day caps, on free accounts too. The browser runtimes cover the common case; the sandbox covers what a browser cannot honestly fake.

The measured cost tracks either way. Across a recent two-week window the managed model spend for every free-tier session on the platform came to about 25 cents total, and compute for running Python and JavaScript was zero, because it happened on the learner’s machine in a tab they already had open.

The languages that cannot run in a browser fall back to a real sandbox on our infrastructure: bash, ruby, perl and php, with a real shell and real files, no network, and per-user rate and daily caps. That sandbox is Alpine with BusyBox, which produced its own class of bug. The grader would demand a GNU coreutils flag that BusyBox does not implement, then fail a learner whose solution was correct. The prompt now names that constraint directly and instructs the grader to accept any working BusyBox-compatible approach.

Dockerfile lessons get static linting instead of a real build. Letting anonymous users build containers on my infrastructure is a speedrun to an incident writeup, and the lesson value was in the file, not the image.

Things that broke in front of real users

If you are going to skip ahead to anything in this article, here is the interesting stuff.

The model emitted JSON that was not JSON. Specifically, feedback containing a stray backslash, usually from a regex or a Windows path in the learner’s code, which invalidated an otherwise perfect verdict. The parser now repairs unescaped backslashes before parsing, and falls back to a retry carrying the raw text if it still cannot be read. A tutor that hard-fails because a learner typed \d is not a tutor that can function well.

Older model output used plain strings for checks instead of objects with a label and a boolean. Rather than break those, the coercion layer treats a bare string as a passing check. It is not elegant. It is the difference between rendering something useful and rendering an error.

The tutor asked permission to continue. Early versions ended a correct submission with “ready to move on?” Every single time, learners answered the question instead of coding, and the lesson turned into a conversation. Advancing is now the confirmation, stated as a hard rule in the prompt: when the submission is correct, advance, do not ask.

Browser Python is not Python. No subprocess, no real files, no network. A learner following an authored step involving subprocess was hitting a wall the tutor had built for them. The grader is now told to never fault the learner for a runtime limit, because that step should not have been written in the first place, and to guide toward a runnable approach or simply advance. I will be improving this behaviour into the future as well.

Repo mode and the patch at the end

The free tier teaches on examples and any public repository you point it at. Paid repo mode runs against a real checkout on your machine, which raises a question I got wrong at first: what happens to the code you wrote?

The first design applied each step as its own commit. It felt responsive and it was a mistake. Nobody wants seven commits titled “step 3” in their history, and a half-finished lesson left the working tree in a state the learner had to clean up.

Now the lesson holds everything and proposes one patch at the end, containing only the code you typed, against a branch you choose. You review it like any other diff. If you abandon the lesson, nothing touches your repository at all.

What I’m tuning next

The tutor is at its best on a focused concept in code it can read directly. Lesson scope is the dial still getting tuned in: a tight lesson on one module lands harder than one that ranges across a whole framework, so more of the authoring work is going into keeping lessons narrow by default.

Browser Python runs on Pyodide, whose package set is a subset of PyPI. The prompt-level guardrails already steer authored steps toward what the runtime can actually import, and widening that coverage, including routing more lesson types to the server sandbox where it makes sense, is the next piece of that work.

The grading loop is a model making a judgment, so the design choice was to make the judgment inspectable. Every criterion is listed with a pass or fail rather than one overall score, which means you can always see the basis for a verdict instead of blind faith in the model itself. That visibility is also our own best feedback channel: every reported step comes back with the checks attached, which is how the tuning above gets prioritised.

Try to break it

The free tier runs in your browser, ten lessons a month, no card and no trial clock: codetrain.ai. Point it at a public repository you did not write and ask for a lesson on the part you understand least. That is the fastest honest test of everything above.

If you talk it into writing your code outright, send me the screenshot. I will fix the loop, and probably frame the screenshot!

— Ethan L.