I audit things for a living — fraud and risk work, a CIA/CISSP track, no CS degree. So when people say “the model can fix real bugs now,” my reflex is the auditor’s reflex: show me the run, show me the grader, and let me poke the grader. This is the first of a weekly set of field notes on building an SWE-bench agent harness in the open. The scoreboard here is honest, including the parts that make me look worse.
What the task actually is
SWE-bench Lite is a 300-instance test split. Each instance is a real GitHub issue against a real repository, and you only “pass” if your patch applies cleanly and the project’s own test suite goes green. I started with instance #0: astropy__astropy-12907, a bug where separability_matrix computes the wrong result on nested compound models. Its gold patch is exactly one line:
- cright[...] = 1
+ cright[...] = rightThat one line taught me the whole shape of the problem. The issue text is long; the real fix is a single character of meaning. So the hard part is not writing code — it’s localization (finding that line in a repo of thousands of files) and emitting a diff that actually applies. The model itself is cheap: my first single-task solve cost $0.00107.
The naive baseline: 1 out of 20
Before crediting a harness with anything, you have to know what the model does with nothing. So I ran a bare baseline — the same deepseek-v4-flash, handed only the issue text and the repository name, asked for a unified diff. No localization, no file contents, no search/replace, no retries, no line-number computation.
On the first 20 tasks it resolved 1 of 20. And the failure mode was instructive: 11 of the 20 patches never applied at all. The model invented filenames and line numbers that didn’t exist, so git apply rejected the hunks outright (Hunk FAILED). Eight more applied but failed the tests — confidently wrong. Exactly one got lucky. That’s the floor. Everything above it is engineering.
The harness: three moves
Localization by paths, not contents. Instead of dumping file bodies into the prompt, I give the model the issue plus every .py path in the repo — pulled from the GitHub tree API, no full clone — and ask it to name the 1–3 files worth touching. On the first 15 tasks (astropy + django) it hit the right file top-1 on 14 of 15, 93% (hit@3 was identical). Honest caveats, carried straight from my log: n=15, only two repositories, and these issues frequently quote the offending module path themselves. Finding the file is genuinely not the bottleneck here — I don’t yet know if that holds on repos where the issue is vaguer.
Search/replace, not line numbers. Early on I noticed the model’s content fix was right but its hunk header was off by 22 lines — @@ -220 where gold said @@ -242. The root cause is a real asymmetry: models copy content well and count lines badly. So I stopped asking for line numbers. The model now emits “replace this exact block with that block,” and difflib computes the true line numbers. Same task, after the change: @@ -220 → @@ -242, the SEARCH block matches verbatim, and the patch is guaranteed to apply.
Retry with tolerant matching. When a SEARCH block failed to match, I first assumed a whitespace bug. Re-running the diagnostic showed the four “empty” tasks could all match — it was flakiness, not whitespace. The fix was general: retry the model up to 3 times, plus line-level rstrip-tolerant matching. That moved non-empty outputs from 16/20 to 20/20.
The result, with the caveats attached
On the same 20-task subset, the harness resolves 11 of 20 (9 unresolved, 0 empty, 0 error). The path there was not clean, and I’m not going to smooth it over. Version 1 of the harness scored 9/20. Adding retry + tolerant matching took it to 11 — but that move flipped 3 tasks green and knocked 1 back to red. Net +2: gain 3, lose 1. django-11179 was green in v1 and regressed in v2. Which is the honest headline caveat: this is a single run, the agent is flaky, and one run wobbles. On top of that, these are the first 20 of Lite — astropy and django, the easy end. The back half (sympy, sphinx, matplotlib) is much harder. 11/20 is a real baseline on this slice, not a Lite score.
Here’s the number that answers “so what did you add, versus the model?” Naive 1 vs harness 11 on the identical subset — a gap of 10. Of the 11 resolves, roughly 10 were earned by the harness (localization + fetching the file + search/replace + difflib + retry), not the model. About 91% of the result came from engineering. Same model, both times. The gap is the scaffolding.
The audit lens: trust, but verify the verifier
The part I’m proudest of isn’t the 11. It’s that I didn’t believe a number until I broke the thing producing it.
First submission: I ran the 20 through the pipeline, got a predictions.json (16 non-empty, 4 empty), and installed sb-cli — the free cloud grader that spares you local Docker. Got an API key, submitted. Every patch came back failed — not “unresolved,” not “error,” failed. Zero of twenty.
That smelled wrong, so I tested the instrument instead of trusting it. Two checks. First: locally, I cloned the real repositories and ran git apply — our patches went on cleanly, and for django-10914 the patch was byte-for-byte identical to the gold patch. Second, the decisive one: I submitted three official gold patches as if they were my own. The grader failed all three. A grader that fails the gold patch is not grading. (A side tell: the eval time didn’t scale with the number of tasks — it almost certainly wasn’t running Docker at all.) The agent was fine; the verifier was lying.
So I ran the evaluation myself, on my own Hetzner box (swebench 4.1.0, real Docker, max_workers 1 to avoid disturbing the 18 production containers sharing that machine). Smoke test on django-10914: resolved in 62 seconds — the exact task sb-cli had called failed. That’s the first real resolved, and the confirmation that the whole 0/20 was a false negative from a free grader. Trust, but verify the verifier — especially the free one.
What’s next
Two threads. First, variance reduction. The “gain 3, lose 1” wobble means I can’t yet claim a real improvement from a single run. Next steps are multi-seed / take-best on the repair step, and eventually real eval science — bootstrap confidence intervals — so that “better” means something you could defend. Second, the 9 unresolved: patches that apply but fail the tests, i.e. fixes that are confidently wrong. Those need more context, multi-file edits, and iterative repair. That’s the next hard bone.
Weekly notes. The scoreboard stays honest.