Does a More Complex Agent Deliver Better Performance?

TL;DR: On SWE-bench Pro, the more elaborate SWE-agent underperforms mini-swe-agent and suffers from instances that hang indefinitely.

PS: less than 20% of this article is AI-generated.

Let me start with a common intuition: the more complete an agent framework is, the stronger it should perform. Everyone seems to take this for granted, yet nobody has rigorously demonstrated it. I therefore wanted to check whether the intuition actually holds on SWE tasks 🤔.

💡 The idea is straightforward:

Evaluate two agent frameworks of differing complexity on SWE-bench Pro and compare their scores.

The outcome, however, was that the simpler agent framework scored higher 🤯.

With the background in place, the question becomes: under the same model and the same benchmark, does SWE-agent’s “more complete” engineering pay off relative to mini-swe-agent?

The more complete SWE-agent is not the stronger one

Using Claude Sonnet 4.5, I ran both agent frameworks over all 731 problems of SWE-bench Pro, capping the number of calls at 50 per problem.

The results:

For reference, the officially reported result for Sonnet 4.5 is around 43.6%¹; mini’s 44.0% matches it closely, which lends credibility to this setup.

The minimalist mini-swe-agent comes out 2.7 points ahead of SWE-agent. More surprisingly, after SWE-agent had worked through 722 of the 731 instances, the final 9 hung outright — containers stayed up for 5 to 12 hours with no log activity for hours on end, and had to be killed. Running those same 9 problems, mini-swe-agent showed no such issue.

For me, the surprise in the numbers mattered less than the question of why those nine instances hung 🤔.

Why did 9 containers hang?

Before killing the containers, I grabbed the docker logs of each one, which showed:

INFO  ... 200 OK  POST /run_in_session
🦖 ERROR Bashlex fail: here-document at line 0 delimited by end-of-file (wanted "'EOF'")

The containers were not dead and the swerex-remote process was still returning 200 OK — the agent was simply spinning in place. Following the agent’s trace, it was writing a large file with a heredoc:

cat > some_file.go <<'EOF'
... a large block of Go code ...
EOF

That pins down the root cause: SWE-agent’s execution backend, swe-rex, first parses every command with bashlex (a bash parser written in pure Python) before sending it into the container.

And that is exactly where the trouble lies. bashlex’s support for heredocs is incomplete; when it encounters a large block write such as cat <<'EOF' … a large block of code … EOF, parsing fails outright and raises Bashlex fail.

Once parsing collapses, swe-rex cannot tell whether the command finished or what its exit code was. The agent receives a broken observation, does not rescue itself by trying a different formulation, and just retries the same command again and again — leaving the container stuck for 5 to 12 hours.

Why does SWE-agent go to the trouble of parsing commands first?

swe-rex keeps a long-lived shell session alive so that state such as the working directory, environment variables, and any activated virtual environment carries over from one command to the next.

The price is that when commands run inside a continuously flowing session, “where a command ends and what its return code is” is no longer as self-evident as it is when a standalone process finishes. The only way out is to have bashlex split the command apart and inject sentinel strings so the exit code can be fished back out of the output stream.

The upside is that once a command has been parsed into a structure, the backend can layer on safety checks, command rewriting, and other fine-grained wrapping — precisely the Agent-Computer Interface idea that SWE-agent champions. It spends extra complexity to buy stronger session semantics.

mini-swe-agent goes the opposite way, taking the minimalist route. It maintains no session whatsoever; every command is handed straight to the system’s real shell with a single subprocess.run(shell=True):

# minisweagent/environments/local.py
result = subprocess.run(
    command,
    shell=True,     # hand it to the system shell (/bin/sh -c)
    text=True, cwd=cwd, timeout=timeout,
    stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
)

Doing so throws away session state — every command starts from scratch, and the agent has to spell out paths and environment itself — but for that very reason it sidesteps every headache of parsing bash yourself. A heredoc, however large, is the real shell’s own business; once the command finishes and the process exits, the exit code is simply there.

So the same large-file-writing command that runs into bashlex’s limitation and drags the container to a halt under SWE-agent passes through mini-swe-agent without incident. This is a textbook engineering trade-off: swe-rex spends more complexity on stronger session semantics and thereby takes on an extra class of failures, while mini gives up the convenience of a session in exchange for a smaller, more controllable space of errors.

Closing thoughts

In a broad sense, this finding is a nod to Occam’s razor: the simpler thing turns out to be the more effective one (perhaps a first-principles matter as well).

That said, I regard this as a very simple toy experiment. It does not establish that more complex, more sophisticated agents perform poorly; it may just be that SWE-agent happens to have this one bug. A carefully tuned, more elaborate agent could well beat a mini agent.

Either way, these are only conjectures for now. Next I plan to dig into the opening question with more rigorous experiments — stay tuned if you are interested.

Appendix

Per-language comparison

Almost all of mini’s advantage comes from Go, where it leads by 6 percentage points, a difference of 17 problems — and within Go, a single repository, gravitational/teleport, accounts for most of it (16 solved only by mini versus just 5 solved only by SWE-agent). This is not really surprising: Go problems tend to be large repositories with substantial edits, exactly where SWE-agent’s heredoc hang is easiest to trigger. Switch to Python, though, and SWE-agent is actually 2 points ahead.

Paired significance

Next, paired significance. Aligning all 731 problems one-to-one by instance_id splits them into four groups:

What genuinely separates the two are the 134 problems solved by exactly one side, of which mini takes 77 and SWE-agent 57. The tilt does favor mini, but McNemar’s exact test gives p = 0.10, which falls short of significance. The other two groups are more telling: 245 problems both can solve and 352 neither can, showing that the problems they cover overlap heavily.

Fairness accounting for the hangs

Finally, a fairness check. Going through the 9 hung instances one by one (5 from flipt, 2 from teleport, plus one each from vuls and tutanota), only 2 really count as “unfair” losses — the problems mini solved on which SWE-agent was scored 0 purely because it hung: vuls e4728e38 and teleport 47530e1f. Even crediting those 2 back, SWE-agent only climbs from 302 to 304 (41.6%), while mini stays at 44.0%; the gap actually narrows, and the “not significant” conclusion is entirely unchanged. Besides, looked at another way, the stability of the execution backend is itself part of an agent’s end-to-end capability, so under a “measure the whole scaffold” standard, scoring 0 is hardly a miscarriage of justice 😅.