更复杂的 Agent 能带来更好的性能吗？

TL;DR. On SWE-bench Pro, the more elaborate SWE-agent underperforms the minimalist mini-swe-agent, and additionally suffers from instances that hang indefinitely.

A widely held intuition holds that the more complete an agent framework is, the better it should perform. Although this assumption is rarely questioned, to my knowledge it has never been rigorously verified. I therefore set out to test it on a software-engineering (SWE) task.

The design is simple: evaluate two agent frameworks of differing complexity on SWE-bench Pro and compare their scores. The result was counterintuitive: the simpler framework achieved the higher score.

With this background in place, the central question becomes: under the same model and the same benchmark, does SWE-agent’s heavier engineering actually translate into a higher payoff than mini-swe-agent?

The more elaborate SWE-agent is not stronger

Using Claude Sonnet 4.5, I evaluated both frameworks on the full 731-problem SWE-bench Pro public set, capping the call budget at 50 per problem.

The results are as follows:

For reference, the officially reported figure for Sonnet 4.5 is around 43.6%¹; mini’s 44.0% aligns closely with it, which lends credibility to the present setup.

The minimalist mini-swe-agent is, in fact, 2.7 percentage points higher than SWE-agent. More surprisingly, after SWE-agent had completed 722 of the 731 instances, its final 9 instances hung outright: the containers stayed up for 5 to 12 hours with no log activity for hours on end, and had to be killed manually (and thus scored 0). Running the very same 9 problems, mini-swe-agent exhibited no such behavior.

For me, beyond the surprise in the numbers, the more intriguing question was why these 9 instances hung in the first place.

Why did 9 containers hang?

Before killing the containers, I captured the docker logs of each one:

INFO  ... 200 OK  POST /run_in_session
🦖 ERROR Bashlex fail: here-document at line 0 delimited by end-of-file (wanted "'EOF'")

The containers were alive and swerex-remote was still returning 200 OK; in other words, the agent was merely spinning in place. Tracing the agent back, it was writing a large file via a heredoc:

cat > some_file.go <<'EOF'
... a large block of Go code ...
EOF

This pins down the root cause: SWE-agent’s execution backend, swe-rex, first parses every command with bashlex (a pure-Python bash parser) before dispatching it into the container.

That is precisely where things break. bashlex’s support for heredocs is incomplete; confronted with a large block-write such as cat <<'EOF' … a large block of code … EOF, it fails outright and raises Bashlex fail.

Once parsing collapses, swe-rex can no longer determine whether the command finished or what its exit code was. The agent receives a malformed observation, does not recover by attempting a different approach, and simply retries the same command over and over, leaving the container hung for 5 to 12 hours.

Why does SWE-agent bother parsing commands in the first place?

The answer follows from its design goals. swe-rex maintains a long-lived shell session, so that state such as the working directory, environment variables, and any activated virtual environment persists across commands.

The cost, however, is that when commands run inside a continuously flowing session, “where a command ends and what its return code is” is no longer as self-evident as it is when a standalone process exits. swe-rex must therefore rely on bashlex to split commands and inject sentinel strings in order to recover exit codes from the output stream.

The benefit is that, once a command has been parsed into a structured form, the backend can additionally perform safety checks, command rewriting, and other fine-grained wrapping. This is exactly the Agent-Computer Interface philosophy that SWE-agent champions; on its own terms, the design is sound, trading extra complexity for stronger session semantics.

mini-swe-agent takes the opposite, minimalist route. It maintains no session at all; every command is handed directly to the system’s real shell through a single subprocess.run(shell=True):

# minisweagent/environments/local.py
result = subprocess.run(
    command,
    shell=True,     # hand it to the system shell (/bin/sh -c)
    text=True, cwd=cwd, timeout=timeout,
    stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
)

This sacrifices session state (each command starts afresh, and the agent must spell out paths and environment itself), but for exactly that reason it sidesteps all the trouble of parsing bash in-process. A heredoc, however large, is the real shell’s native job; once the command finishes and the process exits, the exit code is simply there.

As a result, the same large-file-writing command that crashes the container under SWE-agent, by hitting bashlex’s limitation, runs uneventfully under mini-swe-agent. This is a textbook engineering trade-off: swe-rex buys stronger session semantics at the price of an additional failure surface, whereas mini forgoes the convenience of a session in exchange for a smaller and more controllable space of errors.

Conclusion

Broadly speaking, this finding echoes Occam’s razor: when capability is comparable, the simpler solution is often the more effective, and arguably the more robust, one.

That said, this is admittedly a small toy experiment. It does not establish that more complex, more sophisticated agents are necessarily worse; SWE-agent’s shortfall here is, to a large extent, dragged down by one specific bug. A carefully tuned, more elaborate agent could well surpass mini.

These remain, for now, preliminary conjectures. In follow-up work I intend to investigate the opening question more rigorously: whether a more complete agent is genuinely worth it. Stay tuned.

Appendix

Per-language comparison

mini’s advantage comes almost entirely from Go, where it leads by 6 percentage points (17 problems); within Go, a single repository, gravitational/teleport, accounts for most of it (16 solved only by mini versus 5 only by SWE-agent). This is unsurprising: Go problems tend to involve large, heavily modified repositories, which is precisely where SWE-agent’s heredoc hang is most likely to be triggered. On Python, by contrast, SWE-agent is actually 2 points higher.

Paired significance

Aligning all 731 problems by instance_id yields four categories:

What actually separates the two are the 134 problems solved by exactly one side: 77 for mini and 57 for SWE-agent. The tilt favors mini, but McNemar’s exact test yields p = 0.10, which is not significant. More telling are the other two categories: 245 problems solved by both and 352 solved by neither, indicating that the two scaffolds cover a highly overlapping set of problems.

Fairness accounting for the hangs

Finally, a fairness check. Of the 9 hung instances (5 from flipt, 2 from teleport, and one each from vuls and tutanota), only 2 constitute genuinely “unfair” losses, i.e., problems that mini solved but on which SWE-agent was scored 0 purely because it hung: vuls e4728e38 and teleport 47530e1f. Even crediting those 2 back, SWE-agent rises only from 302 to 304 (41.6%), while mini remains at 44.0%; the gap narrows but the “not significant” conclusion is unchanged. Moreover, the robustness of the execution backend is itself part of an agent’s end-to-end capability, so scoring 0 is not unfair under a “whole-scaffold” evaluation.