Long-Insight: A Platform for Long-running Agent Trajectory Analysis

Background: When Agent Trajectories Are Too Long to Read

When frontier LLMs tackle real software-engineering tasks, the resulting execution traces are staggeringly long. Take a recent Long-running benchmark of 370 trajectories as an example:

A single application_development trajectory averages 8.1M tokens — 61.9× a 128K context window. At least 85% of the 370 trajectories exceed any model’s context window.

This creates two concrete problems:

1. Humans can’t read them — no matter how patient an engineer, 400+ turns of interaction logs cannot be reviewed line by line.
2. Models can’t process them — even with million-token context windows, most trajectories still won’t fit.

Yet these trajectories contain extremely valuable signals. As we summarized while filtering SWE SFT data:

“It’s fine if there’s only some gold in the sand — but there can’t be broken glass.”

What we need is not blind selection of “good-looking” trajectories, but precise identification and filtering of bad patterns. To do that, we need tools that understand what these ultra-long trajectories are doing — and how well.

That is the origin of Long-Insight.

Core Idea

Long-Insight solves three problems:

1. Unreadable → decompose linear trajectories into a structured step DAG with types, summaries, and parent–child dependencies.
2. Doesn’t fit → smart compression that cuts 60–80% of tokens while preserving causal structure.
3. Hard to score → a two-stage LLM auto-scorer that quantifies trajectory difficulty and quality.

Part 1: Trajectory Step Decomposition

Design

• Initialization: create an empty JSON file.
• Loop: read trajectory turns one by one → call an LLM to analyze → decide “new step” vs. “continuation” → update the step DAG.
• Finalization: produce a complete step-partition JSON with 8 step types, causal narratives, and parent–child links.

Each step is classified into one of: Task Understanding, Project Exploration, Environment Setup, Code Implementation, Test Validation, Problem Debugging, Documentation, and Summary & Planning.

Macro-level Analysis

Below is one Sonnet 4.5 trajectory on SWE-bench:

Although the trajectory looks roughly linear at the macro level, a closer look reveals that the Agent isn’t simply “marching straight to the end.” It continuously cycles through diverge (gather information) → converge (summarize & plan) → trial-and-error (rollback) → execute again.

The whole trajectory naturally splits into six behavioral phases:

Phase 1: Environment Sensing and Baseline Building (Steps 1–24)

The Agent is decidedly test-first, spending substantial effort analyzing test_package.py and reverse-engineering requirements from test code rather than guessing blindly.

Phase 2: Strategy Adjustment and Replanning (Steps 25–34)

After the rollback at step 28, the Agent doesn’t rush back into coding. Instead, it enters a “test-discovery phase,” probing project state with non-invasive scripts.

Phase 3: Infrastructure Construction (Steps 38–51)

Before touching the core algorithms, the Agent fixes/implements low-level dependencies — for example, the Dimension class constructor and the utility function execute_decomposition_method.

Phase 4: Core Algorithms, One by One (Steps 52–79)

This is the longest stretch of the trajectory. The Agent adopts an “analogy-clone” strategy: first crack the hardest base class PDDP’s fit method, then quickly replicate it to subclasses DePDDP, IPDDP, KMPDDP, and BisectingKmeans. Each implementation is immediately followed by unit tests (TDD style).

Phase 5: Fixing Errors (Steps 80–110)

The Agent doesn’t just patch a single class — it systematically walks through every related class, uniformly adding input validation at the entry point of fit methods. This demonstrates the Agent’s awareness of global consistency.

Phase 6: Full Regression and Delivery (Steps 111–120)

Run the full test suite (all 57 cases pass), validate in real scenarios, generate delivery documentation, and submit.

Local Structure Analysis

Fan-in

Step 15 (Create TODO and OVERVIEW notes) has parents [12, 13, 14] — after separately searching function definitions, class definitions, and verifying data loading, the Agent fuses the scattered information into one project document.

Similar fan-in patterns:

Backtrace

Step 28 (Abort implementation and reassess) — the Agent executed git checkout or git reset --hard, representing pruning a dead-end branch. It realized the current path was wrong, cut the branch, and reverted to an earlier state.

Part 2: Trajectory Compression

Why Compression Is Mandatory

Token consumption varies dramatically across task categories:

Core issues: 85% of trajectories exceed any model’s context window, over-long inputs degrade the judging LLM’s performance, and API costs balloon.

Compression Strategy

Core principle: preserve everything relevant to the Agent’s decisions; drop system metadata and redundancy.

• Drop: metadata such as uuid, parentUuid, timestamp, sessionId, version; toolUseResult (internal system records invisible to the Agent).
• Keep in full: the Agent’s thinking, code outputs, and tool calls (including code, arguments, and Todo lists).
• Selectively truncate: user messages capped at 200 chars; tool results truncated when they exceed 200 chars.

Compression Results

Part 3: Automatic Scoring

Scoring System

We designed a two-stage scoring pipeline that evaluates trajectories along two dimensions: task difficulty and improvement potential.

Stage 1: Task Difficulty Score (0–10)

Inputs include the problem description, number of issues, evaluation results (resolved or not, patch status, test logs), total tokens, and total turns. Dimensions assessed:

• Inherent problem complexity (number of files involved, logical depth)
• Fix difficulty (how deep an architectural understanding is required)
• Clarity of the problem description
• Project complexity (issue counts as a proxy for project size)
• Empirical solving difficulty (token consumption, turn count, test outcomes)

Stage 2: Improvement-Potential Score (0–10)

The input is the compressed full conversation history messages. Higher scores mean higher-quality trajectories. We defined several stereotypical bad patterns — the more bad patterns observed, the lower the score:

• Test-respect — does the Agent take failing tests seriously?
• Validation-loop completeness — is the discover → analyze → fix → verify loop intact?
• Root-cause accuracy — did the Agent locate the actual root cause?
• Behavioral repetition — is it repeating ineffective operations?
• Exploration efficiency — is it exploring code productively?
• Error-handling capability — does it deal well with error messages?
• Reasoning quality — do thinking blocks contain substantive content?
• Trajectory stability — does it frequently drift off the main thread?
• Action effectiveness — are late-stage actions still meaningful?

Application

We auto-scored 12,839 Sonnet 4.5 trajectories on SWE-bench, at a cost of roughly ¥1 per trajectory.

Filtering thresholds derived from the scores:

These scores feed directly into the SWE SFT quality filter — removing bad patterns from low-scoring trajectories to improve training data quality, rather than introducing selection bias by difficulty or domain.