Long-Insight: A Platform for Long-running Agent Trajectory Analysis

Background: When Agent Trajectories Are Too Long to Read

When an agent with context compression is pointed at real software-engineering tasks, the execution traces it leaves behind are far longer than one would expect. Across a sample of several hundred trajectories, a single run routinely runs to tens of multiples of a 128K context window — the shortest cohort sits around 7×, the longest well past 50×. The heaviest class of task averages 8.1M tokens per trajectory, or 61.9× a 128K window, and at least 85% of the sampled trajectories exceed any model’s context window.

This creates two concrete problems:

1. Humans can’t read them — no matter how patient an engineer, 400+ turns of interaction logs cannot be reviewed line by line.
2. Models can’t process them — even with million-token context windows, most trajectories still won’t fit.

Yet these are exactly the traces worth studying. Making use of them takes tooling that can first read what an ultra-long trajectory is actually doing.

That is the origin of Long-Insight.

Core Idea

Long-Insight solves two problems:

1. Unreadable → decompose linear trajectories into a structured step DAG with types, summaries, and parent–child dependencies.
2. Doesn’t fit → smart compression that cuts 60–80% of tokens while preserving causal structure.

Part 1: Trajectory Step Decomposition

Design

• Initialization: create an empty JSON file.
• Loop: read trajectory turns one by one → call an LLM to analyze → decide “new step” vs. “continuation” → update the step DAG.
• Finalization: produce a complete step-partition JSON with 8 step types, causal narratives, and parent–child links.

Each step is classified into one of: Task Understanding, Project Exploration, Environment Setup, Code Implementation, Test Validation, Problem Debugging, Documentation, and Summary & Planning.

Macro-level Analysis

Below is one Sonnet 4.5 trajectory on SWE-bench:

Although the trajectory looks roughly linear at the macro level, a closer look reveals that the Agent isn’t simply “marching straight to the end.” It continuously cycles through diverge (gather information) → converge (summarize & plan) → trial-and-error (rollback) → execute again.

The whole trajectory naturally splits into six behavioral phases:

Phase 1: Environment Sensing and Baseline Building (Steps 1–24)

The Agent is decidedly test-first, spending substantial effort analyzing test_package.py and reverse-engineering requirements from test code rather than guessing blindly.

Phase 2: Strategy Adjustment and Replanning (Steps 25–34)

After the rollback at step 28, the Agent doesn’t rush back into coding. Instead, it enters a “test-discovery phase,” probing project state with non-invasive scripts.

Phase 3: Infrastructure Construction (Steps 38–51)

Before touching the core algorithms, the Agent fixes/implements low-level dependencies — for example, the Dimension class constructor and the utility function execute_decomposition_method.

Phase 4: Core Algorithms, One by One (Steps 52–79)

This is the longest stretch of the trajectory. The Agent adopts an “analogy-clone” strategy: first crack the hardest base class PDDP’s fit method, then quickly replicate it to subclasses DePDDP, IPDDP, KMPDDP, and BisectingKmeans. Each implementation is immediately followed by unit tests (TDD style).

Phase 5: Fixing Errors (Steps 80–110)

The Agent doesn’t just patch a single class — it systematically walks through every related class, uniformly adding input validation at the entry point of fit methods. This demonstrates the Agent’s awareness of global consistency.

Phase 6: Full Regression and Delivery (Steps 111–120)

Run the full test suite (all 57 cases pass), validate in real scenarios, generate delivery documentation, and submit.

Local Structure Analysis

Fan-in

Step 15 (Create TODO and OVERVIEW notes) has parents [12, 13, 14] — after separately searching function definitions, class definitions, and verifying data loading, the Agent fuses the scattered information into one project document.

Similar fan-in patterns:

Backtrace

Step 28 (Abort implementation and reassess) — the Agent executed git checkout or git reset --hard, representing pruning a dead-end branch. It realized the current path was wrong, cut the branch, and reverted to an earlier state.

Part 2: Trajectory Compression

Why Compression Is Mandatory

Token consumption varies dramatically across task categories:

Core issues: 85% of trajectories exceed any model’s context window, over-long inputs degrade the judging LLM’s performance, and API costs balloon.

Compression Strategy

Core principle: preserve everything relevant to the Agent’s decisions; drop system metadata and redundancy.

• Drop: metadata such as uuid, parentUuid, timestamp, sessionId, version; toolUseResult (internal system records invisible to the Agent).
• Keep in full: the Agent’s thinking, code outputs, and tool calls (including code, arguments, and Todo lists).
• Selectively truncate: user messages capped at 200 chars; tool results truncated when they exceed 200 chars.

Compression Results