Title: A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challenges

Institute: Beijing Academy of Artificial Intelligence (BAAI); Institute of Software, Chinese Academy of Sciences; University of Chinese Academy of Sciences; Dartmouth College; Beijing University of Posts and Telecommunications; Peking University

Authors: Xinrun Xu, Yuxin Wang, Chaoyi Xu, Ziluo Ding, Jiechuan Jiang, Zhiming Ding, Börje F. Karlsson

Arxiv Link: https://arxiv.org/abs/2403.10249

Date: 2024.3.15

Introduction

Large models (LMs), covering both language models and multimodal models, have made substantial progress in many important areas. That progress has prompted researchers to explore using LMs as agents that carry out complex tasks. Compared with conventional models, LM-based agents (LMAs) generalize considerably better in many settings.

The capabilities demonstrated by LM-based agents have generated strong interest in applying them to games. Games offer agents a virtual environment whose challenges demand complex reasoning and cognitive ability, making them an ideal benchmark for evaluating agents. Relative to real-world experiments, collecting data in a game environment offers unmatched advantages in cost, controllability, safety, and diversity, while still preserving a meaningful level of difficulty.

Before asking how to make large models play games, it is worth first considering how humans play them.

The human brain, as a complex information-processing system, first converts sensory input into perceptual representations, then uses those representations to build knowledge about the world and make decisions, and finally carries them out through action. This sequence can be abstracted into an iterative agent loop: perception, inference, and action.

• Perception: converting the raw information observed during gameplay into an actionable understanding of the game that supports subsequent interaction. Early work turned text into semantic information; more recent work attempts to bring visual information into the picture as well.
• Inference: this stage involves many of an agent’s key capabilities, including memory, learning, reasoning, reflection, and decision making.
  • Memory: storing and retrieving learned knowledge.
  • Learning: adjusting policies through experience and through skills acquired in multi-agent systems.
  • Reasoning: the process of solving problems by processing and aggregating information. In complex games, decision making requires multi-hop reasoning and long-horizon planning, task decomposition, and collaborative decisions to cope with dynamic game environments.
  • Reflection: the process of self-improvement, in which the agent evaluates and adjusts its strategy based on feedback.
• Action: covering interaction with the game environment, that is, the agent’s response to the game state and to environmental feedback.
  • Code generation: many agent behaviors are executed by generating code, using techniques such as iterative prompting, role-specific prompting, and code generation. GPT-4 in fact does this all the time. For example, if we ask GPT to analyze a table for us, it will write a Python snippet to read, analyze, and visualize the table, and then report the result back to us as text.
  • Dialogue interaction: dialogue covers not only agent-to-agent communication but also human-agent interaction. These interactions are realized through collaboration frameworks and dialogue control mechanisms that enable dynamic and effective communication, the goal being to flexibly adapt conversational behavior to the needs and context of the interaction.
  • Behavioral consistency: behavioral consistency means that the agent acts in a predictable, stable, and coherent pattern across different situations. Structured approaches such as DAGs, feedback mechanisms, and reinforcement learning can be used to achieve this.

Note that two notions Chinese renders with the same word are distinct in English: inference and reasoning. Inference emphasizes extending existing information logically and extracting conclusions from it, whereas reasoning covers a broader range of thinking and logical deduction.

The “world model” mentioned above can be understood as the knowledge about the world that the human brain builds from perceptual representations. Here the world model is the brain’s internal representation of the external environment — an understanding of objects, events, scenes, and the relations among them — which helps an individual make sense of their surroundings, predict future events, and, on that basis, decide and act.

Perception

Perceiving the raw scene of a game and extracting information from it is crucial to every step that follows. The main function of perception is to convert the multimodal space — visual, semantic, and auditory — into input for the agent.

Modern digital games integrate multimodal perceptual channels to give human players a richer, more immersive experience, letting them sink more deeply into the game world. Unfortunately, however, there is not much existing work on incorporating audio data into the training of large models or into the optimization of game agents.

Semantic Perception

Perception on the semantic side refers mainly to the textual elements and textual descriptions in a game, ranging from natural-language instructions to dialogue. It means that the player understands the various elements of the game environment and the events taking place — and makes decisions and acts — by reading and interpreting text.

Semantic perception can be divided into four categories:

• Simple text input: the user’s simple ideas or descriptions, basic game state variables, and dialogue between characters.
• Structured or role-based input: a character’s attributes, backstory, related information, skills, and so on.
• Environmental and context-specific input: detailed game descriptions and contextual information, as well as natural-language instructions for tasks.
• Multimodal input: integrating visual, auditory, and textual data for decision making, and combining dialogue, game state, and scripts to enable richer interaction.

Visual Perception

We mentioned above that multimodal information such as vision can be converted into text. Converting other modalities into text, however, may discard a great deal of important information, easily leaving the agent unable to understand the situation it faces or to navigate the game. LMAs therefore need more comprehensive perceptual data.

Multimodal large language models (MLLMs) have become one solution to this problem. With an MLLM, an agent can obtain a richer perception of its surroundings, which in turn supports more sophisticated cognitive processes and decision making in environments resembling the real world.

For this visual space, three distinct perception approaches exist today:

1. Obtaining game-related information through dedicated APIs and feeding that information to the agent.
2. Pre-training the model on image and action data.
3. Feeding in pure image information directly.

Inference

In the opening stage of a game, the agent needs to absorb basic commonsense and game-specific background knowledge, either through pre-training or through real-time perception. During gameplay, the agent synthesizes past game events, manages knowledge storage and retrieval, and performs core cognitive functions such as learning, reasoning, decision making, and reflection.

Memory

To correctly represent learned knowledge or past events and to use that information during inference, an agent must operate on these “memories” effectively. How to design a high-quality memory mechanism that lets an agent retrieve and store memories efficiently while fully exploiting the underlying large language model and respecting its constraints remains an open problem.

Commonsense

Commonsense knowledge is knowledge that people typically acquire early in life and that often goes unstated in a given situation; it is essential for reasoning and for avoiding misunderstandings.

Large language models, pre-trained on diverse internet text, already demonstrate advanced language understanding and reasoning, and can bring this implicit knowledge to bear on new challenges. That knowledge, however, may not always align perfectly with a specific application context such as a particular game scenario. Current research therefore aims to strengthen a language model’s commonsense understanding in the following ways:

1. Embedding structured knowledge through instruction fine-tuning.
2. Using an SOP to integrate commonsense into a specific role and task. For instance, if a model is used to play the role of a doctor, the SOP might contain medical commonsense, guidelines for communicating with patients, and standard procedures for diagnosis and treatment. This allows the model to act more effectively and adaptively in simulated or real situations, since it draws on the commonsense in the SOP to guide its decisions and behavior.
3. Using RAG (retrieval-augmented generation).

Game Background Knowledge

For large models, game background knowledge means an understanding of the various aspects of the game environment.

Current techniques incorporate basic game rules, basic procedures, reasoning ability, and one-shot demonstrations into the prompt. For example:

1. Feeding a detailed character profile to the large model through the prompt.
2. Using knowledge graphs to describe relations among entities.
3. Using detailed scenarios drawn from simulated environments and role-playing game (RPG) logs, which markedly improves the language model’s understanding of game dynamics.
4. Building an external knowledge base from the Minecraft Wiki and from documentation of item crafting/smelting recipes, creating an exhaustive source of knowledge about the Minecraft world for the agent to consult.

Retrieval

Memory retrieval in LMAs means strengthening reasoning and action by recalling learned information.

Memory retrieval methods for LMAs fall into three categories: structured memory systems, dynamic adaptive retrieval processes, and advanced memory retrieval techniques.

1. Structured memory systems: these techniques aim to organize and index information efficiently for fast retrieval. Skill libraries, shared message pools, hierarchical structures, and structured communication all exist so that an agent can quickly access previously acquired skills and knowledge, enabling effective memory management. A “summarize and forget” memory mechanism prioritizes critical memory items by summarizing important information and discarding the rest, thereby lowering computational cost.
2. Dynamic adaptive retrieval processes: this approach retrieves memories by combining relevance, recency, and importance, with strategic collection emphasizing the relevance and freshness of historical game information. The process also involves continually adjusting plans in light of new circumstances, ensuring that the agent’s behavior adapts as the game changes. Put simply, the approach updates and adjusts the agent’s memory in real time so that better decisions can be made.
3. Advanced memory retrieval techniques: these include (1) using long short-term memory (LSTM) to retain memories of past events, (2) generating more concise state descriptions to help the agent quickly grasp the current game situation, and (3) compressing historical text sequences to save space and improve efficiency. In short, these techniques help agents manage and use their memories more precisely and effectively.

Learning

Learning in the context of large models (LMs) involves interpreting and integrating information from a variety of sources, including training data, user interaction, and environmental feedback.

The learning process can be explicit — for example, fine-tuning on a specific dataset or task-focused training — or implicit, as when pre-existing knowledge is used to adapt to a new scenario.

1. Iterative and feedback-based learning: combines iterative prompting, role prompting, feedback mechanisms, and behavior cloning to strengthen capability through continuous feedback and adaptation.

2. Reinforcement learning and offline learning: includes reinforcement learning with environmental feedback, offline learning, and reinforcement learning driven by language instructions. Whereas iterative feedback-based learning focuses on optimizing model behavior through continuous real-time feedback, this approach focuses on maximizing long-term reward or achieving a specific goal through interaction with the environment, whether live or based on historical data.

3. Experience-based learning with memory: focuses on drawing on past experience, memory retrieval, and experience pools to guide current and future actions.

4. Collaborative and adaptive learning: emphasizes learning through multi-agent cooperation, adaptive strategies, and interactive learning processes.

Reasoning

Reasoning involves abstracting the foundational knowledge acquired during the learning stage into higher forms of understanding, via deductive, inductive, or abductive inference.

Decision Making

In the vast worlds of games, particularly in open-world and immersive environments, players and AI agents face complex decision and action spaces.

These spaces contain an enormous number of possible actions, interactions, and decisions, with a wide range of characters, item uses, and skill combinations. Multi-agent planning can add yet another layer of complexity, requiring coordination and collaboration among multiple agents under incomplete information.

Multi-hop Reasoning

Multi-hop reasoning refers to the cognitive process of considering multiple layers of information and dependencies before making a decision.

In a strategy game, for example, the agent must assess the current situation, anticipate the opponent’s moves, evaluate the potential outcomes of various actions, and only then settle on the most advantageous course.

Long-horizon Planning

Long-horizon planning refers to the strategic process by which an agent sets a goal and determines the actions that will achieve it over an extended period.

Some researchers have proposed using chain-of-thought (CoT) planning to imitate human reasoning, generating the intermediate steps that lead to a final decision, where each step builds on the previous one until the goal is reached.

Reflection

Reflection refers to the ability of LM-based agents (LMAs) to assess, evaluate, and possibly adjust their own processes, decisions, or outputs as a task unfolds.

• Feedback-based self-improvement: the process of self-verification, analysis, and iterative enhancement using external feedback such as environmental responses or code execution results.
• Iteratively refining plans and actions: methods that continually refine plans or actions based on feedback and new information, the better to achieve or adapt to a goal.
• Collaboration and multi-agent interaction: emphasizes collaboratively re-evaluating and adjusting decisions within a multi-agent system.
• Debate and theory-of-mind-based reflection: uses a debate or exchange format as a model of the reflection process.

Action

The action space in which LMAs operate within games can be classified into three distinct types.

1. Text-based interaction
2. Interaction through APIs
3. Direct control via IO operations, such as keyboard and mouse.

Behavior

LMAs perform specific operations through generative programming techniques. Iterative prompting combined with environmental feedback, execution errors, and self-verification continually refines the generated program, ensuring its validity and relevance to the task at hand.

Dialogue

LMAs interact with humans or with other agents by taking into account the current environment and past experience.

Examples include agent-agent interaction and human-agent interaction.

Past research has shown that LMAs exhibit human-like social behavior in collaboration, such as teamwork, leadership, persuasion, deception, and confrontation.

Consistency

Action consistency means that an agent’s actions must remain coherent and aligned with its goals.

To maintain such consistency, the agent takes past experience and the current situation into account, exploits feedback mechanisms, and revises its actions and plans when necessary.

Concrete strategies include:

• Look-ahead checking mechanisms: assessing whether an action is feasible.
• Reinforcement learning (RL): training a policy that selects the most appropriate action.
• Self-consistency (SC): generating multiple action predictions and selecting the most frequent one for consistent decision making, as in the PokéLLMon case.

Structural approaches include:

• Using a directed acyclic graph to control the progression of actions.
• Leveraging a skill library.
• Defining structured actions with clear semantics.
• Assigning specific roles to agents and following a structured workflow.
• Neuro-symbolic methods.

Challenges

Hallucination

An LM can produce output that does not match the source material or the facts, a phenomenon usually called hallucination and one that is very hard to eliminate.

Introducing critic agents, SOPs, and role-specific prompts can reduce logical hallucinations.

Researchers have also tried in-context reinforcement learning (ICRL), retrieval-augmented generation (RAG), multi-agent debate, and other methods to address hallucination.

Error Correction

Here, much research adopts iterative prompting mechanisms, feedback mechanisms, or chat chains among agents to identify and correct errors.

Methods such as using critic agents for selection and feedback, and iterative prompting with environmental feedback, are employed for error correction.

Generalization

Generalization refers to an agent’s ability to apply the knowledge, skills, and capabilities acquired in past environments to new, previously unseen environments so that it can still achieve goals and carry out tasks.

Proposed solutions include context-aware chain-of-thought (CoT) reasoning, GITM (Ghost in the Minecraft), which exploits long-horizon planning and external knowledge in Minecraft, and information extraction and collaborative planning in multi-agent settings.

Interpretability

Some studies use CoT processes, modular decision processes, and clear conversational programming paradigms to make agent behavior more interpretable.

Future Directions

Multimodal Perception

Strengthening multimodal capability, including visual and auditory perception, may allow agents to handle more complex tasks.

It is worth noting that sound effects in games are essential for perceiving context, understanding tasks, and receiving feedback, yet very little literature explores how auditory information can be used to improve the performance of game agents.

Authenticity of the Game Experience

People prefer human-written content to dialogue and ideas generated by large language models. This suggests that future research needs to tie LLM-generated content more closely to the game’s narrative and state.

External Tools

LM-based agents have yet to master the use of external tools to improve their own performance in games.

Real-time Performance

Given the inference process and computational demands inherent to large models, mastering real-time, high-speed games poses a formidable challenge.