VillagerBench is a multi-agent cooperation evaluate platform in Minecraft, support more than 40 APIs for minecraft agents to interact with the world and cooperate with each other, we have designed three scenarios: construction building tasks, farm to table cooking tasks and escape the room challenge.
Abstract
In this paper, we aim to evaluate multi-agent systems against complex dependencies, including spatial, causal, and temporal constraints. First, we construct a new benchmark, named VillagerBench, within the Minecraft environment. VillagerBench comprises diverse tasks crafted to test various aspects of multi-agent collaboration, from workload distribution to dynamic adaptation and synchronized task execution. Second, we introduce a Directed Acyclic Graph Multi-Agent Framework (VillagerAgent) to resolve complex inter-agent dependencies and enhance collaborative efficiency. This solution incorporates a task decomposer that creates a directed acyclic graph (DAG) for structured task management, an agent controller for task distribution, and a state manager for tracking environmental and agent data.
Our empirical evaluation on VillagerBench demonstrates that VillagerAgent outperforms the existing AgentVerse model, reducing hallucinations and improving task decomposition efficacy. The results underscore VillagerAgent’s potential in advancing multi-agent collaboration, offering a scalable and generalizable solution in dynamic environments.
BenchMark
- Introduce a new multi-agent benchmark focusing on complex dependencies in Minecraft. As we discussed above (see our responses to Q1), our work distinguishes itself by focusing on complex multi-agent interactions within the Minecraft environment. This is a underexplored area compared to the prevalent single-agent or task-specific benchmarks. VillagerBench offers a fresh perspective for evaluating multi-agent teamwork in environments that are both complex and dynamic.
VillagerAgent Framework
- Introduce a new DAG-based multi-agent framework to tackle complex dependencies in multi-agent collaborations. Our VillagerAgent framework consists of a Task Decomposer that breaks down tasks into graph nodes, an Agent Controller responsible for assigning agents to nodes, and a State Manager in charge of managing experience and state information. Our approach enables the decomposition of tasks into sub-task nodes, differing from previous methods that fix agents in specific locations or execute steps simultaneously according to rules. Our DAG structure accurately describes the sequential and parallel order and dependencies of different sub-tasks, allowing agents to precisely execute synchronous and sequential sub-tasks, thereby enhancing the degree of multi-agent cooperation.
Other Demo
Demo in Overcooked Benchmark
@article{Dong2024,
title={VillagerAgent: A Graph-Based Multi-Agent Framework for Coordinating Complex Task Dependencies in Minecraft},
author={Yubo Dong, Xukun Zhu, Zhengzhe Pan, Linchao Zhu, Yi Yang},
journal={Journal Name},
volume={xx},
number={yy},
pages={zz-aa},
year={2024},
publisher={Publisher}
}