论文标题
通过自适应网络分区进行交通信号控制的封建多机构增强学习
Feudal Multi-Agent Reinforcement Learning with Adaptive Network Partition for Traffic Signal Control
论文作者
论文摘要
多代理增强学习（MARL）已被应用并显示出在多交流信号控制中的巨大潜力，其中多种代理，每个交叉路口都必须合作以优化交通流。为了鼓励全球合作，先前的工作将交通网络分为几个地区，并在封建结构中学习代理商的政策。但是，静态网络分区无法适应动态流量，这会随着时间的推移而经常变化。为了解决这个问题，我们提出了一种具有自适应网络分区的新颖封建MARL方法。具体来说，我们根据交通流量将网络首先将网络划分为多个区域。为此，我们提出了两种方法：一种是直接使用图形神经网络（GNN）生成网络分区，另一个是使用Monte-Carlo Tree搜索（MCT）来找到使用GNN计算的标准找到最佳分区。然后，我们使用GNN设计了QMIX的变体来处理由动态网络分区给出的输入的各个维度。最后，我们使用封建层次结构来管理每个分区中的代理商并促进全球合作。通过这样做，代理可以根据实践中的要求适应交通流量。我们在文献中广泛使用的三个城市的综合交通网格和现实世界流量网络中，都在经验上评估了我们的方法。我们的实验结果证实，与流量信号控制的几种领先方法相比，我们的方法可以在平均旅行时间和队列长度上实现更好的性能。
Multi-agent reinforcement learning (MARL) has been applied and shown great potential in multi-intersections traffic signal control, where multiple agents, one for each intersection, must cooperate together to optimize traffic flow. To encourage global cooperation, previous work partitions the traffic network into several regions and learns policies for agents in a feudal structure. However, static network partition fails to adapt to dynamic traffic flow, which will changes frequently over time. To address this, we propose a novel feudal MARL approach with adaptive network partition. Specifically, we first partition the network into several regions according to the traffic flow. To do this, we propose two approaches: one is directly to use graph neural network (GNN) to generate the network partition, and the other is to use Monte-Carlo tree search (MCTS) to find the best partition with criteria computed by GNN. Then, we design a variant of Qmix using GNN to handle various dimensions of input, given by the dynamic network partition. Finally, we use a feudal hierarchy to manage agents in each partition and promote global cooperation. By doing so, agents are able to adapt to the traffic flow as required in practice. We empirically evaluate our method both in a synthetic traffic grid and real-world traffic networks of three cities, widely used in the literature. Our experimental results confirm that our method can achieve better performance, in terms of average travel time and queue length, than several leading methods for traffic signal control.
