The task is multi-agent pathfinding on grid environments, where multiple agents must reach their goals while avoiding obstacles, other agents, and path conflicts. The motivating applications include ports, airports, and warehouses containing many mobile robots. Centralized planners can use global information to produce efficient paths, but their computational demand and planning time can grow as the number of agents increases. Decentralized planners scale better to large systems, yet local information can lead to insufficient cooperation, lower success rates, and reduced overall efficiency. The paper frames this tension as a suitable target for multi-agent reinforcement learning.
The main contribution is CoRS, a cooperative reward-shaping technique built around Independent Q-Learning. It evaluates how one agent affects neighboring agents and incorporates that interaction into the reward, encouraging behavior that considers more than the agent’s own immediate objective. Cooperation is introduced during training while execution remains distributed, distinguishing the approach from methods that depend on elaborate communication mechanisms. The paper combines CoRS with DHC, calling the resulting system CoRS-DHC, and compares it with DHC and other state-of-the-art planners. The supplied excerpt does not provide the complete formal contribution list or every mathematical detail, so claims beyond this description are not established.
MAPF is modeled as a partially observed multi-agent reinforcement-learning problem in which each agent selects actions from local observations. The environment uses four-neighbor grid movement; agents have a field of view and can communicate with up to two nearest neighbors during the reported experiments. The base reward gives different values to movement toward or away from the goal, staying on or off the goal, collisions, and task completion. The reported values are minus 0.070 for moving toward the goal, minus 0.075 for moving away, 0 for staying on the goal, minus 0.075 for staying off the goal, minus 0.5 for a collision, and 3 for finishing. CoRS modifies the reward to account for neighborhood influence, aiming to discourage blocking or colliding with other agents. The method uses the DHC network structure and fine-tunes a previously obtained optimal policy network rather than training the network entirely from the initial state.
Training and evaluation are conducted in a standard MAPF environment. Curriculum learning begins with a simple one-agent task; after the success rate exceeds 0.9, the procedure either adds an agent or increases the environment size by 5, eventually training for a setting with 10 agents. Episodes have a maximum of 256 steps, and training is reported to use batches of 192, sequence length 20, a changing learning rate, and at most 500,000 training steps. Sixteen independent environments generate experience in parallel, which is placed into a global buffer for GPU-based learning. Evaluation uses 200 test cases per scenario with obstacle density 0.3, maps of 40 by 40 and 80 by 80, and agent counts extending from 4 to 128. The compared measures include success rate, average steps, and makespan, with DHC, DCC, and PRIMAL among the referenced baselines.
The reported ablation comparison shows that CoRS-DHC outperforms the original DHC policy in every test set. In high-density settings with 64 and 128 agents, the paper states that CoRS-DHC improves pathfinding success rate by more than 20 percent relative to DHC. Its average-step results are also lower than DHC on both reported map sizes across the listed agent counts. Against DCC, CoRS-DHC is reported to exceed success rate in most scenarios and to have comparable or shorter makespan except for the 64-agent grid. This is presented as a favorable trade-off because DCC uses a substantially more complex communication mechanism during training and execution, whereas CoRS adds reward shaping during training. However, reward dependencies on neighbors may harm training stability or the efficiency of the converged policy, and the excerpt does not establish how often or under exactly which conditions this failure occurs.
MAPF seeks collision-free routes for multiple agents and is commonly applied to grid abstractions of multi-robot environments. The paper distinguishes centralized algorithms, which exploit global information, from decentralized algorithms, which prioritize scalability and local decision making. MARL can learn distributed policies from partial observations and is therefore attractive for large agent populations. Centralized-training distributed-execution methods can use global information during training, while distributed-training distributed-execution methods lack that information and may encourage agents to maximize only their individual rewards. The related-work discussion mentions PRIMAL, DHC, DCC, IQL, imitation learning, causal reward ideas, and Shapley-value-based cooperation, but the supplied excerpt does not give complete specifications for these methods.
Independent Q-Learning is scalable to many agents but often performs poorly when the task requires strong coordination. In MAPF, an action that is locally beneficial can block another agent’s route or create a collision, even if it improves the acting agent’s own reward. This mismatch between individual optimization and system-level traffic motivates a reward that reflects neighborhood effects. CoRS is intended to reduce this mismatch without requiring centralized execution or a complex communication policy.
CoRS measures or represents the effect of an agent on its neighbors and uses that interaction in reward shaping. The paper states that the method becomes equivalent to a previously described neighborhood reward when the neighborhood radius approaches infinity. Its implementation is positioned as a training-time reward modification rather than a new execution-time communication protocol. The exact equations, named assumptions, and procedure for calculating neighborhood influence are missing from the supplied excerpt, so those technical details cannot be established here. The base reward values are explicitly reported, but they do not by themselves specify the full CoRS reward.
The training system uses 16 independent environments in parallel to generate experiences, a global buffer, and a GPU learner. Training uses a batch size of 192, sequence length 20, a dynamic learning rate that is halved at 100,000 and 300,000 steps, and a maximum of 500,000 steps; the excerpt does not preserve all learning-rate values. Fine-tuning keeps the learning rate at a separate stated value, but that value is not visible in the supplied text. Tests use 200 cases per scenario, obstacle density 0.3, 40 by 40 and 80 by 80 maps, and up to 128 agents, with maximum episode horizons of 256 and 386 steps respectively. Hardware is reported as an Intel i5-13600KF and an Nvidia RTX2060 6G.
For a 40 by 40 map, the reported average steps for DHC and CoRS-DHC are 64.15 and 50.36 with 4 agents, 77.67 and 64.77 with 8, 86.87 and 68.48 with 16, 115.72 and 95.42 with 32, 179.69 and 151.02 with 64, and 213.75 and 193.50 with 128. For an 80 by 80 map, the corresponding values are 114.69 and 92.14 with 4 agents, 133.39 and 109.15 with 8, 147.55 and 121.25 with 16, 158.58 and 137.06 with 32, and 183.44 and 153.06 with 64. The paper reports more than a 20 percent success-rate improvement over DHC in the 64- and 128-agent high-density cases. It also reports that CoRS-DHC surpasses DCC in success rate in most scenarios and has comparable or shorter makespan except for the 64-agent case, but the excerpt does not include the complete success-rate values or uncertainty estimates.
The paper concludes that CoRS can promote cooperation in MARL-based solutions to standard MAPF. The CoRS-DHC combination is reported to improve efficiency and success rate over DHC and to perform comparably to, or better than, DCC in many tested conditions. The authors suggest that the reward-shaping idea may be extended to other multi-agent reinforcement-learning environments. The supplied evidence supports these conclusions for the reported experiments, not for all possible MAPF settings.
The paper itself identifies a potential drawback: making an agent’s reward depend on neighboring actions may reduce training stability and the efficiency of the converged strategy. The supplied excerpt does not establish the exact assumptions behind the shaping method, its computational overhead, sensitivity to communication range or field of view, variance across random seeds, statistical significance, or generalization to unseen map distributions. It also does not provide the full mathematical specification needed to reproduce every part of CoRS. Consequently, the reported superiority should not be generalized automatically to every environment, agent density, or MAPF objective.
The work is important as a relatively lightweight way to inject cooperation into distributed MARL for MAPF. Rather than relying on a complicated communication mechanism at execution time, it changes the training reward while retaining the DHC network structure. The reported gains in dense, many-agent scenarios make the approach relevant to large robot fleets and other settings where centralized planning is costly. Its broader research value lies in treating neighborhood influence as a bridge between individual rewards and system-level traffic performance.
This paper is suitable for researchers studying MAPF, distributed multi-agent reinforcement learning, reward design, and cooperative behavior. It is especially relevant to readers comparing scalable local policies with communication-heavy coordination methods, or examining performance as agent density increases. Readers who need the exact CoRS equations, assumptions, or a fully reproducible training specification should consult the complete paper because those details are not contained in the supplied excerpt.