darl-libsignal / libsignal Goto Github PK

Describe the bug
frap agent file has a bug

To Reproduce
Steps to reproduce the behavior:

1. Go to 'agent' folder
2. Click on 'frap'
3. Scroll down to Line 440
4. See “demand = states[:, i:i+self.demand_shape]”

Expected behavior
The expression i: i+demandshape in this code is incorrect when the demandshape is not 1.
it should be written as demand = states[:, i * self.demand_shape:(i + 1) * self.demand_shape]

I like this project very much, thanks to the work of all researchers, I have a little question about the code implemented by MPLight.
In the official paper of MPLight, its agent observation is the pressure of the traffic flow at the intersection, which is delta_lane_num_vehicle, and the reward is the pressure of the intersection. However, when our LibSignal reproduces MPLight, the observation setting is lane_count, The documentation states that it should be the number of vehicles driving on each lane. The reward setting is the lane_waiting_count, which should be the number of vehicles waiting on the lane. These seem to be different from the design in the official paper of MPLight. May you tell me what is going on? And the second question is that CoLight code cannot run whether you implement it.

I conducted experiments using colight methods on a hangzhou_4x4 network，and the final results were as follows:
2023-12-25 02:09:05 (INFO): Final Travel Time is 343.5820, mean rewards: -21.5031, queue: 1.2889, delay: 0.0717, throughput: 2732.
The environment of my experimental system is: win10, sumo Version 1.16.0.
There are many differences between my experimental results and those in the paper. Do cityflow and sumo perform experiments with similar results with the same data and the same hyperparameter settings?

Hi,

I ran the MPLight agent on the Manhattan_28x7 network and got the following error:
File "/common/home/qy129/DaRL/LibSignal/agent/mplight.py", line 52, in init
self.phase_pairs = self.dic_traffic_env_conf.param['signal_config'][map_name]['phase_pairs']
KeyError: 'manhattan_28x7'

Could you help me to solve this problem? Thanks!

Best regards,
Qinchen Yang

I got this error when running MPLight algo, no problem with FRAP btw. Please take a look, thanks

  File "/home/ubuntu/LibSignal/run.py", line 72, in <module>
    test.run()
  File "/home/ubuntu/LibSignal/run.py", line 66, in run
    self.task.run()
  File "/home/ubuntu/LibSignal/task/task.py", line 30, in run
    self.trainer.train()
  File "/home/ubuntu/LibSignal/trainer/tsc_trainer.py", line 131, in train
    obs, rewards, dones, _ = self.env.step(actions.flatten())
  File "/home/ubuntu/LibSignal/environment.py", line 44, in step
    self.world.step(actions)
  File "/home/ubuntu/LibSignal/world/world_cityflow.py", line 518, in step
    self.intersections[i].step(action, self.interval)
  File "/home/ubuntu/LibSignal/world/world_cityflow.py", line 121, in step
    self._change_phase(self.phases[self.action_before_yellow], interval,'add')
TypeError: list indices must be integers or slices, not NoneType```

For multi-agent systems that share information among agents, there is confusion regarding the handling of sub-agents, num_agent, and generators in the code. In the 4x4 road network, on the one hand, the sub-agents of Colight are 16 (the number of intersections) and num_agent is 1. However, in Colight, isn't each intersection independently controlled by an agent? It appears that the entire road network is controlled by one agent, with each intersection having its own generator and decision model. On the other hand, is Presslight a Multi-Agent approach? Its sub-agents are 1 and num_agent is 16. Each intersection has its own agent and decision model. This confusion is significant, especially if one intends to extend other Multi-Agent models that share information among agents.

Describe the bug
A clear and concise description of what the bug is.

I modified the agent as "ppo", the world as "sumo", the network as "sumo4x4". The rest codes of the "run.py" remained unchanged. When I run "run.py", there is an error as follow:
File "/home/xxx/traffic_signal_control/DaRL/LibSignal/trainer/tsc_trainer.py", line 90, in create_agents
agent = Registry.mapping['model_mapping'][Registry.mapping['command_mapping']['setting'].param['agent']](self.world, 0)
KeyError: 'ppo'

the similar errors occured in such agent "ppo_pfrl, maddpg, magd".
I also test the other agents and they can operate normally.
Can you solve the problem and answer me?
Thanks a lot!

Describe the bug
Thanks for your work. I want to run the simulation in SUMO and see the simulation progress in sumo-gui, take sumo1x1.cfg for example, I changed the gui from "false" to "true", but it doesn't appear the simulation progress in sumo-gui. If I changed the gui from "false" to "True", it will occur errors. How can I solve this problem?

Expected behavior

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

I conducted experiments using dqn and colight methods on a 6x6 network, and the final results were as follows:
Dqn: Final Travel Time is 214.5438, mean rewards: -14.1031, queue: 0.4005, delay: 0.0308, throughput: 4652 Total time token: 18397.876304388046
Colight: Final Travel Time is 206.8853, mean rewards: -4.6692, queue: 0.1022, delay: 0.0111, throughput: 4650 Total time token: 16563.93673968315

It seems that there is not much difference in the results between the two methods. Is it my mistake in implementing the method?

Can anyone rectify where I'm going wrong?
I've attached my current working directory and the error

In Paper "Table 3: Performance comparison of agents w.r.t. average travel time (in seconds) before and after calibration"

Then I run the code and got some output as below:

As you can see, in episode 193~200, real avg travel time( comes from self.metric.real_average_travel_time()) is getting bigger. And it is not equl to travel time( still comes from self.metric.real_average_travel_time()). The last two line has Final Travel Time( still comes from same func).

self.metric.real_average_travel_time() func return self.world.get_average_travel_time(). But in mplight or colight, even AttentionLight, they do not use the get_average_travel_time() as average travel time.

So which number should be the num in Table 3 as final score?

2024-01-22 10:23:45 (INFO): episode:2/200, real avg travel time:859.4379020286987
2024-01-22 10:23:48 (INFO): Test step:2/200, travel time :1314.948540326571, rewards:-286.4108072916667, queue:191.225, delay:0.7417051527235243, throughput:510
2024-01-22 10:23:51 (INFO): step:3600/3600, q_loss:1752.2039794921875, rewards:-219.97369791666668, queue:146.84166666666667, delay:0.7382037692599827, throughput:1338
2024-01-22 10:23:51 (INFO): episode:3/200, real avg travel time:820.210786739238
2024-01-22 10:23:53 (INFO): Test step:3/200, travel time :1051.2508659079663, rewards:-278.2649956597222, queue:185.8, delay:0.7478469000922309, throughput:848
2024-01-22 10:23:56 (INFO): step:3600/3600, q_loss:345.8424377441406, rewards:-188.90455729166666, queue:125.9888888888889, delay:0.7571139865451388, throughput:1706
2024-01-22 10:23:56 (INFO): episode:4/200, real avg travel time:363.3983176645225
2024-01-22 10:23:59 (INFO): Test step:4/200, travel time :1258.8179119247898, rewards:-286.99205729166664, queue:191.62777777777777, delay:0.7472451951768663, throughput:577
2024-01-22 10:24:02 (INFO): step:3600/3600, q_loss:489.8249206542969, rewards:-192.62042100694444, queue:128.56944444444446, delay:0.7700509813096789, throughput:1699
2024-01-22 10:24:02 (INFO): episode:5/200, real avg travel time:351.0336467095497
2024-01-22 10:24:05 (INFO): Test step:5/200, travel time :1205.414646214745, rewards:-282.0478949652778, queue:188.30833333333334, delay:0.7466458638509115, throughput:609
2024-01-22 10:24:08 (INFO): step:3600/3600, q_loss:541.1013793945312, rewards:-172.99289279513889, queue:115.34722222222223, delay:0.731319342719184, throughput:1673
2024-01-22 10:24:08 (INFO): episode:6/200, real avg travel time:322.04700643245917
2024-01-22 10:24:10 (INFO): Test step:6/200, travel time :1179.8758040573973, rewards:-279.40833333333336, queue:186.54166666666666, delay:0.7450555589463975, throughput:625
2024-01-22 10:24:13 (INFO): step:3600/3600, q_loss:562.461181640625, rewards:-185.64453125, queue:123.79166666666667, delay:0.7646314832899306, throughput:1700
2024-01-22 10:24:13 (INFO): episode:7/200, real avg travel time:347.7654626422563
2024-01-22 10:24:16 (INFO): Test step:7/200, travel time :1047.5264720435428, rewards:-276.15959201388887, queue:184.3972222222222, delay:0.7471856858995226, throughput:863
2024-01-22 10:24:19 (INFO): step:3600/3600, q_loss:562.9974975585938, rewards:-177.25579427083332, queue:118.18055555555556, delay:0.7533392164442274, throughput:1698

...

2024-01-22 10:38:56 (INFO): episode:193/200, real avg travel time:183.683819891143
2024-01-22 10:38:58 (INFO): Test step:193/200, travel time :212.267194458189, rewards:-98.22127821180555, queue:65.46388888888889, delay:0.716658189561632, throughput:1848
2024-01-22 10:39:01 (INFO): step:3600/3600, q_loss:584.511474609375, rewards:-95.20210503472222, queue:63.425, delay:0.7212346394856771, throughput:1836
2024-01-22 10:39:01 (INFO): episode:194/200, real avg travel time:205.3117268678872
2024-01-22 10:39:03 (INFO): Test step:194/200, travel time :243.0212765957447, rewards:-90.86710069444445, queue:60.47222222222222, delay:0.6561620924207899, throughput:1818
2024-01-22 10:39:06 (INFO): step:3600/3600, q_loss:591.1795654296875, rewards:-93.15581597222223, queue:62.09722222222222, delay:0.7047738817003039, throughput:1884
2024-01-22 10:39:06 (INFO): episode:195/200, real avg travel time:196.57001484413658
2024-01-22 10:39:08 (INFO): Test step:195/200, travel time :226.32558139534885, rewards:-95.26252170138889, queue:63.44444444444444, delay:0.6850776248508029, throughput:1824
2024-01-22 10:39:11 (INFO): step:3600/3600, q_loss:626.0236206054688, rewards:-87.17833116319444, queue:58.05555555555556, delay:0.6814577738444011, throughput:1857
2024-01-22 10:39:11 (INFO): episode:196/200, real avg travel time:210.8753092528451
2024-01-22 10:39:13 (INFO): Test step:196/200, travel time :192.32261256803562, rewards:-84.28835177951389, queue:56.172222222222224, delay:0.6935267978244357, throughput:1872
2024-01-22 10:39:15 (INFO): step:3600/3600, q_loss:607.767578125, rewards:-89.33538953993056, queue:59.49166666666667, delay:0.6868301815456814, throughput:1868
2024-01-22 10:39:15 (INFO): episode:197/200, real avg travel time:193.19495299356754
2024-01-22 10:39:17 (INFO): Test step:197/200, travel time :158.76991588322613, rewards:-76.43204752604167, queue:50.93888888888889, delay:0.6811378055148655, throughput:1917
2024-01-22 10:39:20 (INFO): step:3600/3600, q_loss:641.4354248046875, rewards:-90.39168294270833, queue:60.21111111111111, delay:0.6758840772840712, throughput:1834
2024-01-22 10:39:20 (INFO): episode:198/200, real avg travel time:250.10192973775358
2024-01-22 10:39:21 (INFO): Test step:198/200, travel time :188.54576942107866, rewards:-93.7296115451389, queue:62.42777777777778, delay:0.65865478515625, throughput:1897
2024-01-22 10:39:24 (INFO): step:3600/3600, q_loss:645.171875, rewards:-91.09501953125, queue:60.669444444444444, delay:0.6866438971625434, throughput:1840
2024-01-22 10:39:24 (INFO): episode:199/200, real avg travel time:222.88174171202374
2024-01-22 10:39:26 (INFO): Test step:199/200, travel time :243.66996536368134, rewards:-113.17209201388889, queue:75.41944444444445, delay:0.6514779832628038, throughput:1762
2024-01-22 10:39:29 (INFO): Final Travel Time is 243.6700, mean rewards: -113.1721, queue: 75.4194, delay: 0.6515, throughput: 1762
2024-01-22 10:39:29 (INFO): Total time taken: 956.3847246170044

When I ran the CoLight code about the Hangzhou 4x4 dataset, I saw a problem the CPU (GPU not used) memory kept increasing with the episodes training. In the end, the memory (64G) was directly exploded and the program was killed. Can you tell me how to solve this problem?

Thank you very much for your team's contribution to LibSignal, providing a benchmark for comparison. However, while using the maxpressure and sotl algorithms on the Hangzhou 4x4 dataset, I have observed significant discrepancies compared to the results reported by your team in the paper. I would like to inquire whether you are aware of the reasons behind these differences and if you could provide assistance in resolving this issue. Your guidance would be highly appreciated.

Furthermore, I have tested other road networks, and the issue persists in the maxpressure algorithm. I hope to receive your assistance in resolving this matter.

Hello, I would like to ask. In multi-agent reinforcement learning, there are centralized agents, decentralized agents, and centralized training decentralized execution modes. The number of agents in these different modes is different, and even the number of different networks in the agent is also different. Is libsignal compatible? How is it handled?

What is the version of SUMO

And also how to run colight? When I run it, "TypeError: init() missing 5 required positional arguments: 'reward_generator', 'world', 'traffic_env_conf', 'graph_setting', and 'args'"

Please

Thank you very much for your team's contribution in this field, the LibSignal platform facilitates the comparison of different algorithms. But I have some problems with the grid4x4 dataset. Specifically, I didn't find the grid4x4 dataset in GitHub for SUMO environment and I got an error in the code when using the converter.py program for cityflow2sumo.
I wonder if you and your team can solve my problem, if you can upload the grid4x4 dataset in SUMO environment or give me a hint on how to solve the error in converter.py so that I can compare my algorithm on this dataset. Much appreciated.

Finally, thanks again for your contributions!

Hello, in your doc you mention there is conversion between SUMO and CityFlow simulation. I wonder where is the script of convert CityFlow data to SUMO data.
Much thanks to your reply : )

what dose sumo traffic path mean? Is it sumo.rou.xml or trip.xml? Can converter do multiple file conversions? If I have multiple carrier roux files, am I supposed to combine them?

Thank you and your team for the work you have done. I have recently set up Libsignal and it is running smoothly.

However, I would like to know how to integrate my own algorithm into this framework in order to compare it with the existing algorithms. It would be great if there is a documentation available for this purpose.

I understand that creating an instructional document might be challenging, but hope you can consider it, thank you again.

Hello,

I am attempting to run the latest version of the LibSignal project, and while I can successfully execute the code with CityFlow, I am encountering issues when attempting to run it with SUMO. Specifically, when I execute the following command:

python3 run.py -w sumo -n sumo1x1

I receive the following error:
File "/home/burak/LibSignal/trainer/tsc_trainer.py", line 62, in create_world self.world = Registry.mapping['world_mapping'][Registry.mapping['command_mapping']['setting'].param['world']]( KeyError: 'sumo'

Do you have any suggestions for resolving this error? Additionally, I am running this on Ubuntu within a WSL (Windows Subsystem for Linux) environment.

Thank you.

Hello, I would like to ask how the performance in your paper is calculated, is it the average of multiple runs? If yes, how many runs in total?

I want to change the network config, so I used the converter.py to trans my sumo's net.xml and flow.xml to city flow type,then the Segmentation fault popped out , here is the report and the config I made