你好， 可以提供一下NGSIM数据集预处理的代码吗？我想复现你论文中关于NGSIM数据集的结果。希望你百忙之中能提供一下NGSIM数据处理的一些细节。非常感谢。

Hi,
When I run main.py, it occurs that:

C:\Users\lenovo\Anaconda3\lib\site-packages\torch\nn\modules\container.py:435: UserWarning: Setting attributes on ParameterList is not supported.
warnings.warn("Setting attributes on ParameterList is not supported.")
Traceback (most recent call last):
File "F:/LYL/GRIP-master/main.py", line 357, in
run_trainval(model, pra_traindata_path='./train_data.pkl', pra_testdata_path='./test_data.pkl')
File "F:/LYL/GRIP-master/main.py", line 334, in run_trainval
train_model(pra_model, all_loader_train, pra_optimizer=optimizer, pra_epoch_log='Epoch:{:>4}/{:>4}'.format(now_epoch, total_epoch))
File "F:/LYL/GRIP-master/main.py", line 134, in train_model
for iteration, (ori_data, A, _) in enumerate(pra_data_loader):
File "C:\Users\lenovo\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 352, in iter
return self._get_iterator()
File "C:\Users\lenovo\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 294, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "C:\Users\lenovo\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 801, in init
w.start()
File "C:\Users\lenovo\Anaconda3\lib\multiprocessing\process.py", line 112, in start
self._popen = self._Popen(self)
File "C:\Users\lenovo\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\lenovo\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "C:\Users\lenovo\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 65, in init
reduction.dump(process_obj, to_child)
File "C:\Users\lenovo\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
BrokenPipeError: [Errno 32] Broken pipe

Do you know the reason?Thanks.

Hi Xin,
Thank you very much for your paper and code. It is very helpful and inspirational.
When I learn your code of "xin_feeder_baidu.py" file, there are questions about the line 64-79:
Why do you use the angle_mat to multiply the xy and xy_mean?
And why should make sure the angle is greater or equal to pi and it is a random value?
Why construct the angle_mat and why adopt that form of angle_mat in the code?
Thank you again and we look forward to hearing from you.

Hi~Xin
当我使用多GPU（8个）训练时，st_gcn_networks输出channel错误，设定为64，输出为6(=in_channel)。总是在报错：

/home/traffic/anaconda3/envs/GRIP/lib/python3.8/site-packages/torch/nn/modules/container.py:434: UserWarning: Setting attributes on ParameterList is not supported.
  warnings.warn("Setting attributes on ParameterList is not supported.")
#######################################Train
/home/traffic/anaconda3/envs/GRIP/lib/python3.8/site-packages/torch/nn/modules/container.py:489: UserWarning: nn.ParameterList is being used with DataParallel but this is not supported. This list will appear empty for the models replicated on each GPU except the original one.
  warnings.warn("nn.ParameterList is being used with DataParallel but this is not "
Traceback (most recent call last):
  File "/opt/pycharm/pycharm-2020.1.5/plugins/python/helpers/pydev/pydevd.py", line 1438, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/opt/pycharm/pycharm-2020.1.5/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/traffic/GRIP-master/main.py", line 394, in <module>
    run_trainval(model, pra_traindata_path='./train_data.pkl', pra_testdata_path='./test_data.pkl')
  File "/home/traffic/GRIP-master/main.py", line 365, in run_trainval
    train_model(pra_model, all_loader_train,
  File "/home/traffic/GRIP-master/main.py", line 160, in train_model
    predicted = pra_model(pra_x=input_data,
  File "/home/traffic/anaconda3/envs/GRIP/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/traffic/anaconda3/envs/GRIP/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 161, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/traffic/anaconda3/envs/GRIP/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 171, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/traffic/anaconda3/envs/GRIP/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/home/traffic/anaconda3/envs/GRIP/lib/python3.8/site-packages/torch/_utils.py", line 428, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/home/traffic/anaconda3/envs/GRIP/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/home/traffic/anaconda3/envs/GRIP/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/traffic/GRIP-master/model.py", line 86, in forward
    now_predict_car = self.seq2seq_car(in_data=graph_conv_feature, last_location=last_position[:,-1:,:], pred_length=pra_pred_length, teacher_forcing_ratio=pra_teacher_forcing_ratio, teacher_location=pra_teacher_location)
  File "/home/traffic/anaconda3/envs/GRIP/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/traffic/GRIP-master/layers/seq2seq.py", line 68, in forward
    encoded_output, hidden = self.encoder(in_data)
  File "/home/traffic/anaconda3/envs/GRIP/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/traffic/GRIP-master/layers/seq2seq.py", line 22, in forward
    output, hidden = self.lstm(input)
  File "/home/traffic/anaconda3/envs/GRIP/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/traffic/anaconda3/envs/GRIP/lib/python3.8/site-packages/torch/nn/modules/rnn.py", line 737, in forward
    self.check_forward_args(input, hx, batch_sizes)
  File "/home/traffic/anaconda3/envs/GRIP/lib/python3.8/site-packages/torch/nn/modules/rnn.py", line 199, in check_forward_args
    self.check_input(input, batch_sizes)
  File "/home/traffic/anaconda3/envs/GRIP/lib/python3.8/site-packages/torch/nn/modules/rnn.py", line 178, in check_input
    raise RuntimeError(
RuntimeError: input.size(-1) must be equal to input_size. Expected 64, got 6

I watched your code.
I understood that you parsed heading value in your code. but I think you don't use heading value to predict result.
Is grip++using heading?

hi, thank you for sharing the training code for Apollo.

1. Here is the testing results, It seems Root Mean Square Error (RMSE) for trajectory prediction on NGSIM I-80 and US-101 datasets
   #######################################Test
   |2020-06-21 17:24:06.688187|[car] All_All: 0.424 0.722 1.050 1.403 1.770 2.179 7.549
   |2020-06-21 17:24:06.689078|[human] All_All: 0.152 0.282 0.428 0.588 0.746 0.927 3.122
   |2020-06-21 17:24:06.689816|[bike] All_All: 0.419 0.794 1.158 1.568 1.963 2.366 8.269
   |2020-06-21 17:24:06.689913|[WS] All_All: 0.265 0.483 0.713 0.967 1.218 1.494 5.140
   |2020-06-21 17:24:06.691756|[Test_Epoch49] All_All: 0.264 0.473 0.695 0.952 1.210 1.488 5.080
   Is the results correct?

2. Could you please also share the calculation file For ADE and FDE?

@xincoder Hello! I want to ask after you use directly data process code of conv-social-pool, how do you modify the relevant code of data_loader? If it is convenient to you, can you send the data_loader code to my email [email protected]? Thanks very much!

Hi @xincoder,

Thank you for making your code available. I am trying to implement your code with a different dataset, specifically JAAD, for my bachelor's Thesis. Do you think the code is compatible to run with that dataset? Another issue I have is my machine doesn't have NVIDIA gpu. Would it still be possible to run the test without it?

Thank you so much for the help!

如上图，下载的数据集只有车辆信息，并没有车道信息呀

Hello,

Thank you for your great job. I am confused about the display_result() function in main.py. The function seems to have some errors in calculating the RMSE loss which computes the RMSE loss as follows:

rather than as the Eq. (10) in the paper of GRIP++:

Thank you very much! and Looking forward to receive your answer soon

I noticed that in your paper you mention the NGSIM dataset. the input range of the vehicle is 180 feet, All vehicles observed in the same time frame should be outside this range, how did you deal with it.
In Apollo, all vehicles at a time are taken into account, what is the difference between the two?

If the same time frame will be saved as multiple training sequences, will the amount of saved data be too large? Especially in NGSIM because it has a lot of data to process.

您好，有幸阅读您的文章，下面有两个问题希望您能在百忙之中提供帮助。
问题1：请问您文章中固定矩阵和训练矩阵来解决动态的车辆交互问题，在代码中如何体现的呢?
问题2 ：文章中使用平均速度来预测最后的位置信息在代码中是如何体现的呢？

Hi, @xincoder , thank you for the code!
Could you tell me where the implementation of trainable graph is? I read code, but didn't find them.
Maybe edge_importance is the trainable graph part, but I'm not sure.

is:issue is:open Thanks xin! Do you have the code of GRIP but not GRIP++, I cannot fine it! If you have, please sent to my email [email protected], thank you!

I am glad to see your code and paper, It is a interesting research!!!! But I have some questions in dataset. I observated you used the Apollo Scape Sample date to visuslization , the data structure of Sample date is not explained in the Data Structure. I want to know what the nine data represent. For example, such as "1534157110.73 44751 3.924 3.016 1 280.979 838.996 0.000 0.000".

When I run 'graph_conv_block.py' but I found it didnt work.
It said: ModuleNotFoundError: No module named 'layers'
I see the problem comes from 'from layers.graph_operation_layer import ConvTemporalGraphical'
How can I solve the problem?
Looking forward to your receive, thanks.

@xincoder thanks for sharing your algorithm. I have a question, for only one-agent, the prediction looks no good. Do you know the reason?

在Apollo官网没有发现对方向数据的具体描述，想咨询下训练数据中方向是用的以北方向为0度，南方向为180度还是以东方向为0度，西方向为180的方向角测量方式

I'm sorry to bother you again.
As follows,
What do the seven columns represent in these result data?
Is ADE and FDE displayed?
The evaluation results of an Epoch are as follows,
#######################################Test
|2021-07-09 17:58:51.802929|[car] All_All: 0.449 0.773 1.140 1.520 1.927 2.356 8.165
|2021-07-09 17:58:51.802929|[human] All_All: 0.145 0.267 0.403 0.554 0.715 0.891 2.977
|2021-07-09 17:58:51.802929|[bike] All_All: 0.432 0.826 1.207 1.615 2.009 2.440 8.528
|2021-07-09 17:58:51.802929|[WS] All_All: 0.269 0.491 0.727 0.981 1.242 1.525 5.236
|2021-07-09 17:58:51.818551|[Test_Epoch20] All_All: 0.275 0.498 0.739 1.011 1.289 1.584 5.397

model.py文件的主函数中创建model对象，此时传入了graph_args={}空字典，而您在创建这个类的时候，第17行使用了‘A = np.ones((graph_args['max_hop']+1, graph_args['num_node'], graph_args['num_node']))’，我猜测是这里的几个key出现问题了，因为空字典哪来的key呢？作者是否忘记传入参数了呢
期待您的解答

Hi, Thank you for your insightful work.

I have a question about this code when I try to use it on my dataset.

In line 66 of the data_propcess.py, I find that you construct the neighbor_matrix by

neighbor_matrix[:num_visible_object, :num_visible_object] = (dist_xy<neighbor_distance).astype(int)

And I find you construct your feature by:

object_frame_feature[:num_visible_object+num_non_visible_object] = np.transpose(object_feature_list, (1,0,2))

In other words, you constructed your neighbor_matrix by only num_visible_object and your feature by 'num_visible_object'+'num_non_visible_object'. It seems the nodes that should be connected together are now in different dimensions.

I do not know if there is something I missed. I would be very grateful if you could answer this question.

Thank you for your excellent work once again

When I run data_process.py,
it says:

Generating Training Data.
(0,) (0,) (0,)
Generating Testing Data.
(0,) (0,) (0,)

I don't know the reason.

"max_num_object = 120 # maximum number of observed objects is 70"

you annotated max_num_object is 70 In data_process.py line 15
why did you set max_num_object is 120?

Thank you.

Hi Xin,
Thank you very much for your paper and code. It is very helpful and inspirational.
Right now, I am running your code. It works very well on ApolloScape. But I get some trouble when testing the code on NGSIM I-80 and US-101 datasets. The errors are extremely high.
[Test_Epoch0] All_All: 71.520 78.297 83.292 82.014 4.246 71.224 79.316 82.848 83.530 7.253 643.541
data_process.py is utilized to process the NGSIM dataset. The Past 3 seconds are used to predict the next 5 seconds. 2 frames per second.
Can you give me some advice to modify the code?
Thank you for your time.

Where can I find the transformation matrix from simple trajectory to simple image (GPS to camera)coordinate system on ApolloScape? I can't find the corresponding content on the official website of the dataset. I want to ask for help. Thank you!

再次仔细阅读您文章后：有以下问题
问题一：您在文章中解决车辆动态变化问题上使用的训练矩阵，是通过测试的结果表格得知能改善预测的效果，在原理上为什么能提高预测的效果呢，可以告知下您的思路吗？
问题二： 如果训练矩阵训练出来的参数与固定矩阵为何是选择相加而不是相乘呢？
问题三：论文中您未来是打算将得出的预测位置信息与车辆规划结合起来，对您这个想法感兴趣，您能给点启发吗，如何相结合呢？

感谢大佬精彩的分享，我最近使用Argoverse数据集10步的历史轨迹（1s）去预测30步（3s）的未来轨迹，在训练过程中，前几个epoch结果良好，到了五六个epoch时发生了梯度爆炸，损失函数从原来的0.01升到了10,20，请问，您知道这是怎么回事吗

Hello, can you send me a code about GRIP model? If you can, please send it to my email: [email protected] you

Hi,I read your paper, it inspired me a lot and opened up a new direction for me to think, thank you very much. But I had a hard time running experiments with this dataset of NGSIM.Is it possible that you can share your code for the NGSIM dataset or the data loader code with me? It will be very helpful. my email address is [[email protected]].

我有一个简单的问题，关于输入数据的深度的，也就是main.py中Model的in_channels为什么是4呢。是指的[position_x, position_y，heading] + [mask]，这四个特征吗。
如果我扩充每个对象的特征到6个，是应该设置in_channels=6吗？

感谢您的回答，对此我有点迷惑。

epoch20
WSADE: 15.483413498840125
ADEv, ADEp, ADEb: 15.903188312281609 15.339882580782012 15.480199725137442
WSFDE: 15.815679429015853
FDEv, FDEp, FDEb: 16.56610081965146 15.41657854746371 16.185653216166408

hello Im using your code but I got some problems like this could you give me some help? thx

training_log.txt

Can ST-GCN be used if the influence of fixed obstacles is considered when constructing the graph? I want to add the influence of fixed obstacles to predict vehicle trajectory as my research content. Do you have any good suggestions? thank you！

When reading the code, in Graph.py I had a hard time understanding the adjacency input, I made this simplified code and tested that it is equivalent :

        def get_adjacency(self, A):
		# compute hop steps
		transfer_mat = [np.linalg.matrix_power(A, d) for d in range(self.max_hop + 1)]
		transfer_mat = (np.stack(transfer_mat) > 0)
		# transfer_mat is True for all adjacent nodes regardless of hop distance
		# Here we filter out adjacency when it is already in lower hop distance:
		cum_A = np.zeros_like(A[0], dtype='bool')
		for i in range(0, self.max_hop + 1):
			transfer_mat[i, cum_A] = 0
			cum_A = np.logical_or(cum_A, transfer_mat[i])
		return transfer_mat

	def normalize_adjacency(self, A):
		A = A/np.maximum(1, A.sum((0, 1)))
		return A

Unlike the original code, if the input A is not of size (n_nodes, n_nodes), the output will still have the same size as A. Otherwise, it gives the same results.

Hi, I'm enjoying reading your code and paper. I have some questions about fixed graph and something in Graph Convolutional Model.

1. What 'hop_step' means in 'graph.py -> def get_adjacency(self, A)'? So, I can't understand 'compute hop steps' and 'compute adjacency' in 'def get_adjacency(self, A)' ( Creating graph A )

2. I can't match image(equation 5,6 in your paper) shown below and 'def normalize_adjacency(self, A)' in graph.py. Especially, I can't find Lamda you explained in your paper at your code, and also, I can't understand what Lamda means. So, I can't figure out caculating normalized A
   

3. Is equation 7 (image shown below) in your paper coded at 'graph_conv_block'? I can't find correctly.
   

4. In 'xin_feeder_baidu -> getitem', you explained 'self.all_adjacency' shape is (5010, 120, 120). Do 5010 means batch_size or the number of frames? Do 120 means node(agent) number?

I'm sorry because I have many questions about creating fixed graph and the process of Graph Convolutional Model. I can't understand them although I'm reading your paper a few times, and I'am interested in studying creating graph in your paper.

Again, thanks for your code and paper.

I read grip++ takes velocity as input.

I am trying to test your algorithm on real vehicle.

should I conversion traffic object's position to velocity ?

or automatically changed in your code??

If I take input from real time tracking data, Are there any things to consider?

2. If I change max_num_object = 120 # maximum number of observed objects is 70 in data_process.py

when set 8

Generating Training Data.
Traceback (most recent call last):
File "data_process.py", line 193, in
generate_data(train_file_path_list, pra_is_train=True)
File "data_process.py", line 164, in generate_data
now_data, now_adjacency, now_mean_xy = generate_train_data(file_path)
File "data_process.py", line 116, in generate_train_data
object_frame_feature, neighbor_matrix, mean_xy = process_data(now_dict, start_ind, end_ind, observed_last)
File "data_process.py", line 66, in process_data
neighbor_matrix[:num_visible_object, :num_visible_object] = (dist_xy<neighbor_distance).astype(int)
ValueError: could not broadcast input array from shape (13,13) into shape (6,6)

when set 70

Generating Training Data.
Traceback (most recent call last):
File "data_process.py", line 193, in
generate_data(train_file_path_list, pra_is_train=True)
File "data_process.py", line 164, in generate_data
now_data, now_adjacency, now_mean_xy = generate_train_data(file_path)
File "data_process.py", line 116, in generate_train_data
object_frame_feature, neighbor_matrix, mean_xy = process_data(now_dict, start_ind, end_ind, observed_last)
File "data_process.py", line 91, in process_data
object_frame_feature[:num_visible_object+num_non_visible_object] = np.transpose(object_feature_list, (1,0,2))
ValueError: could not broadcast input array from shape (73,12,11) into shape (70,12,11)

how can I solve this problem?

Hi, i would like to know if GRIP model can be used for an intersection scenario. For instance to predict left or right turning vehicles trajectories.

您好，有幸阅读到您的文章（GRIP++）。根据您提供的代码，有3处疑惑，希望您能在百忙之中提供一些帮助，谢谢!
问题1：
论文中Graph Convolutional Model，Graph Operation的处理没有包含卷积操作，直接将上一层的输出作为输入；而代码中Graph Operation前面都是加了一个卷积操作，也就是说每个小block由Graph Operation+BN+CN+BN，变成CN+Graph Operation+BN+CN+BN。不知道是不是我看的不仔细？
问题2 ：
代码中test_model时，是不是没有将预测的RMSE计算步骤的代码实现，结果只有预测的坐标
问题3：
训练中验证的指标，结果大致如[WS] All_All: 0.261 0.475 0.710 0.957 1.200 1.493 5.096和[Test_Epoch161] All_All: 0.257 0.460 0.685 0.935 1.187 1.477 5.001
每个数值代表的是什么指标？是预测的每帧对应的ADE吗？

Thanks for your great work.

I'm facing a problem when I run the main.py.

Traceback (most recent call last):
  File "/home/xxx/anaconda3/envs/DRL/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 176, in flatten_parameters
    self.batch_first, bool(self.bidirectional))  # type: ignore
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
python-BaseException

Can you give me some suggestions?
Thanks a lot!

HI,

I enjoyed reading through your paper. I am working on trajectory prediction of objects in a confined space.
Do you think GRIP can be applied in 3D space instead of 2D space ?
Do you already have some work on this?

Hello, I would like to ask which part of the following data set should be downloaded

I would like to improve the performance by integrating vector map information (with lane information) provided by nuscenes or argoverse dataset into your model, do you think it would be possible to change your model a little? I think I should concatenate the feature, but I am still studying this part.

Thank you for your help, as always

Hey,

thanks for sharing your code!
I am soon going to try it our but while reading your paper, I wondered how you handle varying sizes of vehicles in the traffic scene.
Your paper always uses n vehicles. However, this cannot be true for all traffic situations that you encounter in the datasets.
Same questions goes for the interactions graphs. Don't you need fixed graph dimensions with fixed number of nodes at least to have a fixed adjacency matrix dimension (for the neighbor graphs).

Looking forward to your answer!

Cheers