bnn-upc / routenet-erlang Goto Github PK

View Code? Open in Web Editor NEW

20.0 20.0 7.0 34.67 MB

License: Apache License 2.0

Python 100.00%

routenet-erlang's People

Stargazers

Watchers

Forkers

ruixiaozhang bluelancer miquelferriol mengxue12 superjdz zzj1hello chuyu-jpg

routenet-erlang's Issues

Loading checkpoints fails

Found a pretrained model, restoring...
BEST CHECKOINT FOUND: 69-9.91.data-00000-of-00001
2022-05-02 11:41:06.107104: W tensorflow/core/util/tensor_slice_reader.cc:96] Could not open ./ckpt_dir/69-9.91.data-00000-of-00001: DATA_LOSS: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
Traceback (most recent call last):
  File "main.py", line 77, in <module>
    model.load_weights('./ckpt_dir/{}'.format(best))
  File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 2400, in load_weights
    raise ValueError(
ValueError: Unable to load weights saved in HDF5 format into a subclassed Model which has not created its variables yet. Call the Model first, then load the weights.

Request for Addition of Missing Poisson Distribution Code in Data Generation

Hi. I noticed that the training scripts in TrafficModels are intended to support five types of distributions as mentioned in the paper.. However, it appears that the segment for training data generated by Poisson distribution is missing.

Could you please provide the missing Poisson distribution training code? The inclusion of this part would be immensely helpful for me and others in the community attempting to replicate your results fully.

Thank you for considering this request. I look forward to the complete set of scripts.

`queue_shape` uses `path_state_dim` instead of `queue_state_dim`

The variable queue_shape is using the parameter path_state_dim instead of queue_state_dim.

RouteNet-Erlang/Scheduling/model.py

Lines 123 to 129 in 269df1e

 # Compute the shape for the all-zero tensor for path_state 

 queue_shape = tf.stack([ 

 n_queues, 

 int(self.config['HYPERPARAMETERS']['path_state_dim']) - 

 int(self.config['DATASET']['max_num_queues']) - 

 1 

 ], axis=0)

It makes no difference with the current configuration since path_state_dim is equal to queue_state_dim.

I cann't access the dataset

Checkpoints in `/Scalability/ckpt_dir/` are ignored

There are some checkpoints in /Scalability/ckpt_dir/, 69-9.91.data-00000-of-00001 and 69-9.91.index. These are ignored when calling main.py to train the model.

Starting training from scratch...

If this is supposed to showcase a training from a checkpoint, I think a checkpoint file is required and it should contain something like the following

model_checkpoint_path: "69-9.91"
all_model_checkpoint_paths: "69-9.91"

Otherwise they might be just some leftovers, in which case they can be safely deleted.

ModuleNotFoundError in Losses experiment

RouteNet-Erlang/Scheduling/Losses/main.py

Line 7 in 3102516

from Scheduling.model import GNN_Model

Traceback (most recent call last):
  File "main.py", line 7, in <module>
    from Scheduling.model import GNN_Model
ModuleNotFoundError: No module named 'Scheduling'

problem about processing of datasets in traffic Models

when i run main.py to train a model in this folder

，the program stopped and showed

，the problem also occurs when i run prediction.py, so i suppose that this problem may related to processing of datasets .what's more, i can only successfully run scalability module well, so could u please give me some instruction on how to fix this problem???plz!

KeyError: 'Failed to format this callback filepath: "ckpt_dir/{epoch:02d}-{val_loss:.2f}". Reason: \'val_loss\''

Training is failing at the second epoch in scalability/main.py

Epoch 1

Epoch 1/200
/anaconda/envs/azureml_py38/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:447: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/GatherV2_9_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/GatherV2_9_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/GatherV2_9_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/anaconda/envs/azureml_py38/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:447: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/GatherV2_8_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/GatherV2_8_grad/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradients/GatherV2_8_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/anaconda/envs/azureml_py38/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:447: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/GatherV2_7_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/GatherV2_7_grad/Reshape:0", shape=(None, 32), dtype=float32), dense_shape=Tensor("gradients/GatherV2_7_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/anaconda/envs/azureml_py38/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:447: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/GatherV2_6_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/GatherV2_6_grad/Reshape:0", shape=(None, 32), dtype=float32), dense_shape=Tensor("gradients/GatherV2_6_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/anaconda/envs/azureml_py38/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:447: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/GatherV2_5_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/GatherV2_5_grad/Reshape:0", shape=(None, 32), dtype=float32), dense_shape=Tensor("gradients/GatherV2_5_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/anaconda/envs/azureml_py38/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:447: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/GatherV2_4_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/GatherV2_4_grad/Reshape:0", shape=(None, 32), dtype=float32), dense_shape=Tensor("gradients/GatherV2_4_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/anaconda/envs/azureml_py38/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:447: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/GatherV2_3_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/GatherV2_3_grad/Reshape:0", shape=(None, 32), dtype=float32), dense_shape=Tensor("gradients/GatherV2_3_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/anaconda/envs/azureml_py38/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:447: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/GatherV2_2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/GatherV2_2_grad/Reshape:0", shape=(None, 32), dtype=float32), dense_shape=Tensor("gradients/GatherV2_2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/anaconda/envs/azureml_py38/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:447: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/GatherV2_1_grad/Reshape:0", shape=(None, 32), dtype=float32), dense_shape=Tensor("gradients/GatherV2_1_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/anaconda/envs/azureml_py38/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:447: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/GatherV2_grad/Reshape:0", shape=(None, 32), dtype=float32), dense_shape=Tensor("gradients/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
2022-04-16 20:41:49.784273: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
2022-04-16 20:42:05.646761: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 26 of 1000
2022-04-16 20:42:15.469365: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 55 of 1000
2022-04-16 20:42:25.503645: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 94 of 1000
2022-04-16 20:42:35.598784: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 134 of 1000
2022-04-16 20:42:45.273349: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 174 of 1000
2022-04-16 20:42:55.381876: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 223 of 1000
2022-04-16 20:43:05.389659: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 280 of 1000
^[>2022-04-16 20:43:15.323200: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 314 of 1000
2022-04-16 20:43:25.298741: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 350 of 1000
2022-04-16 20:43:35.351326: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 426 of 1000
2022-04-16 20:43:45.920267: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 466 of 1000
2022-04-16 20:43:55.303526: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 513 of 1000
2022-04-16 20:44:05.397849: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 546 of 1000
2022-04-16 20:44:15.733292: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 597 of 1000
2022-04-16 20:44:25.349787: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 652 of 1000
2022-04-16 20:44:35.425504: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 723 of 1000
2022-04-16 20:44:45.438521: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 769 of 1000
2022-04-16 20:44:55.287286: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 812 of 1000
2022-04-16 20:45:05.381177: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 873 of 1000
2022-04-16 20:45:15.400310: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 935 of 1000
2022-04-16 20:45:25.343842: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:175] Filling up shuffle buffer (this may take a while): 974 of 1000
2022-04-16 20:45:28.238022: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:228] Shuffle buffer filled.
4000/4000 [==============================] - ETA: 0s - loss: 24.3972WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches (in this case, 20 batches). You may need to use the repeat() function when building your dataset.
4000/4000 [==============================] - 1059s 208ms/step - loss: 24.3972 - val_loss: 24.5493

Epoch 00001: saving model to ckpt_dir/01-24.55

Epoch 2

Epoch 2/200
4000/4000 [==============================] - 632s 158ms/step - loss: 10.6697
Traceback (most recent call last):
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/keras/callbacks.py", line 1458, in _get_file_path
    file_path = self.filepath.format(epoch=epoch + 1, **logs)
KeyError: 'val_loss'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 92, in <module>
    model.fit(ds_train,
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/keras/engine/training.py", line 1230, in fit
    callbacks.on_epoch_end(epoch, epoch_logs)
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/keras/callbacks.py", line 413, in on_epoch_end
    callback.on_epoch_end(epoch, logs)
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/keras/callbacks.py", line 1368, in on_epoch_end
    self._save_model(epoch=epoch, batch=None, logs=logs)
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/keras/callbacks.py", line 1403, in _save_model
    filepath = self._get_file_path(epoch, batch, logs)
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/keras/callbacks.py", line 1463, in _get_file_path
    raise KeyError('Failed to format this callback filepath: "{}". '
KeyError: 'Failed to format this callback filepath: "ckpt_dir/{epoch:02d}-{val_loss:.2f}". Reason: \'val_loss\''

Packages installed

Name: tensorflow
Version: 2.6.3
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email: [email protected]
License: Apache 2.0
Location: /anaconda/envs/azureml_py38/lib/python3.8/site-packages
Requires: typing-extensions, tensorboard, wheel, google-pasta, h5py, absl-py, flatbuffers, keras-preprocessing, opt-einsum, gast, protobuf, astunparse, numpy, grpcio, termcolor, clang, keras, wrapt, six, tensorflow-estimator
Required-by: autokeras
---
Name: networkx
Version: 2.5
Summary: Python package for creating and manipulating graphs and networks
Home-page: http://networkx.github.io/
Author: Aric Hagberg
Author-email: [email protected]
License: UNKNOWN
Location: /anaconda/envs/azureml_py38/lib/python3.8/site-packages
Requires: decorator
Required-by: visions, scikit-image, responsibleai, dowhy
---
Name: configparser
Version: 5.2.0
Summary: Updated configparser from Python 3.8 for Python 2.6+.
Home-page: https://github.com/jaraco/configparser/
Author: Łukasz Langa
Author-email: [email protected]
License: UNKNOWN
Location: /anaconda/envs/azureml_py38/lib/python3.8/site-packages
Requires: 
Required-by: azureml-defaults
---
Name: pandas
Version: 1.3.5
Summary: Powerful data structures for data analysis, time series, and statistics
Home-page: https://pandas.pydata.org
Author: The Pandas Development Team
Author-email: [email protected]
License: BSD-3-Clause
Location: /anaconda/envs/azureml_py38/lib/python3.8/site-packages
Requires: python-dateutil, pytz, numpy
Required-by: visions, torch-tb-profiler, statsmodels, sklearn-pandas, shap, seaborn, scrapbook, responsibleai, raiwidgets, pyLDAvis, pycaret, pmdarima, phik, pandas-profiling, pandas-ml, nimbusml, mlxtend, mlflow, ml-wrappers, interpret-community, fbprophet, fastparquet, fastai, fairlearn, erroranalysis, econml, dowhy, dice-ml, datasets, dask-sql, cufflinks, azureml-training-tabular, azureml-train-automl-runtime, azureml-opendatasets, azureml-datadrift, azureml-automl-runtime, autokeras, arch

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

	# Compute the shape for the all-zero tensor for path_state
	queue_shape = tf.stack([
	n_queues,
	int(self.config['HYPERPARAMETERS']['path_state_dim']) -
	int(self.config['DATASET']['max_num_queues']) -
	1
	], axis=0)