Giter VIP home page Giter VIP logo

Comments (3)

candyzone avatar candyzone commented on July 26, 2024

@formath thanks, please show your all error message, including call stack.
It seems that something wrong parsing tensor in CKPT.

from deeprec.

formath avatar formath commented on July 26, 2024

@candyzone
userid_embedding is a partitioned ev embedding. Other variables have no problem. I guess the partitioned variables need a special logic.

Traceback (most recent call last):
  File "prerank_debias.py", line 1854, in <module>
    tf.app.run()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "prerank_debias.py", line 1839, in main
    model.run()
  File "prerank_debias.py", line 1690, in run
    tf.estimator.train_and_evaluate(classifier, train_spec, eval_spec)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/training.py", line 479, in train_and_evaluate
    return executor.run()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/training.py", line 646, in run
    getattr(self, task_to_run)()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/training.py", line 652, in run_chief
    return self._start_distributed_training()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/training.py", line 829, in _start_distributed_training
    saving_listeners=saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 373, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1229, in _train_model_default
    saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1404, in _train_with_estimator_spec
    warm_starting_util.warm_start(*self._warm_start_settings)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/warm_starting_util.py", line 531, in warm_start
    checkpoint_utils.init_from_checkpoint(ckpt_to_initialize_from, vocabless_vars)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/checkpoint_utils.py", line 293, in init_from_checkpoint
    init_from_checkpoint_fn)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/distribute/distribute_lib.py", line 1940, in merge_call
    return self._merge_call(merge_fn, args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/distribute/distribute_lib.py", line 1947, in _merge_call
    return merge_fn(self._strategy, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/checkpoint_utils.py", line 288, in <lambda>
    ckpt_dir_or_file, assignment_map)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/checkpoint_utils.py", line 321, in _init_from_checkpoint
    tensor_name_in_ckpt, ckpt_dir_or_file, variable_map
ValueError: Tensor input_layer/userid_embedding/embedding_weights is not found in /xxx/checkpoint/model.ckpt-755185 checkpoint {'user_dnn/dnn_2/bias/Adam_1': [64], 'user_dnn/dnn_2/bias/Adam': [64], 'user_dnn/dnn_2/bias': [64], 'user_dnn/dnn_1/kernel/Adam': [256, 128], 'user_dnn/dnn_1/kernel': [256, 128], 'user_dnn/dnn_1/bn/moving_variance': [128], 'user_dnn/dnn_1/bn/moving_mean': [128], 'user_dnn/dnn_1/bn/gamma/Adam_1': [128], 'user_dnn/dnn_1/bn/beta': [128], 'user_dnn/dnn_1/bias/Adam_1': [128], ........

from deeprec.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.