I run the code ,but I do not reproduce the accuracy in the paper.first ,the pkl file in the model you provide I always encounter coding problem.second ,when I try to generate the pkl file,but I use classify_sfew alaws meet errors about "DLNP-"..... SO I use classify_sfew_incep generate pkl file,but I do not reproduce the accuracy in the paper.I very need your help

When I run the code, the train loss is nan. Could you give me some advice? Thank you.

I'm trying to finetune the model2 from the Googledrive uploaded models and while trying to use error is coming as:
"Key CovPool2/Bottleneck/BatchNorm/beta not found in checkpoint"

Also, in the covpoolnet2.py there is dlp_cnn() . But there is no dlp_cnn() function in the whole repository.

But on your Notes you wrote you are using model1 although there isn't any model1 inside the models folder.

Hey, I used your code and applied the tensorflow_path, but it's very slow in a batch. How can I fix it? My GPU is 1080Ti. Thanks for your reply.

I attempted to run the "Temporal_pooling". When I solved issues with the TensorFlow versions I had the specific issue mentioned in README: "While computing gradient of eigen-decomposition, NaNs are returned by tensorflow when eigenvalues are identical. This throws error and cannot continue training."
I solved it by the instructions from "tensorflow_patch.txt file". However, using the tf1 and disable tf2 behaviour by:
"import tensorflow.compat.v1 as tf"
"tf.disable_v2_behavior()"
...did not work in this linlg_grad.py library file. I got the error:
"ModuleNotFoundError: No module named 'tensorflow.compat'"

This version workaround worked only for the .py files: "tempPoolFramework", "tempPoolNetwork", and "tempPoolTrain"

The main confusion for me is that the author does not say how the "tf" appeared in the upgraded code so I deduced that it must be somehow imported.
This is the instructions of the code upgrade:
# Added by Zhiwu Huang @ ETH Zurich for replacing nan and inf with 0s
f = tf.where(tf.is_nan(f), tf.zeros_like(f), f)
f = tf.where(tf.is_inf(f), tf.zeros_like(f), f)

I am using python 3.7
Tensorflow 2.2

If anyone knows what to do with this issue I would highly appreciate for your advice.
Thank you

For Pooling Temporal Feature, It is mentioned that features extracted with CNN (model proposed in the paper) from AFEW dataset were used. Could you please specify which model and the output of which layer did you use as the per-frame feature? Thanks!

Can't technically use repo until it has a license due to https://opensource.stackexchange.com/questions/1720/what-can-i-assume-if-a-publicly-published-project-has-no-license

I am running this code, after training several epochs on the two data sets, this error will occur, can you give me some advice? Thank you.
Traceback (most recent call last):
File "src/train.py", line 330, in
main(parse_arguments(sys.argv[1:]))
File "src/train.py", line 192, in main
args.learning_rate_schedule_file)
File "src/train.py", line 229, in train
err, _, _, step, reg_loss = sess.run([loss, train_op, redun, global_step, regularization_losses], feed_dict=feed_dict)
File "/home/miao/.conda/envs/tensorflow_1.2.0/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 789, in run
run_metadata_ptr)
File "/home/miao/.conda/envs/tensorflow_1.2.0/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 997, in _run
feed_dict_string, options, run_metadata)
File "/home/miao/.conda/envs/tensorflow_1.2.0/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1132, in _do_run
target_list, options, run_metadata)
File "/home/miao/.conda/envs/tensorflow_1.2.0/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1152, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: <exception str() failed>

How many epoches need to be trained to stop training?
Do you need a validation set in the dataset during the training network phase?
Could you give me some advice? Thank you.

I didn't apply the patch suggested in tensorflow_patch. And what I use is python 3.5.

The “Loss” value has been floating in a small range, neither increasing nor decreasing.
The “RegLoss” value remained constant at 0.
Could you give me some advice? Thank you.

Hi,
I want to train a binary problem, i set embedding_size=2 and prepare the train datasets is two classes, is it ok?

Hello.

THanks for your code. I've been trying to write a script to infer your model on the webcam. Here is my code

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import cv2
import tensorflow.compat.v1 as tf

tf.disable_v2_behavior()
import numpy as np
import argparse
import framework
import os
import sys
import math
import pickle
from sklearn.svm import SVC
import dlib


def extract_face(image, rect):
    startX = rect.left()
    startY = rect.top()
    endX = rect.right()
    endY = rect.bottom()
    startX = max(0, startX)
    startY = max(0, startY)
    endX = min(endX, image.shape[1])
    endY = min(endY, image.shape[0])
    w = endX - startX
    h = endY - startY
    return image[startX:endX, startY:endY]

face_detector = dlib.get_frontal_face_detector()

video_capture = cv2.VideoCapture(0)

with tf.Graph().as_default():
    with tf.Session() as sess:
        framework.load_model("../../models/model2/20170815-144407/")
        stub = 0
        images_placeholder = tf.get_default_graph().get_tensor_by_name("input:0")
        embeddings = tf.get_default_graph().get_tensor_by_name("embeddings:0")
        embeddings = tf.nn.l2_normalize(embeddings, 1)
        embedding_size = embeddings.get_shape()[1]
        phase_train_placeholder = tf.get_default_graph().get_tensor_by_name("phase_train:0")
        faces = []
        while True:
            _, frame_bgr = video_capture.read()
            frame = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)
            rects = face_detector(frame)
            face = extract_face(frame, rects[0])
            face = cv2.resize(face, (100, 100))
            faces.append(face)
            if (len(faces) < 128):
                continue
            feed_dict = {images_placeholder: np.asarray(faces), phase_train_placeholder: False}
            arr = sess.run(embeddings, feed_dict=feed_dict)
            with open("/home/daddywesker/EmotionRecognition/CovPoolFER/models/model2/svc_144407.pkl", 'rb') as infile:
                (model, class_names) = pickle.load(infile)

Here are two problems when running this code:

1. arr = sess.run(embeddings, feed_dict=feed_dict) give me NaNs as output. DOn't think that is normal.

Here is a problem with TF version i guess. I have tf 2.5 and i throws me NaNs. I've tried tf 1.15 (which will be deprecated after some time) and it gives me some numbers, not NaNs. But what should we do, when there will be no tf 1.15 anymore?..

2. (model, class_names) = pickle.load(infile) gives me No module named 'sklearn.svm.classes'

2nd problem could be related to scikit-learn version mismatch with yours, but i can't install 0.18.1. My version is 0.24. Min version i can install is 0.20.1 and it throws me another error UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 4: ordinal not in range(128)

I am trying to freeze the model4, using the following code:
'''
import tensorflow as tf

meta_path = 'models/model4/20170812-004516/model-20170812-004516.meta' # Your .meta file

with tf.Session() as sess:

# Restore the graph
saver = tf.train.import_meta_graph(meta_path)

# Load weights
saver.restore(sess,tf.train.latest_checkpoint('models/model4/20170812-004516/'))

# Output nodes
output_node_names =[n.name for n in tf.get_default_graph().as_graph_def().node]

# Freeze the graph
frozen_graph_def = tf.graph_util.convert_variables_to_constants(
    sess,
    sess.graph_def,
    output_node_names)

# Save the frozen graph
with open('model4_graph.pb', 'wb') as f:
  f.write(frozen_graph_def.SerializeToString())

''''

But it is giving me the following error:

FailedPreconditionError: Attempting to use uninitialized value DLP-CNN/Conv2dfo_6/BatchNorm/beta/RMSProp_1_8
[[Node: _retval_DLP-CNN/Conv2dfo_6/BatchNorm/beta/RMSProp_1_8_0_1178 = _RetvalT=DT_FLOAT, index=1178, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Please suggest what changes I need to made in the script.