axon-research / c3d-keras Goto Github PK

Can you please upload its trained model especial its weights on repo or send me on my personal email? I am on Windows and installing all these things are the heck process for me.

I was wondering if there is any work done towards being able to replicate the training. I am currently trying to adapt the network to 8 frames instead of 16, for which (as far as I know) the network needs to be trained from scratch.

I am using C3D for a different task, so I was wondering if the mean_cube was calculated from UCF101 dataset.

Hello Sir,

FileNotFoundError: [Errno 2] No such file or directory: 'models/sports1M_weights_tf.json'

I can't find those two files !!

Hi @chuckcho
I have successfully converted C3D v1.0 model to keras(with theano backend) follow https://gist.github.com/albertomontesg/d8b21a179c1e6cca0480ebdf292c34d2

For C3D v1.1,
First, I extracted caffe weights to h5 file following the codes which were used to convert ResNet50 to keras
https://github.com/MoyanZitto/keras-scripts/blob/master/utils/caffe_converter.py (tensorflow backend, I am also using tf backend now).
Second, load weight using

def load_weights(model, weights_path):
    """
    This function load the pretrained weights to the model
    """
    f = h5py.File(weights_path, 'r')
    for layer in model.layers:
        if layer.name[:3] == 'res':
            layer.set_weights([f[layer.name]['weights'][:].transpose((2,3,4,1,0)), f[layer.name]['bias'][:]])
        elif layer.name[:2] == 'bn':
            scale_name = 'scale'+layer.name[2:]
            weights = []
            weights.append(f[scale_name]['weights'][:])
            weights.append(f[scale_name]['bias'][:])
            weights.append(f[layer.name]['weights'][:])
            weights.append(f[layer.name]['bias'][:])
            layer.set_weights(weights)
    model.get_layer('conv1').set_weights([f['conv1']['weights'][:].transpose((2,3,4,1,0)), f['conv1']['bias'][:]])
    model.get_layer('fc8-new').set_weights([f['fc8-new']['weights'][:].T, f['fc8-new']['bias'][:]])
    return model
model = get_resC3D_k2()#keras model architecture
model = load_weights(model, 'c3d_resnet50-1.h5')

But keras model using the converted weights always output one class whatever the input is.
Can you give me some suggestions?

Hi,

Many thanks for this repo.
I ran the code, but got stuck due to ffmpeg inconsistencies which renders cv2.VideoCapture() unusable.

To get a workaround, I bursted frames from the video using:

ffmpeg -i dM06AMFLsrc.mp4 -vf scale=171:128 -r 25 -f image2 'vid_frames/%04d.jpg'

Then read it in the code using scipy.ndimage module:

def load_video_clip():
    print("[Info] Loading a sample video...")
    path_to_vid = "/home/vid_frames/"
    vid = []
    for frame_num in range(2000, 2016):
        frame = scipy.ndimage.imread(path_to_vid + str(frame_num) + ".jpg")
        vid.append(frame)
    return np.array(vid, dtype=np.float32)

The point is that the bursted video frames looks same enough as in https://gist.github.com/albertomontesg/d8b21a179c1e6cca0480ebdf292c34d2.
But, the output I get is:

Position of maximum probability: 359
Maximum probability: 0.13669
Corresponding label: volleyball

Top 5 probabilities and labels:
volleyball: 0.13669
gaelic football: 0.08436
inline speed skating: 0.07865
roller derby: 0.06737
hurdles: 0.06567

What in your opinion might be causing this ?
Thanks again.

• Raghav

Could you publish the pretrained model on UCF-101 in keras?Thank you!

Please set "image_dim_ordering" in ${KERASCONF} to be "tf"

When I try to run bash do_everything.sh I get above error. Can anyone explain how to address this error

Thanks

Regards

It seems that the C3D team have changed the folder structure of their Git repo. When getting the caffe.proto file in do_everything.sh, line 13 needs to be:

wget -N https://raw.githubusercontent.com/facebook/C3D/master/C3D-v1.0/src/caffe/proto/caffe.proto

I was following the steps mentioned but getting this error.

When running the script, I end-up getting the following error:

Traceback (most recent call last):
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/common_shapes.py", line 671, in _call_cpp_shape_fn_impl
    input_tensors_as_shapes, status)
  File "/home/dani/Software/anaconda3/lib/python3.6/contextlib.py", line 89, in __exit__
    next(self.gen)
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Negative dimension size caused by subtracting 2 from 1 for 'pool3/MaxPool3D' (op: 'MaxPool3D') with input shapes:
[?,1,4,28,256].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "convert_caffe_model.py", line 121, in <module>
    main()
  File "convert_caffe_model.py", line 51, in main
    model = c3d_model.get_model(summary=True, backend=dim_ordering)
  File "/home/dani/Git/c3d-keras/c3d_model.py", line 36, in get_model
    border_mode='valid', name='pool3'))
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/keras/models.py", line 455, in add
    output_tensor = layer(self.outputs[0])
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/keras/engine/topology.py", line 545, in __call__
    output = self.call(inputs, **kwargs)
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/keras/layers/pooling.py", line 323, in call
    data_format=self.data_format)
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/keras/layers/pooling.py", line 381, in _pooling_function
    padding, data_format, pool_mode='max')
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 3055, in pool3d
    x = tf.nn.max_pool3d(x, pool_size, strides, padding=padding)
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 1625, in max_pool3d
    strides=strides, padding=padding, name=name)
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op
    op_def=op_def)
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2329, in create_op
    set_shapes_for_outputs(ret)
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1717, in set_shapes_for_outputs
    shapes = shape_func(op)
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1667, in call_with_requiring
    return call_cpp_shape_fn(op, require_shape_fn=True)
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/common_shapes.py", line 610, in call_cpp_shape_fn
    debug_python_shape_fn, require_shape_fn)
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/common_shapes.py", line 676, in _call_cpp_shape_fn_impl
    raise ValueError(err.message)
ValueError: Negative dimension size caused by subtracting 2 from 1 for 'pool3/MaxPool3D' (op: 'MaxPool3D') with input shapes: [?,1,4,28,256].
download_test_video.sh: line 4: youtube-dl: command not found
Using TensorFlow backend.
[Info] image_dim_order (from default ~/.keras/keras.json)=<function image_dim_ordering at 0x7f5a8ad332f0>
[Info] Using backend=<function image_dim_ordering at 0x7f5a8ad332f0>
[Info] Reading model architecture...
Traceback (most recent call last):
  File "test_model.py", line 225, in <module>
    main()
  File "test_model.py", line 136, in main
    model = model_from_json(open(model_json_filename, 'r').read())
FileNotFoundError: [Errno 2] No such file or directory: './models/sports1M_weights_tf.json'

I understand that there are certain problems inherent of using python3, but what about the rest? youtube-dl should be added as dependency. Where can I download sports1M_weights_tf.json?

Thanks,
Dani

Hello,I can't find the sports1M_weights_tf.json in you models, could u please tell me how to get this json? Thank you!!!

I'm getting the error: No such file or directory: './models/sports1M_weights_tf.json'. I would think that the get_weights_and_mean script would be getting the weights. Clearly, it's not. But it's getting a file called "conv3d_deepnetA_sport1m_iter_1900000". It's not clear what I'm supposed to do with this, and when I do try to load this file, I get the error " Unable to open file (File signature not found)", probably because it's not a HDF5 file or it has been corrupted.

Is there supposed to be a weights file that is supposed to be downloaded when you run ./do_everything.sh

Sorry for creating a new issue here (totally unrelated) but there was no "issues" tab on chuckcho/c3d-tensorflow2 repository. It would be great if you can enable it.

Thank you very much.

file "test_model.py",line 169, in main
X = vid[start_frame:(start_frame + 16), :, :, :,]
IndexError:too many indices for array

hi, i run the code "python test_model.py" ,then get this. please help me

Hello,
When i run the convert_caffe_model.py, I have a problem.

[libprotobuf ERROR google/protobuf/io/coded_stream.cc:207] A protocol message was rejected because it was too big (more than 67108864 bytes). To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
Traceback (most recent call last):
File "convert_caffe_model.py", line 121, in
main()
File "convert_caffe_model.py", line 67, in main
p.ParseFromString(open(caffe_model_filename, 'rb').read())
google.protobuf.message.DecodeError: Error parsing message

please, give me some hints, thank you very much!

First, thanks a lot for providing this tensorflow implementation.
I successfully run it and got the same probabilities as given in the script.
However I compared on the snippet used in
https://github.com/Lasagne/Recipes/blob/master/examples/Video%20features%20with%20C3D.ipynb

and get quite different probabilities compared to the caffe reference implementation.
This code:

         wiffle ball: 30.72%
      knife throwing: 22.49%
           boomerang: 5.09%
             croquet: 4.96%
           disc golf: 3.22%
            wallball: 3.10%
            rounders: 2.98%
             archery: 1.96%
      skeet shooting: 1.91%
               bocce: 1.87%

Caffe:

     wiffle ball: 29.91%
  knife throwing: 13.11%
         croquet: 11.27%
       disc golf: 5.29%
        kickball: 5.18%
        rounders: 4.48%
           bocce: 3.53%
       dodgeball: 2.27%
       boomerang: 1.71%
        tee ball: 1.39%

did you ever verify the output against the caffe implementation?
Any idea what the reason for this might be?

Hi! Thanks so much for doing this work, its already saved me a great deal of time since most of my stuff is built in Keras :)!

I'm attempting to fine tune this model for a completely different task, predicting steering angles from dash camera videos. I have the data: driving sequence images and steering angles with each frame. 40k total frames. Though, I'm having some issues and wanted to be sure it doesn't have anything to do with any limitation related to the model/weights your scripts output.

Please note this is my first time fine tuning anything! I read a couple of blogs and such to get an idea but please be critical!

I decided to split the work into two main steps.

Here was my plan of action:

1. Get model/weights from the work you did here.
2. Freeze all the layers.
3. Drop the final layer.
4. Add my own dense layer with one unit and a tanh activation function to predict a steering angle.

The code:

    json_file = open(orignal_model_path, 'r')
    loaded_model_json = json_file.read()
    json_file.close()

    model  = model_from_json(loaded_model_json)
    model.load_weights(orginal_weights_path)
    model.summary()
    model.pop()

    for layer in model.layers:
        layer.trainable = False

    model.add(Dense(1, activation='tanh', name='cutom')) 

5. Train (50 epochs) and save weights and save model. For my optimizer I used adam with a lr(0.0001).

The output I got from this step was already problematic. It was only predicting angles between 10 and 12 degrees. You can check out some of my logs here: http://imgur.com/a/zKTwY

Next:
6. Unfreeze all layers and load model/weights from step 5.
7. Train model again (50 epochs). For my optimizer I used SGD with a lr(0.0001).

The output I got from this layer actually just kept on outputting a single number for every single frames. Which means something went terribly wrong.

Any feedback on where you think I'm going wrong would be awesome!

Did you export model and weights of Sport1M for Keras and Tensorflow like this post? May I download it for pre-trained my networks? Thank you so much

Hello! Have you fixed the issue: Activation outputs begin to differ from conv4a layer for Theano and TensorFlow.

Traceback (most recent call last):
  File "test_model.py", line 225, in <module>
    main()
  File "test_model.py", line 171, in main
    X = vid[start_frame:(start_frame + 16), :, :, :]
IndexError: too many indices for array

I have already checked the closed issue with this exact same problem. My opencv installation is correct since I already use it in other projects. Any ideas?

If I use theano backend with your code it computes different features in fc6 layer, which are not accurate. Is there anyway I can use your code with Theano backend or are C3D weights ported in theano? Please help me out.

I just use the convert_caffe_model.py to convert the c3d model finetuned on UCF101, I have changed the the last fc layer of c3d.py, is that all right? After the conversion, I found the accuracy is very low...I am not sure if it's right to do so..Thank you !