pinatafarms / feartracker Goto Github PK

Thanks for your GREAT work!

Hi，could you descripte clearer about dataset COCO2017 and ILSVRC?
I'm not sure where to download video and annotations from those two datasets.

When python demo_video.py, I encounter a question. But I have installed this module. How to deal with it?

When python demo_video.py, I encounter a question. But I have installed this module. How to deal with it?

Can you provide the version of macos，xcode，ios in your experiments？
I am not familiar with
iOS development and encountered many errors when building the project😢
Thank you！

Hello, Thanks for your GREAT work.
i'm using your project to track any object on jetson nano with 16 FPS. since 16 FPS is disappointing, i have to increase the FPS, i think if i convert the model to ONNX format with FP16 conversion, model will run faster. so my question is how to convert .ckpt model to .onnx(GPU)?

hello！Where is the code for dynamic template updates? I couldn't find it

This seems to be a common issue with 3090 for many projects:

NVIDIA GeForce RTX 3090 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70 sm_75.
If you want to use the NVIDIA GeForce RTX 3090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
Traceback (most recent call last):
File "demo_video.py", line 62, in
Fire(main)
File "/home/dualeco/.conda/envs/py37fear/lib/python3.7/site-packages/fire/core.py", line 138, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/dualeco/.conda/envs/py37fear/lib/python3.7/site-packages/fire/core.py", line 468, in _Fire
target=component.name)
File "/home/dualeco/.conda/envs/py37fear/lib/python3.7/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "demo_video.py", line 55, in main
tracked_bboxes = track(tracker, video, initial_bbox)
File "demo_video.py", line 24, in track
tracker.initialize(frames[0], initial_bbox)
File "/home/dualeco/Documents/FEARTracker-main/model_training/tracker/fear_tracker.py", line 34, in initialize
self._template_features = self.get_template_features(image, rect)
File "/home/dualeco/Documents/FEARTracker-main/model_training/tracker/fear_tracker.py", line 44, in get_template_features
return self.net.get_features(img)
File "/home/dualeco/Documents/FEARTracker-main/model_training/model/fear_net.py", line 64, in get_features
features = self.feature_extractor(crop)
File "/home/dualeco/Documents/FEARTracker-main/model_training/model/fear_net.py", line 60, in feature_extractor
x = stage(x)
File "/home/dualeco/.conda/envs/py37fear/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/dualeco/.conda/envs/py37fear/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
input = module(input)
File "/home/dualeco/.conda/envs/py37fear/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/dualeco/Documents/FEARTracker-main/src/mobile-cv/mobile_cv/arch/fbnet_v2/basic_blocks.py", line 292, in forward
x = self.conv(x)
File "/home/dualeco/.conda/envs/py37fear/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/dualeco/.conda/envs/py37fear/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 423, in forward
return self._conv_forward(input, self.weight)
File "/home/dualeco/.conda/envs/py37fear/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 420, in _conv_forward
self.padding, self.dilation, self.groups)
RuntimeError: CUDA error: no kernel image is available for execution on the device

Hallo, thanks for you awesome work. I found the FEARDemo here only provide swift version on iOS. Will you further provide a version of python? Or could you please give me some instruction to write an python demo? Thanks a lot!

Will you open source the code for this part? I think that's a very important part of your work!

Hello, I encounter a question when I convert FEAR-XS.ckpt to onnx or pt(torchscript).
I want to use the torch.jit.script（） to convert FEAR-XS-NoEmbs.ckpt to onnx or pt(torchscript) to complete a deploying task on the NPU device.
The Neck and Connector of model is ok, but the encoder can not be converted to torchscript.

I use the scirpt following:

from model_training.utils.hydra import load_hydra_config_from_path
from hydra.utils import instantiate
from model_training.utils.torch import load_from_lighting

config_path= "model_training/config",
config_name= "fear_tracker",
weights_path = 'FEAR-XS-NoEmbs.ckpt'
model = instantiate(config["model"])
model = load_from_lighting(model, weights_path).to('cpu').eval()

template_input = torch.randn(1, 3, 128, 128)
search_input = torch.randn(1, 3, 256, 256)
traced_script_module = torch.jit.script(model )
traced_script_module.save('./FEAR-XS-NoEmbs-torchscript.pt')
print('cpkt2pt done')

RuntimeError:
FBNetBackbone has no attribute 'arch_def' (This attribute exists on python module, but we failed to convert Python type "dict" to a Torchscript type).
File "mobile-vision/mobile_cv/model_zoo/fbnet_v2.py"
     @property
    def arch_def(self):
        return self.backbone.arch_def

What can I do for conver FEAR-XS-NoEmbs.ckpt to onnx or pt(torch script)?

Hi! Thank you for publishing your code.

Your released code has a section dedicated to configuration files corresponding to different tracker modules https://github.com/PinataFarms/FEARTracker/tree/main/model_training/config

It has parameters/choices related to training and inference (optimizer, learning rate scheduler, penalty_k, window influence, lr to name a few). Can you please suggest which dataset was used to tune these hyperparameter values? Was it fine-tuned using the test-set itself?

Also, I am particularly intrigued by a statement in the paper: "For each epoch, we randomly sample 20,000 images from LaSOT, 120,000 from COCO, 400,000 from YoutubeBB, 320,000 from GOT10k and 310,000 images from the ImageNet dataset". Can you suggest what was the reason behind choosing such a sampling split and not going for uniform sampling?

There are some errors in training used the window10 OS which are shown behind：
”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“
Error executing job with overrides: ['backend=1gpu']
Error locating target 'model_training.model.fear_net.FEARNet', see chained exception above.
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”“”
It still wrong after I tried method found on the Internet
$ export HYDRA_FULL_ERROR=1
$ python foo.py
Could you tell me how to deal this problem.
By the way,could you upload the code used to create the CSV file. Thanks

Traceback (most recent call last):
File "D:\Anaconda3\envs\feartracker\lib\site-packages\hydra_internal\utils.py", line 644, in _locate
obj = getattr(obj, part)
AttributeError: module 'model_training.model' has no attribute 'fear_net'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\Anaconda3\envs\feartracker\lib\site-packages\hydra_internal\utils.py", line 650, in locate
obj = import_module(mod)
File "D:\Anaconda3\envs\feartracker\lib\importlib_init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 671, in _load_unlocked
File "", line 843, in exec_module
File "", line 219, in _call_with_frames_removed
File "g:\vot\FEARTracker\model_training\model\fear_net.py", line 6, in
from model_training.model.blocks import Encoder, AdjustLayer, BoxTower
File "g:\vot\FEARTracker\model_training\model\blocks.py", line 5, in
from mobile_cv.model_zoo.models.fbnet_v2 import fbnet
ModuleNotFoundError: No module named 'mobile_cv'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "D:\Anaconda3\envs\feartracker\lib\site-packages\hydra_internal\instantiate_instantiate2.py", line 134, in _resolve_target
target = _locate(target)
File "D:\Anaconda3\envs\feartracker\lib\site-packages\hydra_internal\utils.py", line 653, in _locate
raise ImportError(
ImportError: Error loading 'model_training.model.fear_net.FEARNet':
ModuleNotFoundError("No module named 'mobile_cv'")
Are you sure that 'fear_net' is importable from module 'model_training.model'?

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "g:\vot\FEARTracker\demo_video.py", line 62, in
Fire(main)
File "D:\Anaconda3\envs\feartracker\lib\site-packages\fire\core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "D:\Anaconda3\envs\feartracker\lib\site-packages\fire\core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "D:\Anaconda3\envs\feartracker\lib\site-packages\fire\core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "g:\vot\FEARTracker\demo_video.py", line 52, in main
tracker = get_tracker(config_path=config_path, config_name=config_name, weights_path=weights_path)
File "g:\vot\FEARTracker\demo_video.py", line 16, in get_tracker
model = instantiate(config["model"])
File "D:\Anaconda3\envs\feartracker\lib\site-packages\hydra_internal\instantiate_instantiate2.py", line 226, in instantiate
return instantiate_node(
File "D:\Anaconda3\envs\feartracker\lib\site-packages\hydra_internal\instantiate_instantiate2.py", line 333, in instantiate_node
target = _resolve_target(node.get(_Keys.TARGET), full_key)
File "D:\Anaconda3\envs\feartracker\lib\site-packages\hydra_internal\instantiate_instantiate2.py", line 139, in _resolve_target
raise InstantiationException(msg) from e
hydra.errors.InstantiationException: Error locating target 'model_training.model.fear_net.FEARNet', set env var HYDRA_FULL_ERROR=1 to see chained exception.

In the Demo app, how to judge the target is missing? The confidence output is always very high, regardless of whether the target is missing or not

could you update the raw results,thank you.

Hi, I tried to use the demo in the repository and did not make any changes to the code, but I found that I could not complete the tracking. The prediction frame is off target on the second frame. How should I fix it?

Only FEAR-XS in checkpoints

Hello When I run the demo_video file, the following error is displayed, the mobile_cv file is missing：
from mobile_cv.model_zoo.models.fbnet_v2 import fbnet
ModuleNotFoundError: No module named 'mobile_cv'

Hello again.
as you mentioned in the another issue, i tried to convert your model to TRT with torch2trt.
here is a part of my code:
dummy_input_to_model_init_encoder = torch.ones((1, 3, 128, 128)).cuda()
dummy_input_to_model_update_search = torch.ones((1, 3, 256, 256)).cuda()
model_encoder_update = torch2trt(model, [dummy_input_to_model_update_search, dummy_input_to_model_init_encoder])

after execution i got the following error:
TypeError: forward() takes 2 positional arguments but 3 were given.

how to fix it?

Hi. Is this behavior intended or is it a mistake, that a batch size for val set in tracker.yaml is set to 1?