❓ Question I used Torch-TensorRT to compile the torch model

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

This is a similar compilation error. <div class="snippet-clipboard-content notrans

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

❓ [Question] How to solve this warning: Detected this engine is being instantitated in a multi-GPU system with multi-device safe mode disabled. about tensorrt HOT 6 CLOSED

demuxin commented on June 3, 2024

❓ [Question] How to solve this warning: Detected this engine is being instantitated in a multi-GPU system with multi-device safe mode disabled.

from tensorrt.

Comments (6)

demuxin commented on June 3, 2024 1

Hi @gs-olive, this is not right, there is complie error:

error: ‘set_multi_device_safe_mode’ is not a member of ‘torch_tensorrt::core::runtime’
   27 |         torch_tensorrt::core::runtime::set_multi_device_safe_mode(true);

from tensorrt.

demuxin commented on June 3, 2024 1

This is a similar compilation error.

error: ‘torch::ops’ has not been declared
   24 |         torch::ops::tensorrt::get_multi_device_safe_mode(true);

I searched the libtorch and torch-TensorRT header files, and there are no functions related to multi_device_safe_mode.

from tensorrt.

demuxin commented on June 3, 2024

Moreover, when the model infers using forward function, there are these warnings:

WARNING: [Torch-TensorRT] - Using default stream in enqueue()/enqueueV2()/enqueueV3() may lead to performance issues due to additional cudaDeviceSynchronize() calls by TensorRT to ensure correct synchronizations. Please use non-default stream instead.
WARNING: [Torch-TensorRT] - Using default stream in enqueue()/enqueueV2()/enqueueV3() may lead to performance issues due to additional cudaDeviceSynchronize() calls by TensorRT to ensure correct synchronizations. Please use non-default stream instead.
WARNING: [Torch-TensorRT] - Using default stream in enqueue()/enqueueV2()/enqueueV3() may lead to performance issues due to additional cudaDeviceSynchronize() calls by TensorRT to ensure correct synchronizations. Please use non-default stream instead.

from tensorrt.

gs-olive commented on June 3, 2024

Hi @demuxin - thanks for the report - we likely need to add getter/setter methods to toggle this value in C++ as well:

TensorRT/core/runtime/runtime.cpp

Line 10 in 07c5b07

bool MULTI_DEVICE_SAFE_MODE = false;

Similar to the following functions:

TensorRT/core/runtime/register_jit_hooks.cpp

Lines 121 to 122 in 07c5b07

 m.def("get_multi_device_safe_mode", []() -> bool { return MULTI_DEVICE_SAFE_MODE; }); 

 m.def("set_multi_device_safe_mode", [](bool multi_device_safe_mode) -> void {

We are aware of the default stream warning and are working on this. It should not have a substantial effect on inference from what I've seen

from tensorrt.

gs-olive commented on June 3, 2024

These may actually already be accessible in C++, prior to inference, could you try adding the line:
torch_tensorrt::core::runtime::set_multi_device_safe_mode(true)

from tensorrt.

gs-olive commented on June 3, 2024

Does torch::ops::tensorrt::get_multi_device_safe_mode() exist, or does this also cause a compilation error?

from tensorrt.

❓ [Question] How to solve this warning: Detected this engine is being instantitated in a multi-GPU system with multi-device safe mode disabled. about tensorrt HOT 6 CLOSED

Comments (6)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	m.def("get_multi_device_safe_mode", []() -> bool { return MULTI_DEVICE_SAFE_MODE; });
	m.def("set_multi_device_safe_mode", [](bool multi_device_safe_mode) -> void {