pytorch / tensorrt Goto Github PK

With INT8 about to land, would be a pain to have to calibrate from scratch every time. There should be some mechanism to save and load modules with the TRT engine included.

A neural network model may have some operators which aren't supported by TensorRT.
When TensorRT cannot compile a subgraph, can the execution of the subgraph invoke torch operators again? Can a model execution mix TensorRT and vanilla torch operators?

There will likely be some operations that still need plugins, there needs to be some workflow for using them (maybe though the custom converter workflow).

When I run the trt compiler with libtorch 1.5.0 build hash 4ff3872a2099993bf7e8c588f7182f3df777205b I receive the following warning Warning: Deprecation warning: In a future PyTorch release torch.full will no longer return tensors of floating dtype by default. Instead, a bool fill_value will return a tensor of torch.bool dtype, and an integral fill_value will return a tensor of torch.long dtype. Set the optional dtypeorout arguments to suppress this warning. (operator() at ../aten/src/ATen/native/TensorFactories.cpp:361). I dont think it is breaking although it should probably be addressed sooner than later.

Is there a build process outlined for getting this library built on a Jetson Nano or another ARM64 platform (TX2, Xavier, etc.)? The Bazel WORKSPACE seems rather dependent on X86 binaries and since I am new to Bazel I am having a hard time figuring out how to get it to build on Jetson Nano that already has TensorRT, CUDA, cuDNN, and libtorch/pytorch installed. Thanks

as the title.

There are a whole bunch of TODOs re: the Arg class

• Implement Tag checking for the IValue unwrap functions
• Potentially safely consolidate unwrap functions to only support valid types

Need to support:

• Nearest Neighbor
• Linear
• Bilinear
• Trilinear
• Bicubic
• Area

Relevant Documentation:

• https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-601/tensorrt-api/c_api/classnvinfer1_1_1_i_network_definition.html#a12b4eb2416951d1f575d3fc263395bb9

• https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-601/tensorrt-api/c_api/classnvinfer1_1_1_i_resize_layer.html

• https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.interpolate

• https://github.com/pytorch/pytorch/blob/12fb8148e48d3e39d16832a84ac4b59182a00c4e/aten/src/ATen/native/native_functions.yaml#L6159

Having converters for inplace operators (add_) is potentially unsafe. We should use the functionalization pass in the lowering phase to switch these ops to their functional versions.

Right now the same workflow to run FP32 models when used in FP16 gives bad results.

We should either look at containerizing the build, or find someway to pull TensorRT, CUDA etc. with bazel. We already do this with libtorch. This should make it less likely builds will fail because of peoples environment

Seems like if you naively launch the top level test suite with bazel the tests hit resource starvation and everything starts hanging. Need to investigate the correct settings to run these tests.

When compiling libtrtorch.so converters and evaluators are stripped since they are registered using self registering types and those variables have no uses. As a workaround we just include the implementation cpp files at the end of the registry cpp files to keep the symbols around. We should get rid of those includes and correctly handle the issue.

With Libtorch 1.5.0 out we are going to be updating soon:

• Libtorch 1.5.0
• CUDA 10.2
• cuDNN 7.6
• TensorRT 7.0

Hi,

I configure the env and try to build such project.
INFO: Analyzed target //:libtrtorch (34 packages loaded, 1830 targets configured).
INFO: Found 1 target...
INFO: Deleting stale sandbox base /home/xxx/.cache/bazel/_bazel_xxx/2ed6247d0d5238dab6f58f41a8e8ad4b/sandbox
ERROR: missing input file 'external/tensorrt/lib/x86_64-linux-gnu/libnvinfer.so', owner: '@tensorrt//:lib/x86_64-linux-gnu/libnvinfer.so'
ERROR: /home/xxx/2ed6247d0d5238dab6f58f41a8e8ad4b/external/tensorrt/BUILD.bazel:15:1: @tensorrt//:nvinfer_lib: missing input file '@tensorrt//:lib/x86_64-linux-gnu/libnvinfer.so'
Target //:libtrtorch failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /home/xxx/.cache/bazel/_bazel_xxxd5238dab6f58f41a8e8ad4b/external/tensorrt/BUILD.bazel:15:1 1 input file(s) do not exist
INFO: Elapsed time: 2.964s, Critical Path: 0.13s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

could you help solve such errors or provide more env setting details?
I have email @narendasan and please check it.
Thanks.

We need to create a system that lets us build Python manylinux builds instead of just supporting Ubuntu 18.04. This will probably be a Docker system like used in DALI

Need to support:

• conv1d (Need to check on fusing and if it would be better to map to conv2d)
• conv_transpose1d
• conv_transpose2d
• conv_transpose3d
• conv3d?

Relevant Documentation:

• https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-601/tensorrt-api/c_api/classnvinfer1_1_1_i_network_definition.html#a108f9f6d884e642a9e3812b5175fbc55

• https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-601/tensorrt-api/c_api/classnvinfer1_1_1_i_convolution_layer.html

• https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-601/tensorrt-api/c_api/classnvinfer1_1_1_i_network_definition.html#a64423656a50a3bff760743e371e53dff

• https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-601/tensorrt-api/c_api/classnvinfer1_1_1_i_deconvolution_layer.html

• https://pytorch.org/docs/stable/nn.functional.html#convolution-functions

• https://github.com/pytorch/pytorch/blob/12fb8148e48d3e39d16832a84ac4b59182a00c4e/aten/src/ATen/native/native_functions.yaml#L765

Hello. I am working with a model that takes in a tuple of inputs of different sizes. Is there a way to handle this within the existing TRTorch framework? I am attempting to compile the model with the following settings but get the accompanied error. I am running form source version on commit 247c748.

Compile settings:

compile_settings = {
    "input_shapes": [
        {
            "min": ([1, 3, 180, 320], [1,49,2]),
            "opt": ([1, 3, 180, 320], [1,49,2]),
            "max": ([1, 3, 180, 320], [1,49,2])
        }, # For static size [1, 3, 224, 224]
    ],
#     "op_precision": torch.half # Run with FP16
}

Error:

TypeError: (): incompatible function arguments. The following argument types are supported:
    1. (self: trtorch._C.InputRange, arg0: List[int]) -> None

Invoked with: <trtorch._C.InputRange object at 0x7fda486f67b0>, ([1, 3, 180, 320], [1, 49, 2])

The forward function of my model is the following:

    def forward(self,x,seq):
        
        x = self.input_conv(x)
        x = self.input_fc(x).unsqueeze(0)
        
        actions = self.action_fc(seq)
        x, hidden = self.lstm(actions, (x, x))
        
        output = self.output_fc(x)
        
        return output

Thanks

Lowering should run standard jit passes like dead code elimination

The system works pretty well for traced models, but not much work has been done with torch script models that have branching. I noticed some common cases that we should be able to handle include branching for none arguments such as graphs like this:

  %50 : Function = prim::Constant[name="linear"]()
  %53 : bool = prim::Constant[value=0]() # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1368:7
  %54 : None = prim::Constant() # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1368:40
  %55 : int = prim::Constant[value=2]() # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1368:22
  %56 : int = prim::Constant[value=1]() # :0:0
  %57 : int = aten::dim(%input1.1) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1368:7
  %58 : bool = aten::eq(%57, %55) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1368:7
  %59 : bool = prim::If(%58) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1368:7
    block0():
      %60 : bool = aten::__isnot__(%94, %54) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1368:28
      -> (%60)
    block1():
      -> (%53)
  %input2.1 : Tensor = prim::If(%59) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1368:4
    block0():
      %bias0.4 : Tensor = prim::unchecked_cast(%94)
      %101 : Tensor = aten::linear(%input1.1, %95, %bias0.4)
      -> (%101)
    block1():
      %106 : Tensor? = prim::Constant()
      %107 : Tensor = aten::linear(%input1.1, %95, %106)
      %67 : bool = aten::__isnot__(%94, %54) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1373:11
      %output0.6 : Tensor = prim::If(%67) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1373:8
        block0():
          %bias1.4 : Tensor = prim::unchecked_cast(%94)
          %output0.7 : Tensor = aten::add_(%107, %bias1.4, %56) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1374:12
          -> (%output0.7)
        block1():
          -> (%107)
      -> (%output0.6)

The tests pass functionality wise, but in the clean up, the test segfaults. It seems like this is a issue others have seen in libtorch pytorch/pytorch#17658

This is the stream that is used right now
c10::cuda::CUDAStream stream = c10::cuda::getCurrentCUDAStream(inputs[0].device().index());

Will this always be correct? What are the cases where this will give an incorrect stream?

Hi Naren,
glad to see that you check-in py binding and test.
TRTorch needs to install pytorch and torchvision again and I know it is easy to build trt from scratch.
But as a developer, I always build and set pytorch env locally and do need to install it again. Could you help provide options to call local pytorch instead of installing again. @narendasan

Thanks,
Alan

I am running TRTorch V0.0.2 and trying to compile the resnet18 version of the model presented here https://github.com/NVIDIA-AI-IOT/trt_pose in C++. When I try to compile the graph in C++ I get the following error (note that if I try to compile in python there is no error and the kernel dies):

ERROR: [TRTorch Conversion Context] - %input.8 : Tensor = aten::_convolution(%218, %self.1.cmap_up.0.weight, %self.1.cmap_up.0.bias, %7, %5, %5, %11, %4, %8, %10, %10, %11) # /home/michael/anaconda3/envs/nvidia_dl/lib/python3.7/site-packages/torch/nn/modules/conv.py:790:0: kernel weights has count 2097152 but 4194304 was expected
ERROR: [TRTorch Conversion Context] - %input.8 : Tensor = aten::_convolution(%218, %self.1.cmap_up.0.weight, %self.1.cmap_up.0.bias, %7, %5, %5, %11, %4, %8, %10, %10, %11) # /home/michael/anaconda3/envs/nvidia_dl/lib/python3.7/site-packages/torch/nn/modules/conv.py:790:0: count of 2097152 weights in kernel, but kernel dimensions (4,4) with 512 input channels, 512 output channels and 1 groups were specified. Expected Weights count is 512 * 4*4 * 512 / 1 = 4194304
ERROR: [TRTorch Conversion Context] - %input.8 : Tensor = aten::_convolution(%218, %self.1.cmap_up.0.weight, %self.1.cmap_up.0.bias, %7, %5, %5, %11, %4, %8, %10, %10, %11) # /home/michael/anaconda3/envs/nvidia_dl/lib/python3.7/site-packages/torch/nn/modules/conv.py:790:0: kernel weights has count 2097152 but 4194304 was expected
ERROR: [TRTorch Conversion Context] - %input.8 : Tensor = aten::_convolution(%218, %self.1.cmap_up.0.weight, %self.1.cmap_up.0.bias, %7, %5, %5, %11, %4, %8, %10, %10, %11) # /home/michael/anaconda3/envs/nvidia_dl/lib/python3.7/site-packages/torch/nn/modules/conv.py:790:0: count of 2097152 weights in kernel, but kernel dimensions (4,4) with 512 input channels, 512 output channels and 1 groups were specified. Expected Weights count is 512 * 4*4 * 512 / 1 = 4194304

If I add in the line trtorch::CheckMethodOperatorSupport(script_model, "forward"); just before compiling I get the error

terminate called after throwing an instance of 'trtorch::Error'
  what():  [enforce fail at core/conversion/conversion.cpp:240] Expected schema to be true but got false
Unable to get schema for Node %245 : (Tensor, Tensor) = prim::TupleConstruct(%242, %244) (conversion.VerifyCoverterSupportForBlock

So I think this is expected behavior for a converter not being found but I am not entirely sure. If it is a converter missing it looks like it is related to handling the TupleConstruct since the model returns a tuple. Another model that I have tested and that just returns a tensor works fine. Are there any plans to get support for this implemented and is it as straight forward as adding another converter?

Need to support:

• Adaptive Max Pooling 2D

Relevant Documentation:

• https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-601/tensorrt-api/c_api/classnvinfer1_1_1_i_network_definition.html#a9bf1bc8eab4861a6203c090c8ba3d819

• https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-601/tensorrt-api/c_api/classnvinfer1_1_1_i_pooling_layer.html

• https://pytorch.org/docs/stable/nn.functional.html#pooling-functions

• https://github.com/pytorch/pytorch/blob/12fb8148e48d3e39d16832a84ac4b59182a00c4e/aten/src/ATen/native/native_functions.yaml#L5726

Add a section to _compiler.py that wraps ScriptFunctions in Modules to compile the graph.

Right now as if a graph is not supported, it will still run through the converters until it hits an unsupported op. We should have a light weight way to determine support and what ops need to be implemented to support the graph.

Right now as a side effect of registering an engine in the execution manager, a new op specifically for the engine is registered in the op registry. For instance running a ResNet backbone will be implemented with a new op with schema trt::execute_engine_55d1de7b7b50(Tensor in_input_38) -> (Tensor). We could also have a generic op like trt::execute_engine(int id, Tensor in_input_38, ...) -> (Tensor, ...) and rely on information in the engine manager to run the correct engine, as long as variadic arguments (and returns) work.

I have a test model traced by below:

from torchvision.models.alexnet import alexnet
import torch

model = alexnet(pretrained=True).eval().cuda()
x = torch.ones((1, 3, 224, 224)).cuda()

traced_model = torch.jit.trace(model, x)
traced_model.save("alexnet.pt")

then I use trtorchexec, error occur:

error loading the model
open file failed, file path: ../../../../../data/alexnet.pt (FileAdapter at ../../caffe2/serialize/file_adapter.cc:11)
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x6a (0x7f21e5ae192a in /trtorch/bin/TRTorch/bazel-TRTorch/external/libtorch/lib/libc10.so)
frame #1: caffe2::serialize::FileAdapter::FileAdapter(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x2fa (0x7f21eb536b1a in /trtorch/bin/TRTorch/bazel-TRTorch/external/libtorch/lib/libtorch.so)
frame #2: torch::jit::load(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, c10::optional<c10::Device>, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&) + 0x40 (0x7f21ec811c70 in /trtorch/bin/TRTorch/bazel-TRTorch/external/libtorch/lib/libtorch.so)
frame #3: ./trtorchexec() [0x411bd4]
frame #4: __libc_start_main + 0xf0 (0x7f21be3ce830 in /lib/x86_64-linux-gnu/libc.so.6)
frame #5: ./trtorchexec() [0x410459]

my torch version is:

>>> import torch
>>> torch.__version__
'1.4.0'
>>> torch.version.cuda
'10.1'
>>> 

my libtorch use trtorch default libtorch.

How can I solve it?

Right now there are a couple ways that batch norm is handled, there will be a folding pass, that merges applicable Conv2D -> BatchNorm patterns, there is a convertor that acts as a fallback. It would be nice if we could move that converter to be a lowering pass that synthesizes a subgraph of lower level op converters instead of having a large batch norm specific one. Some of the work is done in the unused unpack_batch_norm pass but it needs to be firmed up to support multiple dimensions.

Need to support:

• ReLU
• tanh
• hardtanh
• sigmoid
• Leaky ReLU
• eLU
• SeLU
• SoftSign
• SoftPlus
• ReLU6
• GeLU
• PReLU

Relevant Documentation:

• https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-601/tensorrt-api/c_api/classnvinfer1_1_1_i_network_definition.html#a0382282a59e3841726f6c29c4ac1f684

• https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-601/tensorrt-api/c_api/classnvinfer1_1_1_i_activation_layer.html

• https://pytorch.org/docs/stable/nn.functional.html#non-linear-activation-functions

• https://github.com/pytorch/pytorch/blob/12fb8148e48d3e39d16832a84ac4b59182a00c4e/aten/src/ATen/native/native_functions.yaml#L2337

DEBUG: [TRTorch - Debug Build] - Stride: [-1, -1]
DEBUG: [TRTorch - Debug Build] - Window: [-1, -1]
ERROR: [TRTorch Conversion Context] - Parameter check failed at: ../builder/Network.cpp::addPoolingNd::690, condition: allDimsGtEq(windowSize, 1) && volume(windowSize) < MAX_KERNEL_DIMS_PRODUCT
terminate called after throwing an instance of 'trtorch::Error'
  what():  [enforce fail at core/conversion/converters/impl/pooling.cpp:92] Expected new_layer to be true but got false
Unable to create average pooling layer from node: %x.1 : Tensor = aten::adaptive_avg_pool2d(%447, %6) # /home/narens/.local/lib/python3.6/site-packages/torch/nn/functional.py:889:0

Because the output size is calculated based on the input size and target output size, you get incorrect stride and window when using dynamic shape. Hopefully will get fixed with the work done on generating shape tensors

Hi,

I tried installing TRTorch on my Ubuntu 16.04, PyTorch 1.5 compiled from source, CUDA 10.0, CUDNN 7.6.

I am getting a symbol error in all configurations I tried. I am grateful for any help.

seh2bp@trtorch:/workspace$ python -c "import torch; import trtorch" Traceback (most recent call last): File "<string>", line 1, in <module> File "/opt/conda/lib/python3.7/site-packages/torch/__init__.py", line 165, in <module> from torch._C import * ImportError: /opt/conda/lib/python3.7/site-packages/torch/lib/libshm.so: undefined symbol: _Z8_THErrorPKciS0_z

ldd output is here:

seh2bp@trtorch:/workspace$ ldd /opt/conda/lib/python3.7/site-packages/trtorch/lib/libtrtorch.so linux-vdso.so.1 (0x00007ffdff3c7000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f338cc4d000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f338c8af000) libnvinfer.so.7 => /opt/tensorrt/TensorRT-7.0.0.11/lib/libnvinfer.so.7 (0x00007f337ec34000) libcublas.so.10.0 => /usr/local/cuda-10.0/targets/x86_64-linux/lib/libcublas.so.10.0 (0x00007f337a69e000) libcudnn.so.7 => /usr/lib/x86_64-linux-gnu/libcudnn.so.7 (0x00007f3362e8d000) libtorch.so => /workspace/scer-docker/trtorch/files/libtorch/lib/libtorch.so (0x00007f3362c8b000) libtorch_cuda.so => /workspace/scer-docker/trtorch/files/libtorch/lib/libtorch_cuda.so (0x00007f33211cc000) libtorch_cpu.so => /workspace/scer-docker/trtorch/files/libtorch/lib/libtorch_cpu.so (0x00007f3311fae000) libtorch_global_deps.so => /workspace/scer-docker/trtorch/files/libtorch/lib/libtorch_global_deps.so (0x00007f3311da9000) libc10_cuda.so => /workspace/scer-docker/trtorch/files/libtorch/lib/libc10_cuda.so (0x00007f3311b73000) libc10.so => /workspace/scer-docker/trtorch/files/libtorch/lib/libc10.so (0x00007f3311923000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f331170b000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f331131a000) /lib64/ld-linux-x86-64.so.2 (0x00007f338cfd6000) libcudart.so.10.0 => /usr/local/cuda-10.0/targets/x86_64-linux/lib/libcudart.so.10.0 (0x00007f33110a0000) libmyelin.so.1 => /opt/tensorrt/TensorRT-7.0.0.11/lib/libmyelin.so.1 (0x00007f331088f000) libnvrtc.so.10.0 => /usr/local/cuda-10.0/targets/x86_64-linux/lib/libnvrtc.so.10.0 (0x00007f330f273000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f330f06f000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f330ee50000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f330ec48000) libcudart-80664282.so.10.2 => /workspace/scer-docker/trtorch/files/libtorch/lib/libcudart-80664282.so.10.2 (0x00007f330e9c7000) libnvToolsExt-3965bdd0.so.1 => /workspace/scer-docker/trtorch/files/libtorch/lib/libnvToolsExt-3965bdd0.so.1 (0x00007f330e7bd000) libgomp-75eea7e8.so.1 => /workspace/scer-docker/trtorch/files/libtorch/lib/libgomp-75eea7e8.so.1 (0x00007f330e598000)

I tried different configurations:

• use the downloaded libtorch built for cuda 10.2, I get the same error as above
• use the installed libtorch (for a reason I don't understand I have 2 versions on in
  .../site-packages/torch and one in .../site-packages/torch-1.5.0-py3.7-linux-x86_64.egg/torch. Build with .../site-packages/torch fails because it is missing optimizer.h. I will not give further detail as I believe it will just complicate the issue unnecessarily. Any help is welcome.

TensorRT engines are GPU specific, we should create an API (ideally one that can handle torch devices) to set the correct device before building the engine.

https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__DEVICE.html#group__CUDART__DEVICE_1g159587909ffa0791bbe4b40187a4c6bb

I used to use torch2trt to convert pytorch module, could you explain the advatage over torch2trt?

If the model contain op that tensorrt don't support, can trtorch convert it to engine?
Otherwise run the op supported by tensorrt with tensorrt, and other use libtorch?

I really appreciate for your great works, if you can answer my doubts, I will be very grateful.

Right now the Converter API expects lambdas of the type: (ConversionCtx* ctx, torch::jit::Node* n, kwargs* args) -> bool

Questions:

1. The bool return is a quick way to signal success or failure in converting the op. This could be something more descriptive

2. Right now it is the responsibility of converters to log associations between torch::jit::Values and nvinfer1::ITensorss so that later is significantly easier to assemble the arguments to a node. It may be nice if you could return a vector of unions of IValues and ITensors and have the converter executor do the insertions. This would probably need to rely on some guarantee that order of return is easy to determine and constant

Provide a solution to cross compile TRTorch for aarch64. We already have cc_toolchains in https://www.github.com/NVIDIA/DL4AGX. But we need to determine a good way to build this hermetically without having to setup a full jetpack container or similar

It would be nice if we are able to define converters in python for users to add support for a custom operator

It would be nice to be able to accept TRT engines built outside TRTorch and register them in the execution system.

Right now the main documentation is the comments in the header file. We should probably have a static site for reference and some short tutorials

We currently use a handwritten converter for batch norm which fuses the operations into a single convolution. It would be easier to use the _jit_pass_fold_convbn pass in lowering instead where we can.

To help debugging it would be great to use compilation groups to indicate the work a TRT engine is doing. (This is a good tutorial on this https://jott.live/markdown/Writing%20a%20Toy%20Backend%20Compiler%20for%20PyTorch)

https://pytorch.org/tutorials/advanced/torch_script_custom_classes.html

About one in 20 times, test will complain about misshapen weights, but when rerun, the test passes

We want to be able to use TensorRT's PTQ using a PyTorch dataset to support INT8 execution.

It would be nice if we can support in bool operator==(const nvinfer1::Dims& in1, const nvinfer1::Dims& in2) to return true if we can safely broadcast / reshape between the two Dims (or maybe this should be another function)

Right now you need to use the trt::execute_engine op to run a TensorRT engine. Being able to run the engine class like a module may be nice. Code is stubbed out but need to find the right interface class for the function.

We either need to make the Python Int8Calibrator class from TensorRT compatible with TRTorch or we need to create a system that can bridge the gap.

Now that serialization works, in deployment scenarios with limited resources, it would be nice to have a reduced size library which can run TRTorch modules. All that should be required is the execution section of the core library.