• Forward - A library for high performance deep learning inference on NVIDIA GPUs
  • Forward
  • Features
  • Quick Start
    • Prerequisites
    • Build with CMake
      • CMake build arguments
    • Unit Test
    • Use Forward-Python
      • More Usages
  • FAQ
  • Models & Operators
    • Models
    • Operators
  • Contribution
  • Contributors
  • License

[中文版]

Forward is a library for high performance deep learning inference on NVIDIA GPUs. It provides a well-designed scheme that directly parse Tensorflow/PyTorch/Keras models to high-performance engine based on TensorRT. Compared to TensorRT, it is easy-to-use and easy-to-expand. So far, Forward supports not only mainstream deep learning models in CV, NLP and Recommend fields, but also some advanced models such as BERT, GAN, FaceSwap, StyleTransfer.

• Utilize TensorRT API and customized operators for high-performance deep learning inference.
• Support not only mainstream deep learning models in CV, NLP and Recommend fields, but also advanced models such as BERT, GAN, FaceSwap, StyleTransfer.
• Support FLOAT/HALF/INT8 infer modes.
• Easy to use: Load directly Tensorflow(.pb)/PyTorch(.pth)/Keras(.h5) models and then do inference with TensorRT.
• Easy to expand: Register customized layers refer to add_support_op.md.
• Provide C++ and Python interfaces.

• NVIDIA CUDA >= 10.0, CuDNN >= 7 (Recommended version: CUDA 10.2 )
• TensorRT >= 7.0.0.11, (Recommended version: TensorRT-7.2.1.6)
• CMake >= 3.10.1
• GCC >= 5.4.0, ld >= 2.26.1
• (Pytorch) pytorch == 1.3.1
• (Tensorflow) TensorFlow == 1.15.0 (download Tensorflow 1.15.0 and unzip it to source/third_party/tensorflow/lib)
• (Keras) HDF 5

Generate Makefiles or VS project (Windows) and build. Forward can be built for different framework, such as Fwd-Torch, Fwd-Python-Torch, Fwd-Tf, Fwd-Python-Tf, Fwd-Keras, Fwd-Python-Keras, which controlled by CMake options. For example, Fwd-Python-Tf is built as below.

mkdir build
cd build

cmake ..  \
-DTensorRT_ROOT=/path/to/TensorRT \ 
-DENABLE_LOGGING=ON \  
-DENABLE_PROFILING=ON \  
-DENABLE_DYNAMIC_BATCH=ON \ 
-DBUILD_PTYHON_LIB=ON \
-DENABLE_TORCH=OFF \  
-DENABLE_TENSORFLOW=ON \ 
-DENABLE_KERAS=OFF \ 

make -j

• TensorRT_ROOT [Required]: Path to the TensorRT installation directory containing libraries
• More CMake options refer to CMake Options

When the project is built, unit_test can be used to verify the project is successfully built.

cd build/bin
./unit_test --gtest_filter=TestTfNodes.*

When the project is successfully built, the Forward-Python library can be found in the build/bin directory, named as forward.cpython.xxx*.so in Linux or forward.xxx*.pyd in Windows. Forward-Python library should be copied to the workspace directory of Python project. For example, the directory is organized as:

---- workspace
   |
   -- test.py
   |
   -- forward.cpython.xxx*.so

Then, test.py can import Forward to perform high performance deep learning inference.

# test.py

import forward
import numpy as np

# 1. BUILD step: load TensorFlow-Bert model to build Forward engine
builder = forward.TfBuilder()
batch_size = 16
infer_mode = 'float32'  # Infer mode: 'float32' / 'float16' / 'int8_calib' / 'int8'

# dict_type dummy input
dummy_input = {"input_ids" : np.ones([batch_size , 48], dtype='int32'), 
               "input_mask" : np.ones([batch_size , 48], dtype='int32'),
               "segment_ids" : np.ones([batch_size , 48], dtype='int32')}

# build engine
builder.set_mode(infer_mode); # optional, 'float32' is default.
model_path = 'bert_model.pb'
tf_engine = builder.build(model_path, dummy_input)

need_save = True
if need_save:
    # save engine
    engine_path = 'path/to/out/engine'
    tf_engine.save(engine_path)

    # load saved engine
    tf_engine = forward.TfEngine()
    tf_engine.load(engine_path)

# 2. FORWARD step: do inference with Forward engine
inputs = dummy_input
outputs = tf_engine.forward(inputs) # dict_type outputs

Notice: The name of INPUT in models can be viewed by model viewers, such as Netron.

• PyTorch usages
• TensorFlow usages
• Keras usages

FAQ

• Infer modes
• engine with INT8 mode cannot be built
• Core dumped in multi-thread scenarios

• CV
• NLP
• Recommender

• PyTorch
• TensorFlow

CONTRIBUTING

Aster JIAN

Zexi YUAN

Ao LI

Paul LU

JettHu

Ryosuke1eep

Any form of contribution is welcome. The above contributors have been officially released by Tencent.

We very much welcome developers to contribute to Tencent's open source, and we will also give them incentives to acknowledge and thank them. Here we provide an official description of Tencent's open source contribution. Specific contribution rules for each project are formulated by the project team. Developers can choose the appropriate project and participate according to the corresponding rules. The Tencent Project Management Committee will report regularly to qualified contributors and awards will be issued by the official contact.

Apache License v2.0