The oneflow-xrt from oneflow-inc

oneflow-xrt's Introduction

OneFlow-XRT

OneFlow-XRT is an OneFlow extension that provides an easy to use, flexible and unified way to integrate third-party computing engines in OneFlow.

OneFlow-XRT has support multiple third-party computing engines, such as XLA and TensorRT. Different engines support different backend hardware. For the same computing graph, XRT allows multiple computing engines to be used in combination to obtain better acceleration effects.

engine	device	inference	training
XRT-XLA	X86 CPU + CUDA	✔	✔
XRT-TensorRT	CUDA	✔	✔ only no weights op
XRT-OpenVINO	X86 CPU	✔	✔ only no weights op
XRT-TVM	-	-	-

Architecture

Installation

pip

To install OneFlow-XRT via pip, use the following command:

# TODO
# pip3 install oneflow_xrt

# run the following commands according to your needs
# pip3 install oneflow_xrt_xla
# pip3 install oneflow_xrt_tensorrt
# pip3 install oneflow_xrt_openvino

Building From Source

Prerequisites

install cmake
install oneflow
install CUDA if oneflow supports CUDA device or building TensorRT
install bazel if building XLA
download and unzip TensorRT if building TensorRT
download and unzip OpenVINO runtime if building OpenVINO

Get the OneFlow-XRT Source

git clone https://github.com/Oneflow-Inc/oneflow-xrt

building

Inside OneFlow-XRT source directory, then run the following command to install oneflow_xrt:

python3 setup.py install

The following components are optional, run the command to install it according to your needs,

oneflow_xrt_xla

BUILD_XLA=ON python3 setup.py install

oneflow_xrt_tensorrt

BUILD_TENSORRT=ON TENSORRT_ROOT=/home/TensorRT-8.4.0.6 python3 setup.py install

oneflow_xrt_openvino

BUILD_OPENVINO=ON OPENVINO_ROOT=/home/intel/openvino_2022.1.0.643/runtime python3 setup.py install

Run A Toy Program

# python3

>>> import oneflow as flow
>>> import oneflow_xrt as ofrt
>>> m = flow.nn.Linear(3, 4).to("cuda")
>>> m = ofrt.XRTModule(m, engine=["tensorrt"])
>>> x = flow.randn(4, 3, device="cuda")
>>> y = m(x)
>>> print(y)
tensor([[ 0.2404,  0.7121,  0.4473,  0.4782],
        [-0.8697,  1.5353,  0.2829,  0.4772],
        [-0.3865, -1.2719,  1.0911,  0.1179],
        [ 0.3779,  0.7363,  0.5319,  0.3167]], device='cuda:0', dtype=oneflow.float32)

Documentation

Roadmap

TODO

oneflow-xrt's People

Contributors

Stargazers

Watchers

oneflow-xrt's Issues

Dismatch device between producer and consumer

在测试 adam 的时候遇到1个问题：多个输入中，weight 和 weight_grad 是来自 non-cpu device，而 bias_correction_factor 来自 cpu device。导致在 compile 时，无法确定 input 到底是 device 上的还是 cpu 上的。

在oneflow的compile过程中，adam gen pass 中插入 bias_correction_factor op 时，一般插入的是 cpu 的 op。然后在之后的 build task graph 阶段，会再在 bias_correction_factor 和 adam optimizer 之间再插入一个 copy h2d 来完成转换。

但 xrt 的编译过程是发生在 build task graph 阶段之前，就无法检测到该 copy h2d 节点。其 adam xrt node 的输入就会来自不同的 device，编译成子图时，子图的 inputs 也就来自不同的 device 了。

[Update Request] XRTModule need be updated

oneflow 中 nn.Graph 的内部接口有一些 breaking update（譬如 finish_complie_and_init_runtime 被换成其他实现）。

XRTModule 需要随着一起更新。

Recommend Projects

oneflow-inc / oneflow-xrt Goto Github PK