Hello all, I've tried to test ONNXCompose with GPT2 model. I tried to compose model with preprocessor PreHuggingfaceGPT2 from module pnp. And I got an error which is:
[InvalidGraph: [ONNXRuntimeError] : 10 : INVALID_GRAPH : This is an invalid model. Type Error: Type 'tensor(int64)' of input parameter (it_140322782567616) of operator (GPT2Tokenizer) in node (gpt2tokenizer_14) is invalid.]()

I checked any problem may cause by opset, but all prepared using opset=11. Is preprocessor for GPT2 functional or should I define a customOp for preprocessor?

Thanks a lot

Hello, I have one question about custom op.
When I add a custom op in this project, I add two files in 'onnxruntime-extensions/operators/text/', which are .cc and .hpp.
But how can I set the output dimension in the .cc file?

Recently, I add attribute padding_length(type: int64_t) for GPT2Tokenizer. Unittest of GPT2Tokenizer tests not only the C++ version but also implementation with PyOp:

        @onnx_op(op_type="GPT2Tokenizer",
                 inputs=[PyCustomOpDef.dt_string],
                 outputs=[PyCustomOpDef.dt_int64, PyCustomOpDef.dt_int64],
                 attrs=["padding_length"])
        def bpe_tokenizer(s, **kwargs):
            # The user custom op implementation here.
            TestGPT2Tokenizer.pyop_invoked = True
            max_length = kwargs["padding_length"]
            input_ids = np.array(
                [TestGPT2Tokenizer.tokenizer(st_, max_length=max_length)["input_ids"] for st_ in s])
            attention_mask = np.array(
                [TestGPT2Tokenizer.tokenizer(st_, max_length=max_length)["attention_mask"] for st_ in s])
            return input_ids, attention_mask

When I run the test, I got the following error:

onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: Unable to find attribute 'padding_length' due to 'Attibute name and type don't match'.

After my investigation, I found the root of the error:

  PyCustomOpKernel(OrtApi api, const OrtKernelInfo* info, uint64_t id, const std::vector<std::string>& attrs)
      : api_(api),
        ort_(api_),
        obj_id_(id) {
    size_t size;
    for (std::vector<std::string>::const_iterator it = attrs.begin(); it != attrs.end(); ++it) {
      size = 0;
      OrtStatus* status = api_.KernelInfoGetAttribute_string(info, it->c_str(), nullptr, &size);
      if (api_.GetErrorCode(status) != ORT_INVALID_ARGUMENT) {
        std::string error_message(api_.GetErrorMessage(status));
        api_.ReleaseStatus(status);
        throw std::runtime_error(MakeString(
            "Unable to find attribute '", *it, "' due to '",
            error_message, "'."));
      }
      api_.ReleaseStatus(status);
      attrs_values_[*it] = "";
      attrs_values_[*it].resize(size);
      status = api_.KernelInfoGetAttribute_string(info, it->c_str(), &(attrs_values_[*it][0]), &size);
      if ((status != nullptr) && (api_.GetErrorCode(status) != ORT_OK)) {
        api_.ReleaseStatus(status);
        throw std::runtime_error(MakeString(
            "Unable to retrieve attribute '", *it, "' due to '",
            api_.GetErrorMessage(status), "'."));
      }
      attrs_values_[*it].resize(size - 1);
      api_.ReleaseStatus(status);
    }
  }

We will only try to fetch the attribute by string type, so we meet the incompatibility of the attribute type.

My suggestion is the the attrs parameter of onnx_op could be a dict, the key is the name and the value is the type.

Create this issue to track the use of this repository for .NET

Build ortcustomops.dll

Microsoft.ML.OnnxRuntime.InferenceSession with custom SentencepieceTokenizer operation

var sessionOptions = new Microsoft.ML.OnnxRuntime.SessionOptions();
sessionOptions.RegisterCustomOpLibraryV2("ortcustomops.dll", out var libraryHandle);

var session = new Microsoft.ML.OnnxRuntime.InferenceSession(
    "universal-sentence-encoder-multilingual-3.onnx",
    sessionOptions
);

=> How to use this repo to build ortcustomops.dll for RegisterCustomOpLibraryV2

I'm trying to use TensorFlow model (universal-sentence-encoder-multilingual_3), successfully converted to .onnx format with tf2onnx, in C#. But attempts to initialize an InferenceSession cause OnnxRuntimeException "[ErrorCode:Fail] Fatal error: SentencepieceTokenizer is not a registered function/op". Is it possible to use models converted with the flag --extra_opset in .NET?

The gpt2bs tutorial linked in the ReadMe seems to be not working. When I download the file and run it without any changes I get the following Error:
site-packages\onnxruntime_extensions\onnxprocess_tensor.py", line 264, in getattribute
return object.getattribute(self, attr)
AttributeError: '_EagerTensor' object has no attribute 'dtype'

It's still the last demo script, after fixing the error occurs when getting input size(), I get new error.

import numpy as np
from onnxruntime_extensions.onnxprocess import torch_wrapper as torch
from onnxruntime_extensions.onnxprocess import trace_for_onnx, pyfunc_from_model, pyfunc_from_custom_op, build_customop_model


def authoring_custom_ops():
    inputs = ["sentence A", "sentence B", "sentence C"]

    with trace_for_onnx(inputs) as session:
        # get traced input from session
        inputs = session.get_inputs()[0]
        input_size = inputs.size()[0]

        outputs = []

        # the following line is similar with the lambda in C++ :
        # [states...] (loop_count, condition) => {
        #     for (int i = 0; i < loop_count && condition; i++) {
        #        ...
        #     }
        # }
        # input_size : loop_count
        # True: condition
        ctr_flw = torch.control_flow()
        for states in ctr_flw.loop(input_size, True, inputs, outputs):
            # get index: i and states
            index, inputs, outputs = states
            gpt2_tokenizer = pyfunc_from_custom_op('GPT2Tokenizer', vocab_file="...", merges="...", max_length=100)
            outputs.append(gpt2_tokenizer(inputs[index]))

            ctr_flw.flow_output(outputs)

        outputs = ctr_flw.finalize()

        nn_model = pyfunc_from_model("core_model.onnx")

        result = nn_model(outputs)

        session.save_as_onnx("./e2e_model.onnx", result)
        return result


authoring_custom_ops()

I get error when I try to input states of loop:

/usr/bin/python3.8 /home/toud/Code/scratch/Python/pytorch_convert/custom_convertor/demo.py
/home/toud/Code/ort-customops/onnxruntime_extensions/onnxprocess/_session.py:223: UserWarning: the name number doesn't match the inputs', assign to the ones in the front.
  warnings.warn("the name number doesn't match the inputs', assign to the ones in the front.")
/home/toud/Code/ort-customops/onnxruntime_extensions/onnxprocess/_onnx_ops.py:71: UserWarning: The maximum opset needed by this model is only 8.
  warnings.warn('The maximum opset needed by this model is only %d.' % op_version)
Traceback (most recent call last):
  File "/home/toud/Code/scratch/Python/pytorch_convert/custom_convertor/demo.py", line 43, in <module>
    authoring_custom_ops()
  File "/home/toud/Code/scratch/Python/pytorch_convert/custom_convertor/demo.py", line 25, in authoring_custom_ops
    for states in ctr_flw.loop(input_size, True, inputs, outputs):
  File "/home/toud/Code/ort-customops/onnxruntime_extensions/onnxprocess/_tensor.py", line 581, in loop
    self.loop_states = [_EagerTensor(st_.value, st_.name) for st_ in states]
  File "/home/toud/Code/ort-customops/onnxruntime_extensions/onnxprocess/_tensor.py", line 581, in <listcomp>
    self.loop_states = [_EagerTensor(st_.value, st_.name) for st_ in states]
  File "/home/toud/Code/ort-customops/onnxruntime_extensions/onnxprocess/_tensor.py", line 264, in __getattribute__
    return object.__getattribute__(self, attr)
  File "/home/toud/Code/ort-customops/onnxruntime_extensions/onnxprocess/_tensor.py", line 34, in value
    return self.raw_data if self.raw_data else self._t
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

It's because input itself has already been a EagerTensor or it is still a list... What expected is a torch tensor...

        self.loop_states = [_EagerTensor(st_.value, st_.name) for st_ in states]

I think we should provide a common function to convert EagerTensor/TorchTensor/Numpy/List/BasisElement to EagerTensor.

Hi, I'm not getting all the stakes of this package, but is there a way to use the scripts to get a Java tokenizer ?

I converted my python model to onnx and I aiming to run it with a Java runtime, but I don't get a clear comprehension of what the extension is designed for ? Is there a way to use it with Java Sentencepiece tokenizer ?

I read on your docs that a lot of tokenizers are supported but is it only for Python ? (here)

Best regards

Is there a way in which I could see the list of the currently registered custom ops ? No only the file which is used to register the operators in the the ORT session (so.register_custom_ops_library(get_library_path()))
I have a doubt that I am able to register the same operator multiple times when using PyOp and this creates errors during the ORT inference such as :

Fail: [ONNXRuntimeError] : 1 : FAIL : Failed to add kernel for custom_node_X ai.onnx.contrib CPUExecutionProvider: Conflicting with a registered kernel with op versions.

Hi,
I’m trying to convert a pytorch model to onnx, but the conversion gives the following error:

RuntimeError: ONNX export failed: Couldn’t export Python operator qfn

The definition for qfn is:

class qfn(torch.autograd.Function):
@staticmethod
def forward(ctx, input, k):
n = float(2**k - 1)
out = torch.round(input * n) / n
return out

@staticmethod
def backward(ctx, grad_output):
grad_input = grad_output.clone()
return grad_input, None

Hi Onnxrt team,

I had a requirement of adding two custom ops, exporting to onnx, and executing it. As the first step: I had registered the custom ops and exported to onxx model. But I am not sure how to add the two kernels in onnxruntime. I got the reference for one custom kernel(https://github.com/neginraoof/CustomOperators/tree/master/CustomOperator/ort_custom_op), but I need a reference for adding two custom kernels. Kindly assist me in finding the reference for the same.

Thanks,
Darshan

It happened when I want to test how to get attribute of GPT2tokenizer.

I found there are the following API for us to get attribute, so I want to give it a shot.

  ORT_API2_STATUS(KernelInfoGetAttribute_float, _In_ const OrtKernelInfo* info, _In_ const char* name,
                  _Out_ float* out);
  ORT_API2_STATUS(KernelInfoGetAttribute_int64, _In_ const OrtKernelInfo* info, _In_ const char* name,
                  _Out_ int64_t* out);
  ORT_API2_STATUS(KernelInfoGetAttribute_string, _In_ const OrtKernelInfo* info, _In_ const char* name, _Out_ char* out,
                  _Inout_ size_t* size);

It gets attribute from OrtKernelInfo, so I have to keep the struct. I do in the following way.

struct KernelBpeTokenizer : BaseKernel {
  KernelBpeTokenizer(OrtApi api, const OrtKernelInfo* info, const VocabData* global_data)
      : BaseKernel(api, info)
      , vocab_data_(global_data)
      , token2id_cache_(30 * 1024) {
    std::cout << ort_.KernelInfoGetAttribute<std::string>(info_, "vocabulary_file") << std::endl;
  }

  static size_t const p_max_len = 1024;
  using StringConverter = std::wstring_convert<std::codecvt_utf8<char32_t>, char32_t>;

 private:
  LruCache<std::string, std::list<int>> token2id_cache_;
  std::list<int> byte_list_;
  const VocabData* vocab_data_;

struct BaseKernel {
  BaseKernel(OrtApi api) : api_(api), info_(nullptr), ort_(api_) {}
  BaseKernel(OrtApi api, const OrtKernelInfo *info) : api_(api), info_(info), ort_(api_) {}

 protected:
  OrtApi api_;  // keep a copy of the struct, whose ref is used in the ort_
  const OrtKernelInfo* info_;
  Ort::CustomOpApi ort_;
};

Then I found the tests of GPT2 failed and not only GPT2 but also many other ops.
After long-time investigation, I found the root of the bug:

const OrtApi* _GetApi(Ort::CustomOpApi& ort) {
  if (sizeof(Ort::CustomOpApi) == sizeof(OrtApi*)) {
    // The following line should be replaced when an accessor is available.
    // ort.api_ is not accessible (marked as private)
    // Ort::GetApi() returns null.
    // InitApi is missing when linking.
    // OrtGetApiBase() is missing when linking.
    const OrtApi* api = (const OrtApi*)((OrtApi**)&ort) - 1;
    /*
    // The following code checks api is equal to the expected
    // pointer stored ins Ort::CustomOpApi ort as a private member.
    // The following method can be added `const OrtApi& Api() { return api_; }`
    // to give access to this value.
    if (api != &(ort.Api())) {
      // Ort::InitApi(); - missing from link
      auto diff = (int64_t)(&(ort.Api()) - api);
      throw std::runtime_error(MakeString(
          "Internal error, pointers are different: ",
          api, "!=", &(ort.Api()), " (expected) (other value ", &(Ort::GetApi()),
          ", delta=", diff, ")."));
    }
    */
    return api;
  }
  throw std::runtime_error(MakeString(
      "Unable to get an OrtApi pointer from CustomOpApi. Variable ort is not a pointer. Size ",
      sizeof(Ort::CustomOpApi), "!=", sizeof(OrtApi*), " (expected)."));
}

In function _GetApi, it tries to retrieval OrtApi according to the relative offset in BaseKernel, so it gets wrong element when I insert info_ between api_ and ort_.

I recommend that we should think the inner implement of the C++ language as a black box, it will break our robustness if we take advantage of those inner implement.

@xadupre, could we change the the first parameter from Ort::CustomOpApi& ort to OrtApi api_? Because the BaseKernel also store the OrtApi api_.

I got errors when doing 'python setup.py install'

onnxruntime 1.8.0
python 3.7.3
cmake 3.21.1
system: Ubuntu 16.04

/usr/include/c++/5/bits/stl_vector.h:253:7: note: candidate expects 0 arguments, 1 provided
CMakeFiles/ocos_operators.dir/build.make:355: recipe for target 'CMakeFiles/ocos_operators.dir/operators/math/math.cc.o' failed
gmake[2]: *** [CMakeFiles/ocos_operators.dir/operators/math/math.cc.o] Error 1
CMakeFiles/ocos_operators.dir/build.make:369: recipe for target 'CMakeFiles/ocos_operators.dir/operators/tokenizer/gpt2_tokenizer.cc.o' failed
gmake[2]: *** [CMakeFiles/ocos_operators.dir/operators/tokenizer/gpt2_tokenizer.cc.o] Error 1
CMakeFiles/Makefile2:379: recipe for target 'CMakeFiles/ocos_operators.dir/all' failed
gmake[1]: *** [CMakeFiles/ocos_operators.dir/all] Error 2
Makefile:155: recipe for target 'all' failed
gmake: *** [all] Error 2
error: command 'cmake' failed with exit status 2

log is attached
LOG.log

I am getting an error on building android .so for onnxruntime-extensions library using command -

cmake -G Ninja -DCMAKE_TOOLCHAIN_FILE=${NDK_ROOT}/build/cmake/android.toolchain.cmake 
-DANDROID_NDK=${NDK_ROOT} -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-19 
-DOCOS_ENABLE_STATIC_LIB=OFF ../../.. && cmake --build . --config Release --parallel

Error -

[153/153] Linking CXX shared library libortcustomops.so
FAILED: libortcustomops.so
cmd.exe /C "cd . && C:\Users\shubham.j\AppData\Local\Android\Sdk\ndk\21.0.6113669\toolchains\llvm\prebuilt\windows-x86_64\bin\clang++.exe --target=aarch64-none-linux-android21 --gcc-toolchain=C:/Users/shubham.j/AppData/Local/Android/Sdk/ndk/21.0.6113669/toolchains/llvm/prebuilt/windows-x86_64 --sysroot=C:/Users/shubham.j/AppData/Local/Android/Sdk/ndk/21.0.6113669/toolchains/llvm/prebuilt/windows-x86_64/sysroot -fPIC -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security   -O2 -g -DNDEBUG  -Wl,--exclude-libs,libgcc_real.a -Wl,--exclude-libs,libatomic.a -static-libstdc++ -Wl,--build-id -Wl,--fatal-warnings -Wl,--no-undefined -Qunused-arguments -shared -Wl,-soname,libortcustomops.so -o libortcustomops.so CMakeFiles/ortcustomops.dir/shared/ortcustomops.cc.o  libocos_operators.a  _deps/googlere2-build/libre2.a  _deps/sentencepieceproject-build/src/libsentencepiece.a  -pthread  _deps/blingfire-build/libbingfirtinydll_static.a  _deps/blingfire-build/libfsaClientTiny.a  -latomic -lm && cd ."
_deps/sentencepieceproject-build/src/libsentencepiece.a(common.cc.o): In function `google::protobuf::internal::DefaultLogHandler(google::protobuf::LogLevel, char const*, int, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&)':
C:\Users\shubham.j\Onnx_KFA\onnxruntime-extensions\out\android\Release/_deps/sentencepieceproject-src/third_party/protobuf-lite/common.cc:150: undefined reference to `__android_log_write'
C:\Users\shubham.j\Onnx_KFA\onnxruntime-extensions\out\android\Release/_deps/sentencepieceproject-src/third_party/protobuf-lite/common.cc:158: undefined reference to `__android_log_write'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.

I am able to build static .a by setting DOCOS_ENABLE_STATIC_LIB=ON in the cmake command. But setting the flag to OFF, yields the above error. I require the .so file to register the library in ONNX runtime session with -

Ort::ThrowOnError(Ort::GetApi().RegisterCustomOpsLibrary((OrtSessionOptions*)session_options, custom_op_library_filename, &handle));

In the following code:

import numpy as np
from onnxruntime_extensions.onnxprocess import torch_wrapper as torch
from onnxruntime_extensions.onnxprocess import trace_for_onnx, pyfunc_from_model, pyfunc_from_custom_op, build_customop_model


def authoring_custom_ops():
    inputs = ["sentence A", "sentence B", "sentence C"]

    with trace_for_onnx(inputs) as session:
        # get traced input from session
        inputs = session.get_inputs()
        input_size = inputs.size()[0]

authoring_custom_ops()

Fristly, the script will throw error 'list' object has no attribute 'size', this is because get_inputs() always return a list even when input is single.

After I fixed the list error by inputs = session.get_inputs()[0], I got the following error:

/home/toud/Code/ort-customops/onnxruntime_extensions/onnxprocess/_onnx_ops.py:71: UserWarning: The maximum opset needed by this model is only 8.
  warnings.warn('The maximum opset needed by this model is only %d.' % op_version)
Traceback (most recent call last):
  File "/home/toud/Code/scratch/Python/pytorch_convert/custom_convertor/demo.py", line 43, in <module>
    authoring_custom_ops()
  File "/home/toud/Code/scratch/Python/pytorch_convert/custom_convertor/demo.py", line 12, in authoring_custom_ops
    input_size = inputs.size()[0]
  File "/home/toud/Code/ort-customops/onnxruntime_extensions/onnxprocess/_tensor.py", line 334, in size
    return self.create_and_verify(y, s[0])
  File "/home/toud/Code/ort-customops/onnxruntime_extensions/onnxprocess/_tensor.py", line 297, in create_and_verify
    self.ort_verify(inputs, [ts_y])
  File "/home/toud/Code/ort-customops/onnxruntime_extensions/onnxprocess/_tensor.py", line 286, in ort_verify
    result, model = cls.get_trace_session().runops(ts_from, ts_to)
  File "/home/toud/Code/ort-customops/onnxruntime_extensions/onnxprocess/_session.py", line 252, in runops
    result = oxfunc(*[ts_.numpy() for ts_ in ts_from])
  File "/home/toud/Code/ort-customops/onnxruntime_extensions/eager_op.py", line 96, in __call__
    self._ensure_ort_session()
  File "/home/toud/Code/ort-customops/onnxruntime_extensions/eager_op.py", line 67, in _ensure_ort_session
    sess = _ort.InferenceSession(self.onnx_model.SerializeToString(), self.get_ort_session_options())
  File "/home/toud/.local/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 280, in __init__
    self._create_inference_session(providers, provider_options)
  File "/home/toud/.local/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 309, in _create_inference_session
    sess = C.InferenceSession(session_options, self._model_bytes, False, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidGraph: [ONNXRuntimeError] : 10 : INVALID_GRAPH : This is an invalid model. Error in Node:shape : Node (shape)'s input 0 is marked single but has an empty string in the graph

Process finished with exit code 1

The logic of creating ONNX graph is a bit complex, @wenbingl , can you look at where is wrong?

Hello all, I'm having a problem with importing. This is happened for importing pnp too. Installation onnx and related dependencies installed correctly.
Below you can see the importing error.

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
[<ipython-input-10-42e30ef5538d>](https://localhost:8080/#) in <module>()
----> 1 from onnxruntime_extensions import ONNXCompose

ImportError: cannot import name 'ONNXCompose' from 'onnxruntime_extensions' (/usr/local/lib/python3.7/dist-packages/onnxruntime_extensions/__init__.py)

I've tried to install latest updates with python -m pip install git+https://github.com/microsoft/onnxruntime-extensions.git to overcome the problem. But installation is failed with:

Collecting git+https://github.com/microsoft/onnxruntime-extensions.git
  Cloning https://github.com/microsoft/onnxruntime-extensions.git to /tmp/pip-req-build-4fhetvrz
  Running command git clone -q https://github.com/microsoft/onnxruntime-extensions.git /tmp/pip-req-build-4fhetvrz
 
 Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done

Requirement already satisfied: onnx>=1.9.0 in /usr/local/lib/python3.7/dist-packages (from onnxruntime-extensions==0.4.2+5bf4656) (1.10.2)
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from onnxruntime-extensions==0.4.2+5bf4656) (1.21.5)
Requirement already satisfied: onnxruntime>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from onnxruntime-extensions==0.4.2+5bf4656) (1.10.0)
Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from onnx>=1.9.0->onnxruntime-extensions==0.4.2+5bf4656) (1.15.0)
Requirement already satisfied: typing-extensions>=3.6.2.1 in /usr/local/lib/python3.7/dist-packages (from onnx>=1.9.0->onnxruntime-extensions==0.4.2+5bf4656) (3.10.0.2)
Requirement already satisfied: protobuf in /usr/local/lib/python3.7/dist-packages (from onnx>=1.9.0->onnxruntime-extensions==0.4.2+5bf4656) (3.17.3)
Requirement already satisfied: flatbuffers in /usr/local/lib/python3.7/dist-packages (from onnxruntime>=1.6.0->onnxruntime-extensions==0.4.2+5bf4656) (2.0)
Building wheels for collected packages: onnxruntime-extensions
  Building wheel for onnxruntime-extensions (PEP 517) ... error
  ERROR: Failed building wheel for onnxruntime-extensions
Failed to build onnxruntime-extensions
ERROR: Could not build wheels for onnxruntime-extensions which use PEP 517 and cannot be installed directly

Can you please advise a workaround to solve the problem?
Thanks a lot

Describe the bug
I'm trying to export Pytorch model with custom op to ONNX and run it in ONNX Runtime with python API.
I followed the tutorials here
https://github.com/onnx/tutorials/tree/master/PyTorchCustomOperator
I did not use ONNX Runtime's custom operator C API, as shown in tutorial

I got an error when trying to implement the Operator in ONNX Runtime using python API.
I follow 'testRegisterCustomOpsLibrary' in https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/test/python/onnxruntime_test_python.py
code is attached

ERROR:
File "run_custom_op_onnxruntime_try.py", line 19, in
so1.register_custom_ops_library(shared_library)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Failed to get symbol RegisterCustomOps with error: ./build/lib.linux-x86_64-3.7/try_ncrelu_forward.cpython-37m-x86_64-linux-gnu.so: undefined symbol: RegisterCustomOps

run_custom_op_onnxruntime_try.py

import onnxruntime as rt
import numpy as np
import os
import torch
from onnxruntime.capi.onnxruntime_pybind11_state import Fail

shared_library = './build/lib.linux-x86_64-3.7/try_ncrelu_forward.cpython-37m-x86_64-linux-gnu.so'
if not os.path.exists(shared_library):
raise FileNotFoundError("Unable to find '{0}'".format(shared_library))

so1 = rt.SessionOptions()
so1.register_custom_ops_library(shared_library)

Dear contributors,
please excuse me if my question is stupid - unfortunately I have not found the correct way of using the onnxruntime-extensions package for my use-case; I have not succeeded even after reading the examples in the doc, googling exhaustively and debugging into the package source code...

I would like to have a simple way of registering custom operators residing in ONNX graphs. First I create a really dummy onnx model with a custom node called CUSTOM_ADD

import numpy as np
import onnx

B = 1
C = 3
H = 5
W = 5

# Inputs / outputs
i1 = onnx.helper.make_tensor_value_info('input1',  onnx.TensorProto.FLOAT, [B, C, H, W])
i2 = onnx.helper.make_tensor_value_info('input2',  onnx.TensorProto.FLOAT, [B, C, H, W])
o  = onnx.helper.make_tensor_value_info('output',  onnx.TensorProto.FLOAT, [B, C, H, W])

# Create a single Add node
nodes = [onnx.helper.make_node('Add', ['input1', 'input2'], ['output'])]

# Assemble graph, make model, infer shapes
graph_def = onnx.helper.make_graph(nodes, name="onnx-dummy", inputs=[i1, i2], outputs=[o])
model_def = onnx.helper.make_model(graph_def, producer_name="onnx-example")
model_def.opset_import[0].version = 11
model_def = onnx.shape_inference.infer_shapes(model_def)

# Replace Add with CUSTOM_ADD type
model_def.graph.node[0].op_type = 'CUSTOM_ADD'

# Save protobuf
onnx.save(model_def, "dummy.onnx")

Next, I try to register the behavior of the new node in onnx_extensions and infer the model. I define the behavior of the operation to add up the two inputs plus 6 in the my_little_custom_add_implementation function decorated by onnx_op.

import numpy as np
from onnxruntime_extensions import onnx_op, PyOp, PyOrtFunction

# Register custom operator
@onnx_op(op_type="CUSTOM_ADD", inputs=[PyOp.dt_float, PyOp.dt_float])
def my_little_custom_add_implementation(x, y):
    return x + y + 6

# Load model
model_func = PyOrtFunction.from_model('dummy.onnx')

# Create input data
B = 1
C = 3
H = 5
W = 5

input1 = np.ones((B,C,H,W)).astype(np.float32)
input2 = np.ones((B,C,H,W)).astype(np.float32)

# Infer model
outputs = model_func([input1, input2])

I get the error at inference unfortunately saying:

onnxruntime.capi.onnxruntime_pybind11_state.InvalidGraph: [ONNXRuntimeError] : 10 : INVALID_GRAPH : This is an invalid model. Error in Node: : No Op registered for CUSTOM_ADD with domain_version of 11

So the registration of the new operator over Python obviously didn't work.

Once again sorry for the beginner question... Could please anyone help me make this example work? I'd really appreciate it!

Thank you & Best regards,
RB

Using python==3.9.5, tensorflow==2.5.0, keras2onnx==1.7.0, onnxruntime==1.8.0, keras==2.4.3, tf2onnx==1.9.1

autoKeras_model = StructuredDataClassifier(max_trials=MaxTrials)
autoKeras_model.fit(x=X_train, y=y_train, validation_data=(X_valid, y_valid), epochs=Epochs, verbose=1)
ExportedautoKeras_model = autoKeras_model.export_model()

onnx_model, _ = tf2onnx.convert.from_keras(ExportedautoKeras_model )
content = onnx_model.SerializeToString()
sess = onnxruntime.InferenceSession(content)

InferenceSession throws error

sess = onnxruntime.InferenceSession(content)
  File "C:\Users\plg\Anaconda3\envs\automl04augpy395elk7120\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 283, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "C:\Users\plg\Anaconda3\envs\automl04augpy395elk7120\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 312, in _create_inference_session
    sess = C.InferenceSession(session_options, self._model_bytes, False, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidGraph: [ONNXRuntimeError] : 10 : INVALID_GRAPH : This is an invalid model. Error in Node:model/multi_category_encoding/AsString : No Op registered for AsString with domain_version of 9

AsString() op is not supported

May I have an example to create a custom op for python and .net please? Thanks

How to add custom operators using ort-customops(framework independent)?
I successfully added the operator in onnx. So for the model to be compiled using the runtime it should be added in runtime as well. Need help in this area. Thanks.

hello, i want to set dynamic input type, for example:

ONNXTensorElementDataType CustomOpVectorToString::GetInputType(size_t /index/) const {
return ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64;
};
I only have one input parameter, but the input type may be int64 or float32， how can i modify the code， thank U。

According to the documentation in https://www.tensorflow.org/api_docs/python/tf/strings/split:

If sep is None or an empty string, consecutive whitespace are regarded as a single separator, and the result will contain no empty strings at the start or end if the string has leading or trailing whitespace.

However the current implementation performs a character by character split.

There are open compliance tasks that need to be reviewed for your ort-customops repo.

Action required: 4 compliance tasks

To bring this repository to the standard required for 2021, we require administrators of this and all Microsoft GitHub repositories to complete a small set of tasks within the next 60 days. This is critical work to ensure the compliance and security of your microsoft GitHub organization.

Please take a few minutes to complete the tasks at: https://repos.opensource.microsoft.com/orgs/microsoft/repos/ort-customops/compliance

• The GitHub AE (GitHub inside Microsoft) migration survey has not been completed for this private repository
• No Service Tree mapping has been set for this repo. If this team does not use Service Tree, they can also opt-out of providing Service Tree data in the Compliance tab.
• No repository maintainers are set. The Open Source Maintainers are the decision-makers and actionable owners of the repository, irrespective of administrator permission grants on GitHub.
• Classification of the repository as production/non-production is missing in the Compliance tab.

You can close this work item once you have completed the compliance tasks, or it will automatically close within a day of taking action.

If you no longer need this repository, it might be quickest to delete the repo, too.

GitHub inside Microsoft program information

More information about GitHub inside Microsoft and the new GitHub AE product can be found at https://aka.ms/gim or by contacting [email protected]

FYI: current admins at Microsoft include @wenbingl, @faxu

Hello,

I need to use the onnxruntime-extensions for a project that runs on a Linux machine. For testing purpose I have compiled the project on my mac using ./build.sh. I get a dylib file whose size is 2.7 MB :onnxruntime-extensions/out/Darwin/Release/libortcustomops.dylib . I have used successfully this binary in my local application.

Now I want to compile it on the linux machine and I get a .so file that is now 48MB: onnxruntime-extensions-main/out/Linux/Release/libortcustomops.so

Any idea why there is such a difference between the two? Any idea on how I could reduce it?

Apart from the compiler that is different (GNU 8.3.0 for linux vs AppleClang 12.0.5.12050022), could something else explain the diff?

Thanks

Sorry for my poor knowledge of ONNXRuntime and ask a stupid question.

I am curious about this question because Session.Run() is thread-safe. So how does ONNXRuntime achieve this, by the state-less op or some concurrent mechanism?

How to implement a C++ custom op with cuda support?
Moreover, I didn't even find a detailed example about how to write a c++ costom op in C++, there is just an outdated tutorial: https://github.com/microsoft/onnxruntime-extensions/blob/main/tutorials/cpp_custom_ops_tutorial.md.

I try to execute one node at a time, this node could be ‘conv’ or ‘clip’, etc. for example. And this node not necessarily be the input node of the model. To be more concrete, I put an example model here in Fig 1. Instead of running the whole model (set the input data for the node ‘input’, and expect to get the result of the node ‘clip’), I want to execute the ‘clip’ node (set the input data for the node ‘clip’, and expect to get the result of the node ‘clip’).

Fig.1

I tried to achieve this by change the parameters of the function ‘run’, such as:
res = sess.run([output_name], {'clip': x})
But get an error: onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Invalid Feed Input Name:clip.

Seems like this parameter is used for the model with multiple input node, it doesn’t work for the case that you want to choose which node you really want to INPUT the data other than the input node.

Does anyone know how to achieve this? Or if the ORT has such a function?

Hello all, I have a problem on testing newest features. I can not install repository using python setup.py install or bash ./build.sh without Python dependencies.

Installing with python:

running install
/usr/local/lib/python3.7/dist-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  setuptools.SetuptoolsDeprecationWarning,
/usr/local/lib/python3.7/dist-packages/setuptools/command/easy_install.py:163: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  EasyInstallDeprecationWarning,
running bdist_egg
running egg_info
writing onnxruntime_extensions.egg-info/PKG-INFO
writing dependency_links to onnxruntime_extensions.egg-info/dependency_links.txt
writing requirements to onnxruntime_extensions.egg-info/requires.txt
writing top-level names to onnxruntime_extensions.egg-info/top_level.txt
reading manifest file 'onnxruntime_extensions.egg-info/SOURCES.txt'
writing manifest file 'onnxruntime_extensions.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
cmake -S /content -B build/temp.linux-x86_64-3.7 -DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/content/build/lib.linux-x86_64-3.7/onnxruntime_extensions -DOCOS_ENABLE_PYTHON=ON -DOCOS_ENABLE_CTEST=OFF -DOCOS_EXTENTION_NAME=_ortcustomops.cpython-37m-x86_64-linux-gnu.so -DCMAKE_BUILD_TYPE=Release
-- Fetch googlere2
CMake Error at CMakeLists.txt:105 (include):
  include could not find requested file:

    googlere2


CMake Error at CMakeLists.txt:136 (include):
  include could not find requested file:

    dlib


-- Fetch sentencepiece
CMake Error at CMakeLists.txt:161 (include):
  include could not find requested file:

    sentencepieceproject


-- Fetch json
CMake Error at CMakeLists.txt:191 (include):
  include could not find requested file:

    json


-- Tokenizer needed.
CMake Error at CMakeLists.txt:268 (include):
  include could not find requested file:

    blingfire


-- Fetch pybind11
CMake Error at CMakeLists.txt:342 (include):
  include could not find requested file:

    pybind11


-- Configuring incomplete, errors occurred!
See also "/content/build/temp.linux-x86_64-3.7/CMakeFiles/CMakeOutput.log".
error: command '/usr/local/bin/cmake' failed with exit code 1

Installing requested files seperately did not solve the problem, still can not locate those.

Installing without Python dependencies:

Several errors occured in this process, I've just added last part.

Any advise will be appreciated!
Thanks

Hey there, on the M1 onnxruntime-extensions is currently uninstallable, even in the nightly. This is because onnxruntime-extensions requires onnxruntime >= 1.6.0. On the M1 however, we need to install onnxruntime >= 1.9.0. The solution here would be to loosen up the requirement to > 1.6.0.
I have verified that with onnxruntime >= 1.9.0 it works on my M1 by installing the nightly from test.pypi.org
I'm curious as to why there is a requirement on the onnxruntime in the first place, as it should be interoptable.

following the script, I would like to build the project as a static library and link it into ONNXRuntime, but it failed.

log is here:

[ 75%] Linking CXX executable onnxruntime_perf_test
external/onnxruntime-extensions/libocos_operators.a(math.cc.o): In function `__static_initializati$
n_and_destruction_0(int, int)':
/home/xxx/onnxruntime_customops_integration/build/Debug/_deps/dl$
b-src/dlib/matrix/../threads/../test_for_odr_violations.h:24: undefined reference to `USER_ERROR__$
nconsistent_build_configuration__see_dlib_faq_1_'
/home/xxx/onnxruntime_customops_integration/build/Debug/_deps/dl$b-src/dlib/matrix/../threads/../test_for_odr_violations.h:42: undefined reference to `USER_ERROR__$nconsistent_build_configuration__see_dlib_faq_2'
collect2: error: ld returned 1 exit status
CMakeFiles/onnxruntime_perf_test.dir/build.make:214: recipe for target 'onnxruntime_perf_test' fai$ed
make[2]: *** [onnxruntime_perf_test] Error 1
CMakeFiles/Makefile2:448: recipe for target 'CMakeFiles/onnxruntime_perf_test.dir/all' failed
make[1]: *** [CMakeFiles/onnxruntime_perf_test.dir/all] Error 2
Makefile:181: recipe for target 'all' failed
make: *** [all] Error 2

toolchain versions:
g++ : 8.4.0
gcc: 8.4.0
cmake: 3.19.1
system: ubuntu 18.04.

In addition, I set OCOS_ENABLE_CTEST true in CMakeLists.txt, but the failure log says "The ctest needs the prebuilt onnxruntime libraries directory, please specify it by ONNXRUNTIME_LIB_DIR".

How to set ONNXRUNTIME_LIB_DIR? thanks.

https://github.com/microsoft/onnxruntime-extensions/blob/main/tutorials/gpt2_e2e.py

gpt2_core_model_path = './gpt2.onnx'
gpt2_encoder_model_path = './gpt2_tok.onnx'
gpt2_decoder_model_path = './gpt2_dec.onnx'
gpt2_full_model_path = './gpt2_full.onnx'

I can't judge exactly the roles of the onnx files above in this link. Therefore, I would like to ask you a question as follows.

1. What is the difference between core_model (gpt2.onnx) and full_model (gpt2_full.onnx)?
   I wonder what model plays the role of converting input_tensor into output_tensor(input_tensor -> output_tensor)

2. I am going to do gpt2 text generation using C/C++ API. It seems that encoder_model (gpt2_tok.onnx) converts text into input_tensor(text -> input_tensor) and decoder_model (gpt2_dec.onnx) converts output_tensor into text(output_tensor -> text). If I call 3 onnx files(tok.onnx, dec.onnx, and the onnx file which converts input_tensor into output_tensor) and then do ORT_SESSION_RUN, is it right that I can get the results I want?

3. gpt2_bs.Py seems to contain text_generation_code using the beam search algorithm. What is the exact role difference between gpt2_e2e.py and gpt2_bs.py?

I`m novice of this area. It would be very helpful to me if you explained above questions in detail.

Thank you.

Hi, thanks for your great work.

In the https://github.com/microsoft/ort-customops/blob/main/test/test_torch_ops.py, the custom onnxruntime python operator with single input and output works well.

But if I twist the operator into the one with multiple inputs and outputs, it doesn't work.

Thus, How can I enable multiple inputs and outputs CustomPyOp effectual ?

thanks

unable to merge custom op name: placeholder in onnx, without custom op registration how to merge a custom op with 2nd graph

1. can I ask why we need the onnxruntime-extensions to run this python file? If we only want to run gpt2_full.onnx without the tokenizer part, does it still need onnxruntime-extensions to build into onnxruntime (triton backend)? @wenbingl

2. Can it early stop at eos_token prediction?

3. This is so exciting for gpt2 auto-regressive decoding, which is the unique inference optimized implementation with fusion, past state and int8 quantization for CPU now. Is it possible we have the benchmark on CPU or even GPU documented? (For GPU, it can compare with fastertransformer and lightseq)

Describe the bug
Cann install CustomOps library. Unittests result in FAIL : Failed to get symbol RegisterCustomOps with error: /home/oborchers/ext_packages/ort-customops/out/Linux/onnxruntime_customops/_ortcustomops.so: undefined symbol: RegisterCustomOps

Urgency
None

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04.05
ONNX version: '1.8.0'
ONNX Runtime installed from (source or binary): binary
ONNX Runtime version: '1.6.0'
Python version:Python 3.8.5
GCC/Compiler version (if compiling from source): gcc 9.3.0 / cmake version 3.19.1
CUDA/cuDNN version: 11.1 / cudnn8_8.0.4.30
Tensorrt: cuda11.1-trt7.2.1.6
GPU model and memory: Nvidia V100 SXM2 (Driver 455.32.0)

To Reproduce

git clone https://github.com/microsoft/ort-customops
cd ort-customops/
pip install -r requirements.txt
bash ./build.sh # Success
cd out/Linux/
pip install -e . # Success
./ortcustomops_test # Success
cd ../..
pytest test # Fail

Build Output

-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Build type not set - using RelWithDebInfo
-- Using the single-header code from /home/oborchers/ext_packages/ort-customops/out/Linux/_deps/json-src/single_include/
-- Found Python3: /home/oborchers/anaconda3/envs/onnx_cpu/bin/python3.8 (found version "3.8.5") found components: Interpreter Development Development.Module Development.Embed
CMake Deprecation Warning at out/Linux/_deps/googletest-src/CMakeLists.txt:4 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


CMake Deprecation Warning at out/Linux/_deps/googletest-src/googlemock/CMakeLists.txt:45 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


CMake Deprecation Warning at out/Linux/_deps/googletest-src/googletest/CMakeLists.txt:56 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- Found PythonInterp: /home/oborchers/anaconda3/envs/onnx_cpu/bin/python (found version "3.8.5")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Configuring done
-- Generating done
-- Build files have been written to: /home/oborchers/ext_packages/ort-customops/out/Linux
[1%] ... [100%]

pytest test

platform linux -- Python 3.8.5, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/oborchers/ext_packages/ort-customops
plugins: Faker-5.0.1
collected 35 items

test/test_gpt2tok.py F                                                                                                                                                       [  2%]
test/test_math_ops.py FF                                                                                                                                                     [  8%]
test/test_pyops.py FFFFFF                                                                                                                                                    [ 25%]
test/test_string_ops.py .FFFFFFFFFFFFFFFFFFFFFFFF                                                                                                                            [ 97%]
test/test_torch_ops.py F                                                                                                                                                     [100%]

===================================================================================== FAILURES =====================================================================================
_________________________________________________________________________ TestGPT2Tokenizer.test_tokenizer _________________________________________________________________________

self = <test_gpt2tok.TestGPT2Tokenizer testMethod=test_tokenizer>

    def test_tokenizer(self):
        test_sentence = "I can feel the magic, can you?"
        tokenizer = TestGPT2Tokenizer.tokenizer
        indexed_tokens = tokenizer.encode(test_sentence)

        model = _create_test_model()
        binded_model = _bind_tokenizer(model)

        so = _ort.SessionOptions()
>       so.register_custom_ops_library(_get_library_path())
E       onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Failed to get symbol RegisterCustomOps with error: /home/oborchers/ext_packages/ort-customops/out/Linux/onnxruntime_customops/_ortcustomops.so: undefined symbol: RegisterCustomOps

And all other tests fail as well

The share library (ortcustomops.dll) for non-Python

Currently ortcustomops.dll exports 2 functions

• RegisterCustomOps
• AddExternalCustomOp

Feedback

Do consider more documentation on the how to use the exported functions.

Questions

• Are these restricted to ONNX or reusable in other .NET interop scenario?

• Are the methods described in The pre/post processing code to ONNX model available through (exported functions) of ortcustomops.dll or only through python? If so beyond python, do consider more documentation e.g. for .NET interop scenarios

• Where the ortcustomops.dll unit tests?

There is a unit test to help verify the build.

I tried customize a 2-input operator add2, the operator was exported from a .cpp file. Then I performed the code below to register the operator in onnx, export a small model based on the operator, register the operator in onnxruntime. Add run the model.
Software Configuration:
torch: 1.9.1+cu111
onnx: 1.10.1
onnxruntime: 1.9.0
onnxruntime_extensions: 0.4.2

import onnx
import numpy as np
import io
import torch
print(onnx.__version__)
shared_library = "add2/build/libadd2.so"
torch.ops.load_library(shared_library)
class pool(torch.nn.Module):
    def __init__(self) -> None:
        super().__init__()
    def forward(self, x1, x2):
        return torch.ops.my_ops.add2(x1, x2)

def to_numpy(tensor):
    return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()   


def symbolic_add2(g, x1, x2):
    return g.op("ai.onnx.contrib::add2", x1, x2)
torch.onnx.register_custom_op_symbolic('my_ops::add2', symbolic_add2, 1)


# export
m = pool()
x1 = torch.randn([1, 1, 2, 2])
x2 = torch.rand([1, 1, 2, 2])
m.eval()
f = io.BytesIO()
output = m(x1, x2)
torch.onnx.export(
    m, (x1, x2), f,
    opset_version=12)
onnx_model = onnx.load(io.BytesIO(f.getvalue()))
print(output)

# import and run
from onnxruntime_extensions import onnx_op, PyOp, PyOrtFunction
@onnx_op(op_type='add2')
def add2(x1, x2):
    return torch.ops.my_ops.add2(x1, x2)

onnx_fn = PyOrtFunction.from_model(onnx_model)
out_ort = onnx_fn(x1.numpy(), x2.numpy())

np.testing.assert_allclose(to_numpy(output), out_ort[0], rtol=1e-03, atol=1e-05)
pass

But I got in from "out_ort = onnx_fn([x1.numpy(), x2.numpy()])":
[ONNXRuntimeError] : 10 : INVALID_GRAPH : This is an invalid model. Error in Node:add2_0 : Node (add2_0) has input size 2 not in range [min=1, max=1]
It doesn't work to make a list or tuple of inputs (i.e.[x1, x2]). How can I run a multiple input model with custom operator? Any suggestion?

I tried to replicate fastT5 for a different transformers model (EncoderDecoderModel) but I'm always facing this error:

onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running MatMul node. Name:'MatMul_155' Status Message: /onnxruntime_src/onnxruntime/core/framework/op_kernel.cc:44 OrtValue* onnxruntime::OpKernelContext::OutputMLValue(int, const onnxruntime::TensorShape&, size_t) status.IsOK() was false. Shape mismatch attempting to re-use buffer. {1,1,768} != {1,133,768}. Validate usage of dim_value (values should be > 0) and dim_param (all values with the same string should equate to the same size) in shapes in the model.

environment:
huggingface-hub 0.0.8
importlib-metadata 4.10.0
numpy 1.21.5
onnx 1.10.2
onnxruntime 1.4.0
packaging 21.3
pip 19.2.3
progress 1.6
protobuf 3.19.1
psutil 5.8.0
pyparsing 3.0.6
regex 2021.11.10
requests 2.26.0
sacremoses 0.0.46
sentencepiece 0.1.96
setuptools 41.2.0
six 1.16.0
tokenizers 0.10.3
torch 1.10.1
tqdm 4.62.3
transformers 4.4.2

Anyone having a similar error? Any idea on how to solve it?
Thanks

I wonder if there is version dependency between onnxruntime-extensions, onnx and onnxruntime?
If so, could you offer the version mapping?

when i run python setup.py install
it occurs:
此时不应有 \Microsoft。
D:\python3.6\lib\distutils\dist.py:261: UserWarning: Unknown distribution option: 'long_description_content_type'
warnings.warn(msg)
running install
running bdist_egg
running egg_info
writing onnxruntime_extensions.egg-info\PKG-INFO
writing dependency_links to onnxruntime_extensions.egg-info\dependency_links.txt
writing requirements to onnxruntime_extensions.egg-info\requires.txt
writing top-level names to onnxruntime_extensions.egg-info\top_level.txt
reading manifest file 'onnxruntime_extensions.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'ci_build'
no previously-included directories found matching 'docs'
warning: no previously-included files found matching '.bat'
warning: no previously-included files found matching '.yaml'
warning: no previously-included files found matching '.git'
writing manifest file 'onnxruntime_extensions.egg-info\SOURCES.txt'
installing library code to .setuptools-cmake-build\bdist.win-amd64\egg
running install_lib
running build_py
running build_ext
C:\Program Files\CMake\bin\cmake.exe D:\code\onnx\onnxruntime-extensions-main\onnxruntime-extensions-main -DCMAKE_LIBRARY_OUTPUT_DIRECTORY=D:\code\onnx\onnxruntime-extensions-main\onnxruntime-extensions-main.setuptools-cmake-build\lib.win-amd64-3.6\onnxruntime_extensions -DOCOS_ENABLE_PYTHON=ON -DOCOS_ENABLE_CTEST=OFF -DOCOS_EXTENTION_NAME=_ortcustomops.cp36-win_amd64.pyd -DCMAKE_BUILD_TYPE=Release
-- Building for: Visual Studio 16 2019
-- The C compiler identification is MSVC 19.29.30038.1
-- The CXX compiler identification is MSVC 19.29.30038.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30037/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30037/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Fetch googlere2
CMake Error at C:/Program Files/CMake/share/cmake-3.21/Modules/ExternalProject.cmake:2650 (message):
error: could not find git for clone of googlere2-populate
Call Stack (most recent call first):
C:/Program Files/CMake/share/cmake-3.21/Modules/ExternalProject.cmake:3700 (_ep_add_download_command)
CMakeLists.txt:22 (ExternalProject_Add)

-- Configuring incomplete, errors occurred!
See also "D:/code/onnx/onnxruntime-extensions-main/onnxruntime-extensions-main/.setuptools-cmake-build/temp.win-amd64-3.6/Release/_deps/googlere2-subbuild/CMakeFiles/CMakeOutput.log".

CMake Error at C:/Program Files/CMake/share/cmake-3.21/Modules/FetchContent.cmake:1005 (message):
CMake step for googlere2 failed: 1
Call Stack (most recent call first):
C:/Program Files/CMake/share/cmake-3.21/Modules/FetchContent.cmake:1146:EVAL:2 (__FetchContent_directPopulate)
C:/Program Files/CMake/share/cmake-3.21/Modules/FetchContent.cmake:1146 (cmake_language)
cmake/externals/googlere2.cmake:10 (FetchContent_Populate)
CMakeLists.txt:67 (include)

-- Configuring incomplete, errors occurred!
See also "D:/code/onnx/onnxruntime-extensions-main/onnxruntime-extensions-main/.setuptools-cmake-build/temp.win-amd64-3.6/Release/CMakeFiles/CMakeOutput.log".
error: command 'C:\Program Files\CMake\bin\cmake.exe' failed with exit status 1

Actual: For the attached valid model using operators from onnxruntime-extensions, the onnxruntime crashes when the input_text parameter is an empty string or just whitespace.

Expected: For the attached valid model using operators from onnxruntime-extensions, the onnxruntime should not crash when the input_text parameter is an empty string or just whitespace.

onnx_prefilter_model.extended.optimized.onnx.zip

I converted tensorflow's pre-processing model for BERT i.e. bert_en_uncased_preprocess v3 (https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3) into ONNX model using tf2onnx command -

python -m tf2onnx.convert --saved-model bert_en_uncased_preprocess_3 --output model.onnx --opset 13 --extra_opset ai.onnx.contrib:1 --custom-ops CaseFoldUTF8:ai.onnx.contrib,NormalizeUTF8:ai.onnx.contrib

After conversion, I defined the CaseFoldUTF8 and NormalizeUTF8 ops in python as -

import numpy as np
from onnxruntime_extensions import onnx_op, PyCustomOpDef
import string

@onnx_op(op_type="CaseFoldUTF8",
         inputs=[PyCustomOpDef.dt_string],
         outputs=[PyCustomOpDef.dt_string])
def casefold(x):            # convert to lowercase
    temp = []
    temp.append(x[0].lower()) 
    return np.array(temp)

@onnx_op(op_type="NormalizeUTF8",
         inputs=[PyCustomOpDef.dt_string],
         outputs=[PyCustomOpDef.dt_string])
def normalize(x):          # remove punctuations
    temp = []
    temp.append(x[0].translate(str.maketrans('', '', string.punctuation))) 
    return np.array(temp)

After this, I inferred the ONNX model using the below code -

import onnxruntime as ort
from onnxruntime_extensions import get_library_path

so = ort.SessionOptions()
so.register_custom_ops_library(get_library_path())

sess = ort.InferenceSession("model.onnx",so)
input_name = [node.name for node in sess.get_inputs()]
output_name = [node.name for node in sess.get_outputs()]

tensor1 = np.array(['The movie was great!'])
result = sess.run(output_name, {input_name[0]: tensor1})

sess.run used to be successful with onnxruntime-extensions v0.3.1, where as it is now returning the following error with v0.4.0 -

RuntimeException                          Traceback (most recent call last)
<ipython-input-3-f1ba762f7c85> in <module>
      3 
      4 tensor1 = np.array(['The movie was terrible...'])
----> 5 result = sess.run(output_name, {input_name[0]: tensor1})

c:\users\shubham.j\appdata\local\programs\python\python38\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py in run(self, output_names, input_feed, run_options)
    186             output_names = [output.name for output in self._outputs_meta]
    187         try:
--> 188             return self._sess.run(output_names, input_feed, run_options)
    189         except C.EPFail as err:
    190             if self._enable_fallback:

RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running StringRegexSplitWithOffsets node. Name:'StatefulPartitionedCall/model_1/preprocessing/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/bert_tokenizer/StatefulPartitionedCall/RegexSplitWithOffsets' Status Message: offsets buffer is not equal to tensor size

Please let me know how this could be resolved.

outputs = func_one_step(*states[1:])

[E:onnxruntime:, sequential_executor.cc:346
Message=[ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Equal node. Name:'Equal_2554' Status Message: D:\a\_work\1\s\onnxruntime\core\framework\op_kernel.cc:78 onnxruntime::OpKernelContext::OutputMLValue status.IsOK() was false. Shape mismatch attempting to re-use buffer. {1,1} != {1,4}. Validate usage of dim_value (values should be > 0) and dim_param (all values with the same string should equate to the same size) in shapes in the model.

environment

onnx 1.9.0
onnxconverter-common 1.8.1
onnxruntime 1.9.0
onnxruntime-extensions 0.4.2
onnxruntime-gpu 1.9.0
onnxruntime-tools 1.7.0
transformers 4.11.3
Windows 11

Anyone with similar error?

Suggestion what to do next?

I run " python setup.py develop", it doesn't work, would you give me a hand ?

[ 86%] Building CXX object _deps/googlere2-build/CMakeFiles/re2.dir/util/rune.cc.o
[ 86%] Building CXX object _deps/sentencepieceproject-build/src/CMakeFiles/sentencepiece-static.dir//third_party/protobuf-lite/zero_copy_stream_impl_lite.cc.o
[ 86%] Building CXX object CMakeFiles/ocos_operators.dir/operators/tokenizer/sentencepiece_tokenizer.cc.o
[ 86%] Building CXX object _deps/sentencepieceproject-build/src/CMakeFiles/sentencepiece-static.dir//third_party/protobuf-lite/wire_format_lite.cc.o
[ 86%] Building CXX object _deps/sentencepieceproject-build/src/CMakeFiles/sentencepiece-static.dir/util.cc.o
[ 86%] Building CXX object _deps/sentencepieceproject-build/src/CMakeFiles/sentencepiece-static.dir/model_factory.cc.o
[ 86%] Building CXX object _deps/sentencepieceproject-build/src/CMakeFiles/sentencepiece-static.dir//third_party/protobuf-lite/arenastring.cc.o
[ 86%] Building CXX object _deps/sentencepieceproject-build/src/CMakeFiles/sentencepiece-static.dir//third_party/protobuf-lite/time.cc.o
[ 27%] Building CXX object CMakeFiles/ocos_operators.dir/operators/text/string_regex_split.cc.o
[ 86%] Building CXX object _deps/googlere2-build/CMakeFiles/re2.dir/re2/set.cc.o
[ 86%] Building CXX object CMakeFiles/ocos_operators.dir/operators/tokenizer/wordpiece_tokenizer.cc.o
[ 88%] Building CXX object _deps/sentencepieceproject-build/src/CMakeFiles/sentencepiece-static.dir//third_party/absl/flags/flag.cc.o
[ 88%] Building CXX object _deps/sentencepieceproject-build/src/CMakeFiles/sentencepiece-static.dir/builtin_pb/sentencepiece_model.pb.cc.o
[ 90%] Building CXX object _deps/sentencepieceproject-build/src/CMakeFiles/sentencepiece-static.dir/bpe_model.cc.o
[ 90%] Building CXX object _deps/sentencepieceproject-build/src/CMakeFiles/sentencepiece-static.dir//third_party/absl/strings/string_view.cc.o
[ 90%] Building CXX object _deps/sentencepieceproject-build/src/CMakeFiles/sentencepiece-static.dir//third_party/protobuf-lite/implicit_weak_message.cc.o
[ 91%] Building CXX object _deps/sentencepieceproject-build/src/CMakeFiles/sentencepiece-static.dir//third_party/protobuf-lite/strutil.cc.o
/media/bl/data/bags/onnxruntime-extensions-0.3.0/operators/text/vector_to_string.cc:1:20: fatal error: charconv: No such file or directory
compilation terminated.
CMakeFiles/ocos_operators.dir/build.make:327: recipe for target 'CMakeFiles/ocos_operators.dir/operators/text/vector_to_string.cc.o' failed
make[2]: *** [CMakeFiles/ocos_operators.dir/operators/text/vector_to_string.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....
[ 92%] Building CXX object _deps/sentencepieceproject-build/src/CMakeFiles/sentencepiece-static.dir/error.cc.o
/media/bl/data/bags/onnxruntime-extensions-0.3.0/operators/text/string_to_vector.cc:1:20: fatal error: charconv: No such file or directory
compilation terminated.
CMakeFiles/ocos_operators.dir/build.make:299: recipe for target 'CMakeFiles/ocos_operators.dir/operators/text/string_to_vector.cc.o' failed
make[2]: *** [CMakeFiles/ocos_operators.dir/operators/text/string_to_vector.cc.o] Error 1
In file included from /media/bl/data/bags/onnxruntime-extensions-0.3.0/operators/string_utils.cc:2:0:
/media/bl/data/bags/onnxruntime-extensions-0.3.0/operators/string_utils.h:50:13: error: ‘string_view’ is not a member of ‘std’
std::vectorstd::string_view SplitString(const std::string_view& str, const std::string_view& seps, bool remove_empty_entries = false);
^
/media/bl/data/bags/onnxruntime-extensions-0.3.0/operators/string_utils.h:50:13: error: ‘string_view’ is not a member of ‘std’
/media/bl/data/bags/onnxruntime-extensions-0.3.0/operators/string_utils.h:50:29: error: template argument 1 is invalid
std::vectorstd::string_view SplitString(const std::string_view& str, const std::string_view& seps, bool remove_empty_entries = false);
^
/media/bl/data/bags/onnxruntime-extensions-0.3.0/operators/string_utils.h:50:29: error: template argument 2 is invalid
/media/bl/data/bags/onnxruntime-extensions-0.3.0/operators/string_utils.h:50:54: error: ‘string_view’ in namespace ‘std’ does not name a type
std::vectorstd::string_view SplitString(const std::string_view& str, const std::string_view& seps, bool remove_empty_entries = false);
^
/media/bl/data/bags/onnxruntime-extensions-0.3.0/operators/string_utils.h:50:83: error: ‘string_view’ in namespace ‘std’ does not name a type
std::vectorstd::string_view SplitString(const std::string_view& str, const std::string_view& seps, bool remove_empty_entries = false);
^
/media/bl/data/bags/onnxruntime-extensions-0.3.0/operators/string_utils.cc:4:13: error: ‘string_view’ is not a member of ‘std’
std::vectorstd::string_view SplitString(const std::string_view& str, const std::string_view& seps, bool remove_empty_entries) {
^
/media/bl/data/bags/onnxruntime-extensions-0.3.0/operators/string_utils.cc:4:13: error: ‘string_view’ is not a member of ‘std’
/media/bl/data/bags/onnxruntime-extensions-0.3.0/operators/string_utils.cc:4:29: error: template argument 1 is invalid
std::vectorstd::string_view SplitString(const std::string_view& str, const std::string_view& seps, bool remove_empty_entries) {
^