Comments (2)
wenbingl
commented on August 15, 2024
there is no CJK handling in code base, we will take a look.
from onnxruntime-extensions.
Craigacp
commented on August 15, 2024
After installing ftfy I now get a different output from HF's tokenizer which is closer to the CLIPTokenizer output, but it still differs in the last character before the eos token. This happens with both the slow and fast variants. The last character before EOS from HF is ²</w> and the last one from the ONNX op is <|endoftext|> presumably because it hit an unk. According to HF ²</w> has id 366 in openai/clip-vit-base-patch32, but I'm not sure why that's not being picked up by the ONNX implementation.
from onnxruntime-extensions.
Related Issues (20)
- Update of onnuxruntime-extensions in the PyPi repository HOT 2
- "Export" Custom Ops Library After Registering Ops? HOT 2
- Onnxruntime JAR for darwin aarch64 HOT 2
- poor c# documentation, need help on implementing some py examples HOT 1
- Performance issue on basic string operations HOT 3
- DecodeImage: Failed to decode image when using direct ByteBuffer (Android) HOT 2
- Is it possible to create a custom Step for PrePostProcessor pipeline? HOT 3
- Using Whisper for Chinese ASR in iOS may occasionally output illegal UTF-8 strings.
- How to add preprocessing steps for quantized yolov8n (float 16 or int8) in onnx format? HOT 2
- git tag typo?
- ONNX Runtime Extensions fails to load on Linux due to missing libssl.so.1.1 HOT 1
- com.microsoft.onnxruntime:onnxruntime-extensions:0.11.0: missing RegisterCustomOps in onnxruntime_extensions4j_jni.dll HOT 3
- String operations leaking in 0.11.0
- Missing --cpp-gen for cmd.py HOT 1
- [Feature Request] Add dupport to DLPack in the same fashion as ONNX Runtime has support HOT 1
- the result is not right when add yolo8 pre-post process
- how to register custom operator with attributes of a list? HOT 1
- How to register a custom python operation with variadic input/output and homogeneity?
- Question regarding operators
- Support for YOLOv8 Segmentation
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from onnxruntime-extensions.