Comments (4)
mreso
commented on May 28, 2024
Hi @yolk-pie-L I was not able to reproduce this with the 0.9.0 docker and the Error log is inconclusive. We just released 0.10.0, could you retry with the new version?
from serve.
mreso
commented on May 28, 2024
@lxning do you have any idea what could cause the java.lang.InterruptedException: null?
from serve.
yolk-pie-L
commented on May 28, 2024
This is very strange because I can run it with .mar files from other places, but not with the ones I pack myself. I tried using a public .mar file (which can be obtained via gsutil at gs://kfserving-examples/models/torchserve/image_classifier/v1/model-store/mnist.mar), and it runs normally in the image. However, no matter if it's a diffuser or transformer model, the ones I pack myself cannot run. So, I suspect there is a problem with how the model is packaged.
For packing the model, I use another docker image huggingface/transformers-cpu:3.4.0 so as to use lower version of transformer. Because current version of transformer generate .safetensors file and I don't know how to do with it. Therefore, I am using huggingface/transformers-cpu:3.4.0 for executing commands python Download_Transformer_models.py and torch-model-archiver --model-name BERTSeqClassification --version 1.0 --serialized-file Transformer_model/pytorch_model.bin --handler ./Transformer_handler_generalized.py --extra-files "Transformer_model/config.json,./setup_config.json,./Seq_classification_artifacts/index_to_name.json", and using pytorch/torchserve:latest to execute torchserve --start --model-store model_store --models my_tc=BERTSeqClassification.mar --ncs
I install additional python package in the docker image huggingface/transformers-cpu:3.4.0 to help packing the model.
# pip list
Package Version
---------------------- ----------
absl-py 0.10.0
argon2-cffi 20.1.0
asn1crypto 0.24.0
astunparse 1.6.3
async-generator 1.10
attrs 20.2.0
backcall 0.2.0
bleach 3.2.1
cachetools 4.1.1
certifi 2020.6.20
cffi 1.14.3
chardet 3.0.4
click 7.1.2
coloredlogs 15.0.1
cryptography 2.1.4
dataclasses 0.7
decorator 4.4.2
defusedxml 0.6.0
entrypoints 0.3
enum-compat 0.0.3
filelock 3.0.12
future 0.18.2
gast 0.3.3
google-auth 1.21.3
google-auth-oauthlib 0.4.1
google-pasta 0.2.0
grpcio 1.32.0
h5py 2.10.0
humanfriendly 10.0
idna 2.6
importlib-metadata 2.0.0
ipykernel 5.3.4
ipython 7.16.1
ipython-genutils 0.2.0
ipywidgets 7.5.1
jedi 0.17.2
Jinja2 2.11.2
joblib 0.17.0
jsonschema 3.2.0
jupyter 1.0.0
jupyter-client 6.1.7
jupyter-console 6.2.0
jupyter-core 4.6.3
jupyterlab-pygments 0.1.1
Keras-Preprocessing 1.1.2
keyring 10.6.0
keyrings.alt 3.0
Markdown 3.2.2
MarkupSafe 1.1.1
mistune 0.8.4
mpmath 1.3.0
nbclient 0.5.0
nbconvert 6.0.6
nbformat 5.0.7
nest-asyncio 1.4.1
notebook 6.1.4
numpy 1.18.5
oauthlib 3.1.0
opt-einsum 3.3.0
optimum 1.1.1
packaging 20.4
pandocfilters 1.4.2
parso 0.7.1
pexpect 4.8.0
pickleshare 0.7.5
Pillow 8.4.0
pip 20.2.3
prometheus-client 0.8.0
prompt-toolkit 3.0.7
protobuf 3.13.0
psutil 5.9.8
ptyprocess 0.6.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycparser 2.20
pycrypto 2.6.1
Pygments 2.7.1
pygobject 3.26.1
pyparsing 2.4.7
pyrsistent 0.17.3
python-dateutil 2.8.1
pyxdg 0.25
pyzmq 19.0.2
qtconsole 4.7.7
QtPy 1.9.0
regex 2020.10.15
requests 2.24.0
requests-oauthlib 1.3.0
rsa 4.6
sacremoses 0.0.43
SecretStorage 2.3.1
Send2Trash 1.5.0
sentencepiece 0.1.91
setuptools 50.3.0
six 1.15.0
sympy 1.9
tensorboard 2.3.0
tensorboard-plugin-wit 1.7.0
tensorflow-cpu 2.3.1
tensorflow-estimator 2.3.0
termcolor 1.1.0
terminado 0.9.1
testpath 0.4.4
tokenizers 0.9.2
torch 1.10.2
torch-model-archiver 0.9.0
torchserve 0.9.0
tornado 6.0.4
tqdm 4.50.2
traitlets 4.3.3
transformers 3.4.0
typing-extensions 4.1.1
urllib3 1.25.10
wcwidth 0.2.5
webencodings 0.5.1
Werkzeug 1.0.1
wheel 0.30.0
widgetsnbextension 3.5.1
wrapt 1.12.1
zipp 3.2.0
My model is shared here https://github.com/yolk-pie-L/TorchServeModels. Could you help look at it?
Thank you so much!
from serve.
lxning
commented on May 28, 2024
@yolk-pie-L can you please try the following steps?
- use TorchServe CPU docker: https://hub.docker.com/r/pytorch/torchserve/tags
- add HF transformers version in requirements.txt.. For example:
transformers==4.28.1
- create model artifacts
torch-model-archiver --model-name BERTSeqClassification --version 1.0 --serialized-file Transformer_model/pytorch_model.bin --handler ./Transformer_handler_generalized.py --extra-files "Transformer_model/config.json,./setup_config.json,./Seq_classification_artifacts/index_to_name.json" -r requirements.txt
- start TorchServe
from serve.
Related Issues (20)
- PyTorch Serve: Custom handler not saving inference results HOT 6
- PyTorch serve with custom translators returning Unknown exception HOT 4
- Newer Base Image KServe Container fails with exec /usr/local/bin/dockerd-entrypoint.sh: exec format error HOT 7
- Reduce or remove worker retries for specific failures HOT 6
- Update describe model management API to show all model configuration options
- support install cpp dependency in auto-benchmark
- Custom class handler missing BaseHandler
- Kserve nightly CPU tests failure HOT 1
- gRPC Model Metadata using Open Inference Protocol HOT 1
- `\ready` API for Kubernetes probe to know when TorchServe backend is ready to receive traffic HOT 1
- CPU Launcher fails available check with venv HOT 3
- Can torchserve return image data? HOT 1
- Nightly upload to pypi failed HOT 1
- Conda nightlies not getting uploaded for the last 6 days HOT 1
- Building frontend from source in docker
- v0.10.0 OpenAPI specification does not validate
- improve security doc for model security check
- [RFC]: Deprecation notice for TorchText installation in TorchServe
- Torchserve error for more than 14 python async requests on workflow HOT 2
- TorchServe linux aarch64 plan
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from serve.