Comments (7)
namannandan
commented on May 24, 2024
1
@hungtrieu07 To help improve throughput, consider using batching. Here's an example which shows how to batch frames for near real-time video inference: https://github.com/pytorch/serve/blob/master/examples/image_classifier/near_real_time_video/README.md
from serve.
agunapal
commented on May 24, 2024
1
@hungtrieu07 You can refer to this example.
https://github.com/pytorch/serve/blob/master/test/pytest/test_onnx.py#L49
You can inherit the base handler initialize method.
from serve.
hungtrieu07
commented on May 24, 2024
@hungtrieu07 To help improve throughput, consider using batching. Here's an example which shows how to batch frames for near real-time video inference: https://github.com/pytorch/serve/blob/master/examples/image_classifier/near_real_time_video/README.md
Thank you, this is a good idea, I will work on it now.
from serve.
hungtrieu07
commented on May 24, 2024
Hi @namannandan , thanks for your information. Now I'm working on my model (YOLOv8), but they are in ONNX format with dynamic batching. Do we have any example about this?
This is my handler:
import base64
import onnxruntime
import torch
from torchvision.transforms.functional import to_tensor
from ts.torch_handler.base_handler import BaseHandler
from PIL import Image
import io
from ultils_v8 import *
class FaceDetection(BaseHandler):
def initialize(self, context):
self._context = context
self.initialized = False
# Load the ONNX model
model_dir = context.system_properties.get("model_dir")
model_file_path = model_dir + "/yolov8n-face-batch.onnx" # Adjust 'model.onnx' to your model's filename
self.session = onnxruntime.InferenceSession(model_file_path)
self.initialized = True
def preprocess(self, data):
images = []
for row in data:
image = row.get("data") or row.get("body")
if isinstance(image, str):
# if the image is a string of bytesarray.
image = base64.b64decode(image)
# If the image is sent as bytesarray
if isinstance(image, (bytearray, bytes)):
image = Image.open(io.BytesIO(image))
else:
print(image.shape)
# if the image is a list
image = torch.FloatTensor(image)
image = image.resize((640, 640))
image = to_tensor(image)
images.append(image)
return torch.stack(images) # Create a batch
def inference(self, batch_data):
# ONNX runtime expects numpy arrays
batch_data = batch_data.numpy().astype(np.float32)
inputs = {self.session.get_inputs()[0].name: batch_data}
outputs = self.session.run(None, inputs)
return outputs
def postprocess(self, inference_output):
# just processing the result, not done yet
results = []
for batch in inference_output:
# print(batch)
# print(batch.shape)
bbox_coords, confidences = batch[:4], batch[4]
print(bbox_coords)
print(confidences)
return results
from serve.
lxning
commented on May 24, 2024
@hungtrieu07 onnx is auto-detected by basehandler. (see code). You only need make sure onnx is installed.
from serve.
hungtrieu07
commented on May 24, 2024
Hi @lxning, I'm using BaseHandler as you tell above, but I saw this error when run inference an image:
AttributeError: 'NoneType' object has no attribute 'model_yaml_config'
My model are converted to ONNX format, and using BaseHandler class as handler file.
This is my full ts_log.log:
2024-04-09T10:34:16,954 [WARN ] main org.pytorch.serve.util.ConfigManager - Your torchserve instance can access any URL to load models. When deploying to production, make sure to limit the set of allowed_urls in config.properties
2024-04-09T10:34:16,954 [WARN ] main org.pytorch.serve.util.ConfigManager - Your torchserve instance can access any URL to load models. When deploying to production, make sure to limit the set of allowed_urls in config.properties
2024-04-09T10:34:16,958 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager...
2024-04-09T10:34:16,958 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager...
2024-04-09T10:34:17,041 [INFO ] main org.pytorch.serve.metrics.configuration.MetricConfiguration - Successfully loaded metrics configuration from /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml
2024-04-09T10:34:17,041 [INFO ] main org.pytorch.serve.metrics.configuration.MetricConfiguration - Successfully loaded metrics configuration from /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml
2024-04-09T10:34:17,213 [INFO ] main org.pytorch.serve.ModelServer -
Torchserve version: 0.9.0
TS Home: /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages
Current directory: /mnt/d/dev/api_torchserve_lancs
Temp directory: /tmp
Metrics config path: /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml
Number of GPUs: 0
Number of CPUs: 8
Max heap size: 1970 M
Python executable: /home/hungtrieu07/miniconda3/envs/api_lanc/bin/python
Config file: config.properties
Inference address: http://0.0.0.0:8080
Management address: http://0.0.0.0:8081
Metrics address: http://0.0.0.0:8082
Model Store: /mnt/d/dev/api_torchserve_lancs/model-store
Initial Models: all
Log dir: /mnt/d/dev/api_torchserve_lancs/logs
Metrics dir: /mnt/d/dev/api_torchserve_lancs/logs
Netty threads: 0
Netty client threads: 0
Default workers per model: 8
Blacklist Regex: N/A
Maximum Response Size: 6553500
Maximum Request Size: 6553500
Limit Maximum Image Pixels: true
Prefer direct buffer: false
Allowed Urls: [file://.*|http(s)?://.*]
Custom python dependency for model allowed: true
Enable metrics API: true
Metrics mode: log
Disable system metrics: false
Workflow Store: /mnt/d/dev/api_torchserve_lancs/model-store
Model config: {"FaceDetection": {"1.0": {"defaultVersion": true,"marName": "FaceDetection.mar","minWorkers": 1,"maxWorkers": 1,"batchSize": 256,"maxBatchDelay": 100,"responseTimeout": 120}},"FaceExpression": {"1.0": {"defaultVersion": true,"marName": "FaceExpression.mar","minWorkers": 1,"maxWorkers": 4,"batchSize": 256,"maxBatchDelay": 100,"responseTimeout": 120}},"FaceRecognition": {"1.0": {"defaultVersion": true,"marName": "FaceRecognition.mar","minWorkers": 1,"maxWorkers": 4,"batchSize": 256,"maxBatchDelay": 100,"responseTimeout": 120}},"HumanPose": {"1.0": {"defaultVersion": true,"marName": "HumanPose.mar","minWorkers": 1,"maxWorkers": 4,"batchSize": 256,"maxBatchDelay": 100,"responseTimeout": 120}},"ActionRecognition": {"1.0": {"defaultVersion": true,"marName": "ActionRecognition.mar","minWorkers": 1,"maxWorkers": 4,"batchSize": 256,"maxBatchDelay": 100,"responseTimeout": 120}}}
2024-04-09T10:34:17,213 [INFO ] main org.pytorch.serve.ModelServer -
Torchserve version: 0.9.0
TS Home: /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages
Current directory: /mnt/d/dev/api_torchserve_lancs
Temp directory: /tmp
Metrics config path: /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml
Number of GPUs: 0
Number of CPUs: 8
Max heap size: 1970 M
Python executable: /home/hungtrieu07/miniconda3/envs/api_lanc/bin/python
Config file: config.properties
Inference address: http://0.0.0.0:8080
Management address: http://0.0.0.0:8081
Metrics address: http://0.0.0.0:8082
Model Store: /mnt/d/dev/api_torchserve_lancs/model-store
Initial Models: all
Log dir: /mnt/d/dev/api_torchserve_lancs/logs
Metrics dir: /mnt/d/dev/api_torchserve_lancs/logs
Netty threads: 0
Netty client threads: 0
Default workers per model: 8
Blacklist Regex: N/A
Maximum Response Size: 6553500
Maximum Request Size: 6553500
Limit Maximum Image Pixels: true
Prefer direct buffer: false
Allowed Urls: [file://.*|http(s)?://.*]
Custom python dependency for model allowed: true
Enable metrics API: true
Metrics mode: log
Disable system metrics: false
Workflow Store: /mnt/d/dev/api_torchserve_lancs/model-store
Model config: {"FaceDetection": {"1.0": {"defaultVersion": true,"marName": "FaceDetection.mar","minWorkers": 1,"maxWorkers": 1,"batchSize": 256,"maxBatchDelay": 100,"responseTimeout": 120}},"FaceExpression": {"1.0": {"defaultVersion": true,"marName": "FaceExpression.mar","minWorkers": 1,"maxWorkers": 4,"batchSize": 256,"maxBatchDelay": 100,"responseTimeout": 120}},"FaceRecognition": {"1.0": {"defaultVersion": true,"marName": "FaceRecognition.mar","minWorkers": 1,"maxWorkers": 4,"batchSize": 256,"maxBatchDelay": 100,"responseTimeout": 120}},"HumanPose": {"1.0": {"defaultVersion": true,"marName": "HumanPose.mar","minWorkers": 1,"maxWorkers": 4,"batchSize": 256,"maxBatchDelay": 100,"responseTimeout": 120}},"ActionRecognition": {"1.0": {"defaultVersion": true,"marName": "ActionRecognition.mar","minWorkers": 1,"maxWorkers": 4,"batchSize": 256,"maxBatchDelay": 100,"responseTimeout": 120}}}
2024-04-09T10:34:17,227 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Loading snapshot serializer plugin...
2024-04-09T10:34:17,227 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Loading snapshot serializer plugin...
2024-04-09T10:34:17,238 [DEBUG] main org.pytorch.serve.ModelServer - Loading models from model store: ActionRecognition.mar
2024-04-09T10:34:17,238 [DEBUG] main org.pytorch.serve.ModelServer - Loading models from model store: ActionRecognition.mar
2024-04-09T10:34:17,483 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model ActionRecognition
2024-04-09T10:34:17,483 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model ActionRecognition
2024-04-09T10:34:17,484 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model ActionRecognition
2024-04-09T10:34:17,484 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model ActionRecognition
2024-04-09T10:34:17,484 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model ActionRecognition loaded.
2024-04-09T10:34:17,484 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model ActionRecognition loaded.
2024-04-09T10:34:17,485 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: ActionRecognition, count: 1
2024-04-09T10:34:17,485 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: ActionRecognition, count: 1
2024-04-09T10:34:17,499 [DEBUG] W-9000-ActionRecognition_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:34:17,499 [DEBUG] main org.pytorch.serve.ModelServer - Loading models from model store: FaceDetection.mar
2024-04-09T10:34:17,499 [DEBUG] W-9000-ActionRecognition_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:34:17,499 [DEBUG] main org.pytorch.serve.ModelServer - Loading models from model store: FaceDetection.mar
2024-04-09T10:34:19,198 [INFO ] W-9000-ActionRecognition_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=91650
2024-04-09T10:34:19,200 [INFO ] W-9000-ActionRecognition_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000
2024-04-09T10:34:19,206 [INFO ] W-9000-ActionRecognition_1.0-stdout MODEL_LOG - Successfully loaded /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2024-04-09T10:34:19,207 [INFO ] W-9000-ActionRecognition_1.0-stdout MODEL_LOG - [PID]91650
2024-04-09T10:34:19,208 [INFO ] W-9000-ActionRecognition_1.0-stdout MODEL_LOG - Torch worker started.
2024-04-09T10:34:19,209 [DEBUG] W-9000-ActionRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-ActionRecognition_1.0 State change null -> WORKER_STARTED
2024-04-09T10:34:19,209 [DEBUG] W-9000-ActionRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-ActionRecognition_1.0 State change null -> WORKER_STARTED
2024-04-09T10:34:19,209 [INFO ] W-9000-ActionRecognition_1.0-stdout MODEL_LOG - Python runtime: 3.8.19
2024-04-09T10:34:19,230 [INFO ] W-9000-ActionRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2024-04-09T10:34:19,230 [INFO ] W-9000-ActionRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000
2024-04-09T10:34:19,319 [INFO ] W-9000-ActionRecognition_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000.
2024-04-09T10:34:19,323 [INFO ] W-9000-ActionRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633659323
2024-04-09T10:34:19,323 [INFO ] W-9000-ActionRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633659323
2024-04-09T10:34:19,367 [INFO ] W-9000-ActionRecognition_1.0-stdout MODEL_LOG - model_name: ActionRecognition, batchSize: 256
2024-04-09T10:34:20,012 [INFO ] W-9000-ActionRecognition_1.0-stdout MODEL_LOG - ONNX enabled
2024-04-09T10:34:20,013 [INFO ] W-9000-ActionRecognition_1.0-stdout MODEL_LOG - Torch TensorRT not enabled
2024-04-09T10:34:20,485 [DEBUG] W-9000-ActionRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:34:20,485 [DEBUG] W-9000-ActionRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:34:20,485 [INFO ] W-9000-ActionRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1117
2024-04-09T10:34:20,485 [INFO ] W-9000-ActionRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1117
2024-04-09T10:34:20,486 [DEBUG] W-9000-ActionRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-ActionRecognition_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:34:20,486 [DEBUG] W-9000-ActionRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-ActionRecognition_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:34:20,488 [INFO ] W-9000-ActionRecognition_1.0 TS_METRICS - WorkerLoadTime.Milliseconds:2991.0|#WorkerName:W-9000-ActionRecognition_1.0,Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633660
2024-04-09T10:34:20,489 [INFO ] W-9000-ActionRecognition_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:49.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633660
2024-04-09T10:34:20,717 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model FaceDetection
2024-04-09T10:34:20,717 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model FaceDetection
2024-04-09T10:34:20,718 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model FaceDetection
2024-04-09T10:34:20,718 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model FaceDetection
2024-04-09T10:34:20,719 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model FaceDetection loaded.
2024-04-09T10:34:20,719 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model FaceDetection loaded.
2024-04-09T10:34:20,719 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: FaceDetection, count: 1
2024-04-09T10:34:20,719 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: FaceDetection, count: 1
2024-04-09T10:34:20,721 [DEBUG] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9001, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:34:20,721 [DEBUG] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9001, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:34:20,722 [DEBUG] main org.pytorch.serve.ModelServer - Loading models from model store: FaceExpression.mar
2024-04-09T10:34:20,722 [DEBUG] main org.pytorch.serve.ModelServer - Loading models from model store: FaceExpression.mar
2024-04-09T10:34:22,192 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9001, pid=91696
2024-04-09T10:34:22,194 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9001
2024-04-09T10:34:22,203 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - Successfully loaded /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2024-04-09T10:34:22,204 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - [PID]91696
2024-04-09T10:34:22,204 [DEBUG] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9001-FaceDetection_1.0 State change null -> WORKER_STARTED
2024-04-09T10:34:22,204 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - Torch worker started.
2024-04-09T10:34:22,204 [DEBUG] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9001-FaceDetection_1.0 State change null -> WORKER_STARTED
2024-04-09T10:34:22,205 [INFO ] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9001
2024-04-09T10:34:22,205 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - Python runtime: 3.8.19
2024-04-09T10:34:22,205 [INFO ] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9001
2024-04-09T10:34:22,208 [INFO ] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633662208
2024-04-09T10:34:22,208 [INFO ] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633662208
2024-04-09T10:34:22,208 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9001.
2024-04-09T10:34:22,224 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - model_name: FaceDetection, batchSize: 256
2024-04-09T10:34:22,394 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - ONNX enabled
2024-04-09T10:34:22,394 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - Torch TensorRT not enabled
2024-04-09T10:34:22,474 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - Succesfully setup ort session
2024-04-09T10:34:22,475 [DEBUG] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:34:22,475 [DEBUG] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:34:22,475 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - '/tmp/models/514d97655cd546e9ae0fbdbd23c166bc/index_to_name.json' is missing. Inference output will not include class name.
2024-04-09T10:34:22,476 [INFO ] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 251
2024-04-09T10:34:22,476 [INFO ] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 251
2024-04-09T10:34:22,477 [DEBUG] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9001-FaceDetection_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:34:22,477 [DEBUG] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9001-FaceDetection_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:34:22,477 [INFO ] W-9001-FaceDetection_1.0 TS_METRICS - WorkerLoadTime.Milliseconds:1757.0|#WorkerName:W-9001-FaceDetection_1.0,Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633662
2024-04-09T10:34:22,478 [INFO ] W-9001-FaceDetection_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:19.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633662
2024-04-09T10:34:26,884 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model FaceExpression
2024-04-09T10:34:26,884 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model FaceExpression
2024-04-09T10:34:26,885 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model FaceExpression
2024-04-09T10:34:26,885 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model FaceExpression
2024-04-09T10:34:26,885 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model FaceExpression loaded.
2024-04-09T10:34:26,885 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model FaceExpression loaded.
2024-04-09T10:34:26,886 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: FaceExpression, count: 1
2024-04-09T10:34:26,886 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: FaceExpression, count: 1
2024-04-09T10:34:26,887 [DEBUG] W-9002-FaceExpression_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9002, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:34:26,887 [DEBUG] W-9002-FaceExpression_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9002, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:34:26,888 [DEBUG] main org.pytorch.serve.ModelServer - Loading models from model store: FaceRecognition.mar
2024-04-09T10:34:26,888 [DEBUG] main org.pytorch.serve.ModelServer - Loading models from model store: FaceRecognition.mar
2024-04-09T10:34:28,040 [INFO ] W-9002-FaceExpression_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9002, pid=91765
2024-04-09T10:34:28,041 [INFO ] W-9002-FaceExpression_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9002
2024-04-09T10:34:28,047 [INFO ] W-9002-FaceExpression_1.0-stdout MODEL_LOG - Successfully loaded /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2024-04-09T10:34:28,049 [INFO ] W-9002-FaceExpression_1.0-stdout MODEL_LOG - [PID]91765
2024-04-09T10:34:28,049 [DEBUG] W-9002-FaceExpression_1.0 org.pytorch.serve.wlm.WorkerThread - W-9002-FaceExpression_1.0 State change null -> WORKER_STARTED
2024-04-09T10:34:28,049 [INFO ] W-9002-FaceExpression_1.0-stdout MODEL_LOG - Torch worker started.
2024-04-09T10:34:28,049 [DEBUG] W-9002-FaceExpression_1.0 org.pytorch.serve.wlm.WorkerThread - W-9002-FaceExpression_1.0 State change null -> WORKER_STARTED
2024-04-09T10:34:28,050 [INFO ] W-9002-FaceExpression_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9002
2024-04-09T10:34:28,050 [INFO ] W-9002-FaceExpression_1.0-stdout MODEL_LOG - Python runtime: 3.8.19
2024-04-09T10:34:28,050 [INFO ] W-9002-FaceExpression_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9002
2024-04-09T10:34:28,053 [INFO ] W-9002-FaceExpression_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633668053
2024-04-09T10:34:28,053 [INFO ] W-9002-FaceExpression_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633668053
2024-04-09T10:34:28,053 [INFO ] W-9002-FaceExpression_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9002.
2024-04-09T10:34:28,069 [INFO ] W-9002-FaceExpression_1.0-stdout MODEL_LOG - model_name: FaceExpression, batchSize: 256
2024-04-09T10:34:28,512 [INFO ] W-9002-FaceExpression_1.0-stdout MODEL_LOG - ONNX enabled
2024-04-09T10:34:28,513 [INFO ] W-9002-FaceExpression_1.0-stdout MODEL_LOG - Torch TensorRT not enabled
2024-04-09T10:34:28,623 [DEBUG] W-9002-FaceExpression_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:34:28,623 [DEBUG] W-9002-FaceExpression_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:34:28,624 [INFO ] W-9002-FaceExpression_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 554
2024-04-09T10:34:28,624 [INFO ] W-9002-FaceExpression_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 554
2024-04-09T10:34:28,624 [DEBUG] W-9002-FaceExpression_1.0 org.pytorch.serve.wlm.WorkerThread - W-9002-FaceExpression_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:34:28,624 [DEBUG] W-9002-FaceExpression_1.0 org.pytorch.serve.wlm.WorkerThread - W-9002-FaceExpression_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:34:28,625 [INFO ] W-9002-FaceExpression_1.0 TS_METRICS - WorkerLoadTime.Milliseconds:1739.0|#WorkerName:W-9002-FaceExpression_1.0,Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633668
2024-04-09T10:34:28,626 [INFO ] W-9002-FaceExpression_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:19.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633668
2024-04-09T10:34:56,181 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model FaceRecognition
2024-04-09T10:34:56,181 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model FaceRecognition
2024-04-09T10:34:56,181 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model FaceRecognition
2024-04-09T10:34:56,181 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model FaceRecognition
2024-04-09T10:34:56,182 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model FaceRecognition loaded.
2024-04-09T10:34:56,182 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model FaceRecognition loaded.
2024-04-09T10:34:56,182 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: FaceRecognition, count: 1
2024-04-09T10:34:56,182 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: FaceRecognition, count: 1
2024-04-09T10:34:56,184 [DEBUG] W-9003-FaceRecognition_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9003, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:34:56,184 [DEBUG] W-9003-FaceRecognition_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9003, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:34:56,185 [DEBUG] main org.pytorch.serve.ModelServer - Loading models from model store: HumanPose.mar
2024-04-09T10:34:56,185 [DEBUG] main org.pytorch.serve.ModelServer - Loading models from model store: HumanPose.mar
2024-04-09T10:34:57,432 [INFO ] W-9003-FaceRecognition_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9003, pid=91890
2024-04-09T10:34:57,434 [INFO ] W-9003-FaceRecognition_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9003
2024-04-09T10:34:57,443 [INFO ] W-9003-FaceRecognition_1.0-stdout MODEL_LOG - Successfully loaded /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2024-04-09T10:34:57,446 [INFO ] W-9003-FaceRecognition_1.0-stdout MODEL_LOG - [PID]91890
2024-04-09T10:34:57,447 [DEBUG] W-9003-FaceRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - W-9003-FaceRecognition_1.0 State change null -> WORKER_STARTED
2024-04-09T10:34:57,447 [INFO ] W-9003-FaceRecognition_1.0-stdout MODEL_LOG - Torch worker started.
2024-04-09T10:34:57,447 [DEBUG] W-9003-FaceRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - W-9003-FaceRecognition_1.0 State change null -> WORKER_STARTED
2024-04-09T10:34:57,448 [INFO ] W-9003-FaceRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9003
2024-04-09T10:34:57,448 [INFO ] W-9003-FaceRecognition_1.0-stdout MODEL_LOG - Python runtime: 3.8.19
2024-04-09T10:34:57,448 [INFO ] W-9003-FaceRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9003
2024-04-09T10:34:57,450 [INFO ] W-9003-FaceRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633697450
2024-04-09T10:34:57,450 [INFO ] W-9003-FaceRecognition_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9003.
2024-04-09T10:34:57,450 [INFO ] W-9003-FaceRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633697450
2024-04-09T10:34:57,478 [INFO ] W-9003-FaceRecognition_1.0-stdout MODEL_LOG - model_name: FaceRecognition, batchSize: 256
2024-04-09T10:34:57,942 [INFO ] W-9003-FaceRecognition_1.0-stdout MODEL_LOG - ONNX enabled
2024-04-09T10:34:57,942 [INFO ] W-9003-FaceRecognition_1.0-stdout MODEL_LOG - Torch TensorRT not enabled
2024-04-09T10:34:59,763 [DEBUG] W-9003-FaceRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:34:59,763 [DEBUG] W-9003-FaceRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:34:59,764 [INFO ] W-9003-FaceRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 2286
2024-04-09T10:34:59,764 [INFO ] W-9003-FaceRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 2286
2024-04-09T10:34:59,764 [DEBUG] W-9003-FaceRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - W-9003-FaceRecognition_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:34:59,764 [DEBUG] W-9003-FaceRecognition_1.0 org.pytorch.serve.wlm.WorkerThread - W-9003-FaceRecognition_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:34:59,765 [INFO ] W-9003-FaceRecognition_1.0 TS_METRICS - WorkerLoadTime.Milliseconds:3582.0|#WorkerName:W-9003-FaceRecognition_1.0,Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633699
2024-04-09T10:34:59,765 [INFO ] W-9003-FaceRecognition_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:29.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633699
2024-04-09T10:35:05,420 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model HumanPose
2024-04-09T10:35:05,420 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model HumanPose
2024-04-09T10:35:05,421 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model HumanPose
2024-04-09T10:35:05,421 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model HumanPose
2024-04-09T10:35:05,421 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model HumanPose loaded.
2024-04-09T10:35:05,421 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model HumanPose loaded.
2024-04-09T10:35:05,422 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: HumanPose, count: 1
2024-04-09T10:35:05,422 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: HumanPose, count: 1
2024-04-09T10:35:05,423 [DEBUG] W-9004-HumanPose_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9004, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:35:05,423 [DEBUG] W-9004-HumanPose_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9004, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:35:05,424 [DEBUG] main org.pytorch.serve.ModelServer - Loading models from model store: PhoneDetection.mar
2024-04-09T10:35:05,424 [DEBUG] main org.pytorch.serve.ModelServer - Loading models from model store: PhoneDetection.mar
2024-04-09T10:35:07,297 [INFO ] W-9004-HumanPose_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9004, pid=91965
2024-04-09T10:35:07,298 [INFO ] W-9004-HumanPose_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9004
2024-04-09T10:35:07,310 [INFO ] W-9004-HumanPose_1.0-stdout MODEL_LOG - Successfully loaded /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2024-04-09T10:35:07,311 [INFO ] W-9004-HumanPose_1.0-stdout MODEL_LOG - [PID]91965
2024-04-09T10:35:07,311 [DEBUG] W-9004-HumanPose_1.0 org.pytorch.serve.wlm.WorkerThread - W-9004-HumanPose_1.0 State change null -> WORKER_STARTED
2024-04-09T10:35:07,311 [INFO ] W-9004-HumanPose_1.0-stdout MODEL_LOG - Torch worker started.
2024-04-09T10:35:07,311 [DEBUG] W-9004-HumanPose_1.0 org.pytorch.serve.wlm.WorkerThread - W-9004-HumanPose_1.0 State change null -> WORKER_STARTED
2024-04-09T10:35:07,312 [INFO ] W-9004-HumanPose_1.0-stdout MODEL_LOG - Python runtime: 3.8.19
2024-04-09T10:35:07,312 [INFO ] W-9004-HumanPose_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9004
2024-04-09T10:35:07,312 [INFO ] W-9004-HumanPose_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9004
2024-04-09T10:35:07,314 [INFO ] W-9004-HumanPose_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633707314
2024-04-09T10:35:07,314 [INFO ] W-9004-HumanPose_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633707314
2024-04-09T10:35:07,315 [INFO ] W-9004-HumanPose_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9004.
2024-04-09T10:35:07,423 [INFO ] W-9004-HumanPose_1.0-stdout MODEL_LOG - model_name: HumanPose, batchSize: 256
2024-04-09T10:35:08,072 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model PhoneDetection
2024-04-09T10:35:08,072 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model PhoneDetection
2024-04-09T10:35:08,072 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model PhoneDetection
2024-04-09T10:35:08,072 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model PhoneDetection
2024-04-09T10:35:08,073 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model PhoneDetection loaded.
2024-04-09T10:35:08,073 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model PhoneDetection loaded.
2024-04-09T10:35:08,074 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: PhoneDetection, count: 8
2024-04-09T10:35:08,074 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: PhoneDetection, count: 8
2024-04-09T10:35:08,077 [DEBUG] W-9005-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9005, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:35:08,077 [DEBUG] W-9007-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9007, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:35:08,077 [DEBUG] W-9007-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9007, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:35:08,077 [DEBUG] W-9006-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9006, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:35:08,077 [DEBUG] W-9005-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9005, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:35:08,080 [DEBUG] W-9008-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9008, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:35:08,077 [DEBUG] W-9006-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9006, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:35:08,080 [DEBUG] W-9008-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9008, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:35:08,083 [DEBUG] W-9010-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9010, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:35:08,083 [DEBUG] W-9009-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9009, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:35:08,084 [DEBUG] W-9011-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9011, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:35:08,088 [DEBUG] W-9012-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9012, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:35:08,084 [DEBUG] W-9011-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9011, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:35:08,083 [DEBUG] W-9010-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9010, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:35:08,093 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel.
2024-04-09T10:35:08,088 [DEBUG] W-9012-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9012, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:35:08,093 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel.
2024-04-09T10:35:08,083 [DEBUG] W-9009-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/hungtrieu07/miniconda3/envs/api_lanc/bin/python, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9009, --metrics-config, /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml]
2024-04-09T10:35:08,216 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://0.0.0.0:8080
2024-04-09T10:35:08,216 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://0.0.0.0:8080
2024-04-09T10:35:08,217 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel.
2024-04-09T10:35:08,217 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel.
2024-04-09T10:35:08,282 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://0.0.0.0:8081
2024-04-09T10:35:08,282 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://0.0.0.0:8081
2024-04-09T10:35:08,284 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel.
2024-04-09T10:35:08,284 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel.
2024-04-09T10:35:08,290 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://0.0.0.0:8082
2024-04-09T10:35:08,290 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://0.0.0.0:8082
2024-04-09T10:35:09,160 [INFO ] W-9004-HumanPose_1.0-stdout MODEL_LOG - ONNX enabled
2024-04-09T10:35:09,162 [INFO ] W-9004-HumanPose_1.0-stdout MODEL_LOG - Torch TensorRT not enabled
2024-04-09T10:35:09,568 [WARN ] pool-3-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet.
2024-04-09T10:35:09,568 [WARN ] pool-3-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet.
2024-04-09T10:35:09,790 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633709
2024-04-09T10:35:09,793 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:924.9861183166504|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633709
2024-04-09T10:35:09,794 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:30.652191162109375|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633709
2024-04-09T10:35:09,795 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:3.2|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633709
2024-04-09T10:35:09,796 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:3258.49609375|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633709
2024-04-09T10:35:09,797 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:4309.60546875|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633709
2024-04-09T10:35:09,798 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:58.6|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633709
2024-04-09T10:35:10,132 [DEBUG] W-9004-HumanPose_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:35:10,132 [DEBUG] W-9004-HumanPose_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:35:10,135 [INFO ] W-9004-HumanPose_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 2709
2024-04-09T10:35:10,135 [INFO ] W-9004-HumanPose_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 2709
2024-04-09T10:35:10,136 [DEBUG] W-9004-HumanPose_1.0 org.pytorch.serve.wlm.WorkerThread - W-9004-HumanPose_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:35:10,136 [DEBUG] W-9004-HumanPose_1.0 org.pytorch.serve.wlm.WorkerThread - W-9004-HumanPose_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:35:10,137 [INFO ] W-9004-HumanPose_1.0 TS_METRICS - WorkerLoadTime.Milliseconds:4715.0|#WorkerName:W-9004-HumanPose_1.0,Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633710
2024-04-09T10:35:10,138 [INFO ] W-9004-HumanPose_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:115.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633710
2024-04-09T10:35:11,187 [INFO ] W-9007-PhoneDetection_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9007, pid=92001
2024-04-09T10:35:11,188 [INFO ] W-9007-PhoneDetection_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9007
2024-04-09T10:35:11,202 [INFO ] W-9007-PhoneDetection_1.0-stdout MODEL_LOG - Successfully loaded /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2024-04-09T10:35:11,204 [INFO ] W-9007-PhoneDetection_1.0-stdout MODEL_LOG - [PID]92001
2024-04-09T10:35:11,209 [DEBUG] W-9007-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9007-PhoneDetection_1.0 State change null -> WORKER_STARTED
2024-04-09T10:35:11,209 [DEBUG] W-9007-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9007-PhoneDetection_1.0 State change null -> WORKER_STARTED
2024-04-09T10:35:11,213 [INFO ] W-9007-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9007
2024-04-09T10:35:11,213 [INFO ] W-9007-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9007
2024-04-09T10:35:11,210 [INFO ] W-9007-PhoneDetection_1.0-stdout MODEL_LOG - Torch worker started.
2024-04-09T10:35:11,216 [INFO ] W-9007-PhoneDetection_1.0-stdout MODEL_LOG - Python runtime: 3.8.19
2024-04-09T10:35:11,226 [INFO ] W-9007-PhoneDetection_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9007.
2024-04-09T10:35:11,230 [INFO ] W-9007-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633711230
2024-04-09T10:35:11,230 [INFO ] W-9007-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633711230
2024-04-09T10:35:11,295 [INFO ] W-9007-PhoneDetection_1.0-stdout MODEL_LOG - model_name: PhoneDetection, batchSize: 1
2024-04-09T10:35:11,305 [INFO ] W-9010-PhoneDetection_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9010, pid=92008
2024-04-09T10:35:11,308 [INFO ] W-9010-PhoneDetection_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9010
2024-04-09T10:35:11,326 [INFO ] W-9010-PhoneDetection_1.0-stdout MODEL_LOG - Successfully loaded /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2024-04-09T10:35:11,328 [INFO ] W-9010-PhoneDetection_1.0-stdout MODEL_LOG - [PID]92008
2024-04-09T10:35:11,329 [DEBUG] W-9010-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9010-PhoneDetection_1.0 State change null -> WORKER_STARTED
2024-04-09T10:35:11,329 [DEBUG] W-9010-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9010-PhoneDetection_1.0 State change null -> WORKER_STARTED
2024-04-09T10:35:11,329 [INFO ] W-9010-PhoneDetection_1.0-stdout MODEL_LOG - Torch worker started.
2024-04-09T10:35:11,336 [INFO ] W-9011-PhoneDetection_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9011, pid=92007
2024-04-09T10:35:11,340 [INFO ] W-9010-PhoneDetection_1.0-stdout MODEL_LOG - Python runtime: 3.8.19
2024-04-09T10:35:11,332 [INFO ] W-9010-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9010
2024-04-09T10:35:11,332 [INFO ] W-9010-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9010
2024-04-09T10:35:11,343 [INFO ] W-9011-PhoneDetection_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9011
2024-04-09T10:35:11,361 [INFO ] W-9011-PhoneDetection_1.0-stdout MODEL_LOG - Successfully loaded /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2024-04-09T10:35:11,366 [INFO ] W-9010-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633711366
2024-04-09T10:35:11,366 [INFO ] W-9010-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633711366
2024-04-09T10:35:11,364 [INFO ] W-9011-PhoneDetection_1.0-stdout MODEL_LOG - [PID]92007
2024-04-09T10:35:11,373 [DEBUG] W-9011-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9011-PhoneDetection_1.0 State change null -> WORKER_STARTED
2024-04-09T10:35:11,373 [DEBUG] W-9011-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9011-PhoneDetection_1.0 State change null -> WORKER_STARTED
2024-04-09T10:35:11,367 [INFO ] W-9010-PhoneDetection_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9010.
2024-04-09T10:35:11,385 [INFO ] W-9009-PhoneDetection_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9009, pid=92010
2024-04-09T10:35:11,373 [INFO ] W-9011-PhoneDetection_1.0-stdout MODEL_LOG - Torch worker started.
2024-04-09T10:35:11,397 [INFO ] W-9009-PhoneDetection_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9009
2024-04-09T10:35:11,397 [INFO ] W-9011-PhoneDetection_1.0-stdout MODEL_LOG - Python runtime: 3.8.19
2024-04-09T10:35:11,386 [INFO ] W-9011-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9011
2024-04-09T10:35:11,386 [INFO ] W-9011-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9011
2024-04-09T10:35:11,406 [INFO ] W-9009-PhoneDetection_1.0-stdout MODEL_LOG - Successfully loaded /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2024-04-09T10:35:11,420 [INFO ] W-9009-PhoneDetection_1.0-stdout MODEL_LOG - [PID]92010
2024-04-09T10:35:11,425 [INFO ] W-9011-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633711424
2024-04-09T10:35:11,425 [INFO ] W-9011-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633711424
2024-04-09T10:35:11,420 [INFO ] W-9012-PhoneDetection_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9012, pid=92009
2024-04-09T10:35:11,421 [INFO ] W-9010-PhoneDetection_1.0-stdout MODEL_LOG - model_name: PhoneDetection, batchSize: 1
2024-04-09T10:35:11,422 [INFO ] W-9011-PhoneDetection_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9011.
2024-04-09T10:35:11,428 [DEBUG] W-9009-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9009-PhoneDetection_1.0 State change null -> WORKER_STARTED
2024-04-09T10:35:11,434 [INFO ] W-9012-PhoneDetection_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9012
2024-04-09T10:35:11,422 [INFO ] W-9009-PhoneDetection_1.0-stdout MODEL_LOG - Torch worker started.
2024-04-09T10:35:11,450 [INFO ] W-9012-PhoneDetection_1.0-stdout MODEL_LOG - Successfully loaded /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2024-04-09T10:35:11,450 [INFO ] W-9009-PhoneDetection_1.0-stdout MODEL_LOG - Python runtime: 3.8.19
2024-04-09T10:35:11,434 [INFO ] W-9006-PhoneDetection_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9006, pid=92005
2024-04-09T10:35:11,428 [DEBUG] W-9009-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9009-PhoneDetection_1.0 State change null -> WORKER_STARTED
2024-04-09T10:35:11,451 [INFO ] W-9012-PhoneDetection_1.0-stdout MODEL_LOG - [PID]92009
2024-04-09T10:35:11,451 [INFO ] W-9006-PhoneDetection_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9006
2024-04-09T10:35:11,452 [INFO ] W-9009-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9009
2024-04-09T10:35:11,452 [DEBUG] W-9012-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9012-PhoneDetection_1.0 State change null -> WORKER_STARTED
2024-04-09T10:35:11,452 [INFO ] W-9012-PhoneDetection_1.0-stdout MODEL_LOG - Torch worker started.
2024-04-09T10:35:11,452 [INFO ] W-9009-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9009
2024-04-09T10:35:11,452 [DEBUG] W-9012-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9012-PhoneDetection_1.0 State change null -> WORKER_STARTED
2024-04-09T10:35:11,453 [INFO ] W-9012-PhoneDetection_1.0-stdout MODEL_LOG - Python runtime: 3.8.19
2024-04-09T10:35:11,459 [INFO ] W-9009-PhoneDetection_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9009.
2024-04-09T10:35:11,460 [INFO ] W-9012-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9012
2024-04-09T10:35:11,460 [INFO ] W-9012-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9012
2024-04-09T10:35:11,457 [INFO ] W-9006-PhoneDetection_1.0-stdout MODEL_LOG - Successfully loaded /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2024-04-09T10:35:11,459 [INFO ] W-9009-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633711459
2024-04-09T10:35:11,457 [INFO ] W-9011-PhoneDetection_1.0-stdout MODEL_LOG - model_name: PhoneDetection, batchSize: 1
2024-04-09T10:35:11,462 [INFO ] W-9006-PhoneDetection_1.0-stdout MODEL_LOG - [PID]92005
2024-04-09T10:35:11,463 [INFO ] W-9012-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633711463
2024-04-09T10:35:11,459 [INFO ] W-9009-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633711459
2024-04-09T10:35:11,463 [DEBUG] W-9006-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9006-PhoneDetection_1.0 State change null -> WORKER_STARTED
2024-04-09T10:35:11,463 [DEBUG] W-9006-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9006-PhoneDetection_1.0 State change null -> WORKER_STARTED
2024-04-09T10:35:11,466 [INFO ] W-9006-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9006
2024-04-09T10:35:11,466 [INFO ] W-9006-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9006
2024-04-09T10:35:11,463 [INFO ] W-9006-PhoneDetection_1.0-stdout MODEL_LOG - Torch worker started.
2024-04-09T10:35:11,463 [INFO ] W-9012-PhoneDetection_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9012.
2024-04-09T10:35:11,463 [INFO ] W-9012-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633711463
2024-04-09T10:35:11,471 [INFO ] W-9006-PhoneDetection_1.0-stdout MODEL_LOG - Python runtime: 3.8.19
2024-04-09T10:35:11,480 [INFO ] W-9006-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633711480
2024-04-09T10:35:11,480 [INFO ] W-9006-PhoneDetection_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9006.
2024-04-09T10:35:11,480 [INFO ] W-9006-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633711480
2024-04-09T10:35:11,476 [INFO ] W-9008-PhoneDetection_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9008, pid=92006
2024-04-09T10:35:11,481 [INFO ] W-9008-PhoneDetection_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9008
2024-04-09T10:35:11,524 [INFO ] W-9008-PhoneDetection_1.0-stdout MODEL_LOG - Successfully loaded /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2024-04-09T10:35:11,532 [INFO ] W-9009-PhoneDetection_1.0-stdout MODEL_LOG - model_name: PhoneDetection, batchSize: 1
2024-04-09T10:35:11,587 [INFO ] W-9008-PhoneDetection_1.0-stdout MODEL_LOG - [PID]92006
2024-04-09T10:35:11,601 [DEBUG] W-9008-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9008-PhoneDetection_1.0 State change null -> WORKER_STARTED
2024-04-09T10:35:11,601 [DEBUG] W-9008-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9008-PhoneDetection_1.0 State change null -> WORKER_STARTED
2024-04-09T10:35:11,630 [INFO ] W-9008-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9008
2024-04-09T10:35:11,630 [INFO ] W-9008-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9008
2024-04-09T10:35:11,600 [INFO ] W-9008-PhoneDetection_1.0-stdout MODEL_LOG - Torch worker started.
2024-04-09T10:35:11,617 [INFO ] W-9012-PhoneDetection_1.0-stdout MODEL_LOG - model_name: PhoneDetection, batchSize: 1
2024-04-09T10:35:11,648 [INFO ] W-9008-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633711648
2024-04-09T10:35:11,648 [INFO ] W-9008-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633711648
2024-04-09T10:35:11,646 [INFO ] W-9008-PhoneDetection_1.0-stdout MODEL_LOG - Python runtime: 3.8.19
2024-04-09T10:35:11,657 [INFO ] W-9006-PhoneDetection_1.0-stdout MODEL_LOG - model_name: PhoneDetection, batchSize: 1
2024-04-09T10:35:11,690 [INFO ] W-9008-PhoneDetection_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9008.
2024-04-09T10:35:11,713 [INFO ] W-9005-PhoneDetection_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9005, pid=92000
2024-04-09T10:35:11,727 [INFO ] W-9005-PhoneDetection_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9005
2024-04-09T10:35:11,753 [INFO ] W-9005-PhoneDetection_1.0-stdout MODEL_LOG - Successfully loaded /home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/configs/metrics.yaml.
2024-04-09T10:35:11,756 [INFO ] W-9005-PhoneDetection_1.0-stdout MODEL_LOG - [PID]92000
2024-04-09T10:35:11,759 [DEBUG] W-9005-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9005-PhoneDetection_1.0 State change null -> WORKER_STARTED
2024-04-09T10:35:11,759 [DEBUG] W-9005-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9005-PhoneDetection_1.0 State change null -> WORKER_STARTED
2024-04-09T10:35:11,762 [INFO ] W-9005-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9005
2024-04-09T10:35:11,762 [INFO ] W-9005-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9005
2024-04-09T10:35:11,759 [INFO ] W-9005-PhoneDetection_1.0-stdout MODEL_LOG - Torch worker started.
2024-04-09T10:35:11,764 [INFO ] W-9005-PhoneDetection_1.0-stdout MODEL_LOG - Python runtime: 3.8.19
2024-04-09T10:35:11,781 [INFO ] W-9005-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633711781
2024-04-09T10:35:11,781 [INFO ] W-9005-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1712633711781
2024-04-09T10:35:11,780 [INFO ] W-9005-PhoneDetection_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9005.
2024-04-09T10:35:11,807 [INFO ] W-9008-PhoneDetection_1.0-stdout MODEL_LOG - model_name: PhoneDetection, batchSize: 1
2024-04-09T10:35:11,908 [INFO ] W-9005-PhoneDetection_1.0-stdout MODEL_LOG - model_name: PhoneDetection, batchSize: 1
2024-04-09T10:35:12,593 [INFO ] W-9011-PhoneDetection_1.0-stdout MODEL_LOG - ONNX enabled
2024-04-09T10:35:12,594 [INFO ] W-9011-PhoneDetection_1.0-stdout MODEL_LOG - Torch TensorRT not enabled
2024-04-09T10:35:12,598 [INFO ] W-9007-PhoneDetection_1.0-stdout MODEL_LOG - ONNX enabled
2024-04-09T10:35:12,599 [INFO ] W-9007-PhoneDetection_1.0-stdout MODEL_LOG - Torch TensorRT not enabled
2024-04-09T10:35:12,600 [INFO ] W-9010-PhoneDetection_1.0-stdout MODEL_LOG - ONNX enabled
2024-04-09T10:35:12,600 [INFO ] W-9010-PhoneDetection_1.0-stdout MODEL_LOG - Torch TensorRT not enabled
2024-04-09T10:35:12,639 [INFO ] W-9009-PhoneDetection_1.0-stdout MODEL_LOG - ONNX enabled
2024-04-09T10:35:12,640 [INFO ] W-9009-PhoneDetection_1.0-stdout MODEL_LOG - Torch TensorRT not enabled
2024-04-09T10:35:12,844 [INFO ] W-9012-PhoneDetection_1.0-stdout MODEL_LOG - ONNX enabled
2024-04-09T10:35:12,847 [INFO ] W-9012-PhoneDetection_1.0-stdout MODEL_LOG - Torch TensorRT not enabled
2024-04-09T10:35:12,851 [INFO ] W-9008-PhoneDetection_1.0-stdout MODEL_LOG - ONNX enabled
2024-04-09T10:35:12,852 [INFO ] W-9008-PhoneDetection_1.0-stdout MODEL_LOG - Torch TensorRT not enabled
2024-04-09T10:35:12,898 [INFO ] W-9006-PhoneDetection_1.0-stdout MODEL_LOG - ONNX enabled
2024-04-09T10:35:12,901 [INFO ] W-9006-PhoneDetection_1.0-stdout MODEL_LOG - Torch TensorRT not enabled
2024-04-09T10:35:12,917 [DEBUG] W-9011-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:35:12,917 [DEBUG] W-9011-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:35:12,924 [INFO ] W-9011-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1464
2024-04-09T10:35:12,924 [INFO ] W-9011-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1464
2024-04-09T10:35:12,925 [DEBUG] W-9011-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9011-PhoneDetection_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:35:12,925 [DEBUG] W-9011-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9011-PhoneDetection_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:35:12,925 [INFO ] W-9011-PhoneDetection_1.0 TS_METRICS - WorkerLoadTime.Milliseconds:4846.0|#WorkerName:W-9011-PhoneDetection_1.0,Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633712
2024-04-09T10:35:12,927 [INFO ] W-9011-PhoneDetection_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:39.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633712
2024-04-09T10:35:12,935 [DEBUG] W-9010-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:35:12,935 [DEBUG] W-9010-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:35:12,941 [INFO ] W-9010-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1511
2024-04-09T10:35:12,941 [INFO ] W-9010-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1511
2024-04-09T10:35:12,944 [DEBUG] W-9010-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9010-PhoneDetection_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:35:12,944 [DEBUG] W-9010-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9010-PhoneDetection_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:35:12,945 [INFO ] W-9010-PhoneDetection_1.0 TS_METRICS - WorkerLoadTime.Milliseconds:4866.0|#WorkerName:W-9010-PhoneDetection_1.0,Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633712
2024-04-09T10:35:12,947 [INFO ] W-9010-PhoneDetection_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:70.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633712
2024-04-09T10:35:12,981 [INFO ] W-9005-PhoneDetection_1.0-stdout MODEL_LOG - ONNX enabled
2024-04-09T10:35:12,982 [INFO ] W-9005-PhoneDetection_1.0-stdout MODEL_LOG - Torch TensorRT not enabled
2024-04-09T10:35:13,012 [DEBUG] W-9009-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:35:13,012 [DEBUG] W-9009-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:35:13,013 [INFO ] W-9009-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1488
2024-04-09T10:35:13,013 [INFO ] W-9009-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1488
2024-04-09T10:35:13,013 [DEBUG] W-9009-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9009-PhoneDetection_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:35:13,013 [DEBUG] W-9009-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9009-PhoneDetection_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:35:13,014 [INFO ] W-9009-PhoneDetection_1.0 TS_METRICS - WorkerLoadTime.Milliseconds:4936.0|#WorkerName:W-9009-PhoneDetection_1.0,Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633713
2024-04-09T10:35:13,015 [INFO ] W-9009-PhoneDetection_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:68.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633713
2024-04-09T10:35:13,071 [DEBUG] W-9007-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:35:13,071 [DEBUG] W-9007-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:35:13,071 [INFO ] W-9007-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1780
2024-04-09T10:35:13,071 [INFO ] W-9007-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1780
2024-04-09T10:35:13,072 [DEBUG] W-9007-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9007-PhoneDetection_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:35:13,072 [DEBUG] W-9007-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9007-PhoneDetection_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:35:13,072 [INFO ] W-9007-PhoneDetection_1.0 TS_METRICS - WorkerLoadTime.Milliseconds:4997.0|#WorkerName:W-9007-PhoneDetection_1.0,Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633713
2024-04-09T10:35:13,073 [INFO ] W-9007-PhoneDetection_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:63.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633713
2024-04-09T10:35:13,203 [DEBUG] W-9012-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:35:13,203 [DEBUG] W-9012-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:35:13,203 [INFO ] W-9012-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1585
2024-04-09T10:35:13,203 [INFO ] W-9012-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1585
2024-04-09T10:35:13,203 [DEBUG] W-9012-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9012-PhoneDetection_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:35:13,203 [DEBUG] W-9012-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9012-PhoneDetection_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:35:13,204 [INFO ] W-9012-PhoneDetection_1.0 TS_METRICS - WorkerLoadTime.Milliseconds:5122.0|#WorkerName:W-9012-PhoneDetection_1.0,Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633713
2024-04-09T10:35:13,205 [INFO ] W-9012-PhoneDetection_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:156.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633713
2024-04-09T10:35:13,217 [DEBUG] W-9008-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:35:13,217 [DEBUG] W-9008-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:35:13,217 [INFO ] W-9008-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1404
2024-04-09T10:35:13,217 [INFO ] W-9008-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1404
2024-04-09T10:35:13,221 [DEBUG] W-9008-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9008-PhoneDetection_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:35:13,221 [DEBUG] W-9008-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9008-PhoneDetection_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:35:13,222 [INFO ] W-9008-PhoneDetection_1.0 TS_METRICS - WorkerLoadTime.Milliseconds:5146.0|#WorkerName:W-9008-PhoneDetection_1.0,Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633713
2024-04-09T10:35:13,223 [INFO ] W-9008-PhoneDetection_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:171.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633713
2024-04-09T10:35:13,229 [DEBUG] W-9006-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:35:13,229 [DEBUG] W-9006-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:35:13,230 [INFO ] W-9006-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1552
2024-04-09T10:35:13,230 [INFO ] W-9006-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1552
2024-04-09T10:35:13,230 [DEBUG] W-9006-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9006-PhoneDetection_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:35:13,230 [DEBUG] W-9006-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9006-PhoneDetection_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:35:13,231 [INFO ] W-9006-PhoneDetection_1.0 TS_METRICS - WorkerLoadTime.Milliseconds:5155.0|#WorkerName:W-9006-PhoneDetection_1.0,Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633713
2024-04-09T10:35:13,231 [INFO ] W-9006-PhoneDetection_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:199.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633713
2024-04-09T10:35:13,248 [DEBUG] W-9005-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:35:13,248 [DEBUG] W-9005-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:35:13,249 [INFO ] W-9005-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1341
2024-04-09T10:35:13,249 [INFO ] W-9005-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1341
2024-04-09T10:35:13,249 [DEBUG] W-9005-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9005-PhoneDetection_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:35:13,249 [DEBUG] W-9005-PhoneDetection_1.0 org.pytorch.serve.wlm.WorkerThread - W-9005-PhoneDetection_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED
2024-04-09T10:35:13,250 [INFO ] W-9005-PhoneDetection_1.0 TS_METRICS - WorkerLoadTime.Milliseconds:5176.0|#WorkerName:W-9005-PhoneDetection_1.0,Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633713
2024-04-09T10:35:13,250 [INFO ] W-9005-PhoneDetection_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:128.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633713
2024-04-09T10:36:09,567 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633769
2024-04-09T10:36:09,568 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:924.986083984375|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633769
2024-04-09T10:36:09,568 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:30.652225494384766|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633769
2024-04-09T10:36:09,569 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:3.2|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633769
2024-04-09T10:36:09,570 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:1889.4765625|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633769
2024-04-09T10:36:09,570 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:5678.94921875|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633769
2024-04-09T10:36:09,571 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:76.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633769
2024-04-09T10:37:09,563 [INFO ] pool-3-thread-2 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633829
2024-04-09T10:37:09,565 [INFO ] pool-3-thread-2 TS_METRICS - DiskAvailable.Gigabytes:924.986083984375|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633829
2024-04-09T10:37:09,566 [INFO ] pool-3-thread-2 TS_METRICS - DiskUsage.Gigabytes:30.652225494384766|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633829
2024-04-09T10:37:09,567 [INFO ] pool-3-thread-2 TS_METRICS - DiskUtilization.Percent:3.2|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633829
2024-04-09T10:37:09,567 [INFO ] pool-3-thread-2 TS_METRICS - MemoryAvailable.Megabytes:1905.0078125|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633829
2024-04-09T10:37:09,568 [INFO ] pool-3-thread-2 TS_METRICS - MemoryUsed.Megabytes:5663.77734375|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633829
2024-04-09T10:37:09,569 [INFO ] pool-3-thread-2 TS_METRICS - MemoryUtilization.Percent:75.8|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633829
2024-04-09T10:38:09,561 [INFO ] pool-3-thread-2 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633889
2024-04-09T10:38:09,562 [INFO ] pool-3-thread-2 TS_METRICS - DiskAvailable.Gigabytes:924.986083984375|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633889
2024-04-09T10:38:09,563 [INFO ] pool-3-thread-2 TS_METRICS - DiskUsage.Gigabytes:30.652225494384766|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633889
2024-04-09T10:38:09,565 [INFO ] pool-3-thread-2 TS_METRICS - DiskUtilization.Percent:3.2|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633889
2024-04-09T10:38:09,566 [INFO ] pool-3-thread-2 TS_METRICS - MemoryAvailable.Megabytes:1911.93359375|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633889
2024-04-09T10:38:09,568 [INFO ] pool-3-thread-2 TS_METRICS - MemoryUsed.Megabytes:5656.8515625|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633889
2024-04-09T10:38:09,569 [INFO ] pool-3-thread-2 TS_METRICS - MemoryUtilization.Percent:75.7|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633889
2024-04-09T10:39:09,560 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633949
2024-04-09T10:39:09,560 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:924.986083984375|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633949
2024-04-09T10:39:09,561 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:30.652225494384766|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633949
2024-04-09T10:39:09,562 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:3.2|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633949
2024-04-09T10:39:09,563 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:1905.84375|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633949
2024-04-09T10:39:09,563 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:5662.94921875|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633949
2024-04-09T10:39:09,564 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:75.8|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633949
2024-04-09T10:39:45,692 [INFO ] pool-2-thread-14 ACCESS_LOG - /[0:0:0:0:0:0:0:1%0]:47326 "GET /ping HTTP/1.1" 200 5
2024-04-09T10:39:45,693 [INFO ] pool-2-thread-14 TS_METRICS - Requests2XX.Count:1.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633985
2024-04-09T10:39:49,653 [INFO ] epollEventLoopGroup-3-2 ACCESS_LOG - /[0:0:0:0:0:0:0:1%0]:40140 "GET /models HTTP/1.1" 200 4
2024-04-09T10:39:49,654 [INFO ] epollEventLoopGroup-3-2 TS_METRICS - Requests2XX.Count:1.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712633989
2024-04-09T10:40:09,564 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634009
2024-04-09T10:40:09,565 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:924.986083984375|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634009
2024-04-09T10:40:09,566 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:30.652225494384766|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634009
2024-04-09T10:40:09,566 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:3.2|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634009
2024-04-09T10:40:09,567 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:1857.796875|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634009
2024-04-09T10:40:09,568 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:5710.99609375|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634009
2024-04-09T10:40:09,568 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:76.4|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634009
2024-04-09T10:40:38,197 [INFO ] epollEventLoopGroup-3-3 TS_METRICS - ts_inference_requests_total.Count:1.0|#model_name:FaceDetection,model_version:default|#hostname:DESKTOP-UR1F133,timestamp:1712634038
2024-04-09T10:40:38,299 [INFO ] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd PREDICT to backend at: 1712634038299
2024-04-09T10:40:38,299 [INFO ] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd PREDICT to backend at: 1712634038299
2024-04-09T10:40:38,301 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - Backend received inference at: 1712634038
2024-04-09T10:40:38,302 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - Invoking custom service failed.
2024-04-09T10:40:38,302 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - Traceback (most recent call last):
2024-04-09T10:40:38,303 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - File "/home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/service.py", line 134, in predict
2024-04-09T10:40:38,304 [INFO ] W-9001-FaceDetection_1.0 ACCESS_LOG - /[0:0:0:0:0:0:0:1%0]:49728 "PUT /predictions/FaceDetection HTTP/1.1" 503 109
2024-04-09T10:40:38,304 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - ret = self._entry_point(input_batch, self.context)
2024-04-09T10:40:38,304 [INFO ] W-9001-FaceDetection_1.0 TS_METRICS - Requests5XX.Count:1.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634038
2024-04-09T10:40:38,304 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - File "/tmp/models/514d97655cd546e9ae0fbdbd23c166bc/my_handler.py", line 13, in handle
2024-04-09T10:40:38,305 [DEBUG] W-9001-FaceDetection_1.0 org.pytorch.serve.job.Job - Waiting time ns: 100447902, Inference time ns: 107102902
2024-04-09T10:40:38,305 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - data = _service.preprocess(data)
2024-04-09T10:40:38,305 [DEBUG] W-9001-FaceDetection_1.0 org.pytorch.serve.job.Job - Waiting time ns: 100447902, Inference time ns: 107102902
2024-04-09T10:40:38,308 [DEBUG] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:40:38,308 [DEBUG] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:40:38,308 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - File "/home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/handler_utils/timer.py", line 27, in wrap_func
2024-04-09T10:40:38,309 [INFO ] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 2
2024-04-09T10:40:38,309 [INFO ] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 2
2024-04-09T10:40:38,309 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - "handler" in self.context.model_yaml_config
2024-04-09T10:40:38,310 [INFO ] W-9001-FaceDetection_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:9.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634038
2024-04-09T10:40:38,311 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - AttributeError: 'NoneType' object has no attribute 'model_yaml_config'
2024-04-09T10:41:09,615 [INFO ] pool-3-thread-2 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634069
2024-04-09T10:41:09,616 [INFO ] pool-3-thread-2 TS_METRICS - DiskAvailable.Gigabytes:924.9860763549805|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634069
2024-04-09T10:41:09,617 [INFO ] pool-3-thread-2 TS_METRICS - DiskUsage.Gigabytes:30.652233123779297|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634069
2024-04-09T10:41:09,618 [INFO ] pool-3-thread-2 TS_METRICS - DiskUtilization.Percent:3.2|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634069
2024-04-09T10:41:09,619 [INFO ] pool-3-thread-2 TS_METRICS - MemoryAvailable.Megabytes:1696.00390625|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634069
2024-04-09T10:41:09,619 [INFO ] pool-3-thread-2 TS_METRICS - MemoryUsed.Megabytes:5872.79296875|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634069
2024-04-09T10:41:09,620 [INFO ] pool-3-thread-2 TS_METRICS - MemoryUtilization.Percent:78.5|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634069
2024-04-09T10:42:09,595 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634129
2024-04-09T10:42:09,596 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:924.9860687255859|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634129
2024-04-09T10:42:09,597 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:30.652240753173828|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634129
2024-04-09T10:42:09,598 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:3.2|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634129
2024-04-09T10:42:09,599 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:1615.3359375|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634129
2024-04-09T10:42:09,600 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:5953.15234375|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634129
2024-04-09T10:42:09,600 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:79.5|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634129
2024-04-09T10:42:44,565 [INFO ] epollEventLoopGroup-3-4 TS_METRICS - ts_inference_requests_total.Count:1.0|#model_name:FaceDetection,model_version:default|#hostname:DESKTOP-UR1F133,timestamp:1712634164
2024-04-09T10:42:44,666 [INFO ] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd PREDICT to backend at: 1712634164666
2024-04-09T10:42:44,666 [INFO ] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd PREDICT to backend at: 1712634164666
2024-04-09T10:42:44,668 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - Backend received inference at: 1712634164
2024-04-09T10:42:44,668 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - Invoking custom service failed.
2024-04-09T10:42:44,668 [INFO ] W-9001-FaceDetection_1.0 ACCESS_LOG - /[0:0:0:0:0:0:0:1%0]:50760 "PUT /predictions/FaceDetection HTTP/1.1" 503 103
2024-04-09T10:42:44,669 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - Traceback (most recent call last):
2024-04-09T10:42:44,669 [INFO ] W-9001-FaceDetection_1.0 TS_METRICS - Requests5XX.Count:1.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634164
2024-04-09T10:42:44,669 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - File "/home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/service.py", line 134, in predict
2024-04-09T10:42:44,670 [DEBUG] W-9001-FaceDetection_1.0 org.pytorch.serve.job.Job - Waiting time ns: 100287301, Inference time ns: 103736101
2024-04-09T10:42:44,670 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - ret = self._entry_point(input_batch, self.context)
2024-04-09T10:42:44,670 [DEBUG] W-9001-FaceDetection_1.0 org.pytorch.serve.job.Job - Waiting time ns: 100287301, Inference time ns: 103736101
2024-04-09T10:42:44,670 [DEBUG] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:42:44,670 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - File "/tmp/models/514d97655cd546e9ae0fbdbd23c166bc/my_handler.py", line 13, in handle
2024-04-09T10:42:44,670 [DEBUG] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - sent a reply, jobdone: true
2024-04-09T10:42:44,672 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - data = _service.preprocess(data)
2024-04-09T10:42:44,672 [INFO ] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1
2024-04-09T10:42:44,672 [INFO ] W-9001-FaceDetection_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 1
2024-04-09T10:42:44,673 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - File "/home/hungtrieu07/miniconda3/envs/api_lanc/lib/python3.8/site-packages/ts/handler_utils/timer.py", line 27, in wrap_func
2024-04-09T10:42:44,673 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - "handler" in self.context.model_yaml_config
2024-04-09T10:42:44,673 [INFO ] W-9001-FaceDetection_1.0 TS_METRICS - WorkerThreadTime.Milliseconds:6.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634164
2024-04-09T10:42:44,674 [INFO ] W-9001-FaceDetection_1.0-stdout MODEL_LOG - AttributeError: 'NoneType' object has no attribute 'model_yaml_config'
2024-04-09T10:43:09,589 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634189
2024-04-09T10:43:09,590 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:924.9860649108887|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634189
2024-04-09T10:43:09,591 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:30.652244567871094|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634189
2024-04-09T10:43:09,591 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:3.2|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634189
2024-04-09T10:43:09,592 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:1773.9140625|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634189
2024-04-09T10:43:09,593 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:5794.8828125|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634189
2024-04-09T10:43:09,594 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:77.5|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634189
2024-04-09T10:44:09,561 [INFO ] pool-3-thread-2 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634249
2024-04-09T10:44:09,562 [INFO ] pool-3-thread-2 TS_METRICS - DiskAvailable.Gigabytes:924.9860649108887|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634249
2024-04-09T10:44:09,563 [INFO ] pool-3-thread-2 TS_METRICS - DiskUsage.Gigabytes:30.652244567871094|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634249
2024-04-09T10:44:09,563 [INFO ] pool-3-thread-2 TS_METRICS - DiskUtilization.Percent:3.2|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634249
2024-04-09T10:44:09,564 [INFO ] pool-3-thread-2 TS_METRICS - MemoryAvailable.Megabytes:1769.26953125|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634249
2024-04-09T10:44:09,565 [INFO ] pool-3-thread-2 TS_METRICS - MemoryUsed.Megabytes:5799.52734375|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634249
2024-04-09T10:44:09,566 [INFO ] pool-3-thread-2 TS_METRICS - MemoryUtilization.Percent:77.5|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634249
2024-04-09T10:45:09,589 [INFO ] pool-3-thread-2 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634309
2024-04-09T10:45:09,590 [INFO ] pool-3-thread-2 TS_METRICS - DiskAvailable.Gigabytes:924.9860649108887|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634309
2024-04-09T10:45:09,591 [INFO ] pool-3-thread-2 TS_METRICS - DiskUsage.Gigabytes:30.652244567871094|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634309
2024-04-09T10:45:09,592 [INFO ] pool-3-thread-2 TS_METRICS - DiskUtilization.Percent:3.2|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634309
2024-04-09T10:45:09,593 [INFO ] pool-3-thread-2 TS_METRICS - MemoryAvailable.Megabytes:1771.234375|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634309
2024-04-09T10:45:09,594 [INFO ] pool-3-thread-2 TS_METRICS - MemoryUsed.Megabytes:5797.55859375|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634309
2024-04-09T10:45:09,595 [INFO ] pool-3-thread-2 TS_METRICS - MemoryUtilization.Percent:77.5|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634309
2024-04-09T10:46:09,572 [INFO ] pool-3-thread-2 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634369
2024-04-09T10:46:09,573 [INFO ] pool-3-thread-2 TS_METRICS - DiskAvailable.Gigabytes:924.9860649108887|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634369
2024-04-09T10:46:09,574 [INFO ] pool-3-thread-2 TS_METRICS - DiskUsage.Gigabytes:30.652244567871094|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634369
2024-04-09T10:46:09,575 [INFO ] pool-3-thread-2 TS_METRICS - DiskUtilization.Percent:3.2|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634369
2024-04-09T10:46:09,575 [INFO ] pool-3-thread-2 TS_METRICS - MemoryAvailable.Megabytes:1755.60546875|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634369
2024-04-09T10:46:09,576 [INFO ] pool-3-thread-2 TS_METRICS - MemoryUsed.Megabytes:5813.19140625|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634369
2024-04-09T10:46:09,577 [INFO ] pool-3-thread-2 TS_METRICS - MemoryUtilization.Percent:77.7|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634369
2024-04-09T10:47:09,563 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634429
2024-04-09T10:47:09,564 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:924.9860649108887|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634429
2024-04-09T10:47:09,565 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:30.652244567871094|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634429
2024-04-09T10:47:09,566 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:3.2|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634429
2024-04-09T10:47:09,567 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:1751.53515625|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634429
2024-04-09T10:47:09,567 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:5817.26171875|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634429
2024-04-09T10:47:09,568 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:77.8|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634429
2024-04-09T10:48:09,594 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634489
2024-04-09T10:48:09,595 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:924.9860649108887|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634489
2024-04-09T10:48:09,596 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:30.652244567871094|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634489
2024-04-09T10:48:09,597 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:3.2|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634489
2024-04-09T10:48:09,598 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:1750.94921875|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634489
2024-04-09T10:48:09,598 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:5817.84765625|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634489
2024-04-09T10:48:09,599 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:77.8|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634489
2024-04-09T10:49:09,567 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634549
2024-04-09T10:49:09,568 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:924.9859657287598|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634549
2024-04-09T10:49:09,568 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:30.65234375|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634549
2024-04-09T10:49:09,569 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:3.2|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634549
2024-04-09T10:49:09,569 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:1616.15234375|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634549
2024-04-09T10:49:09,570 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:5952.6484375|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634549
2024-04-09T10:49:09,570 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:79.5|#Level:Host|#hostname:DESKTOP-UR1F133,timestamp:1712634549
from serve.
hungtrieu07
commented on May 24, 2024
@hungtrieu07 You can refer to this example.
https://github.com/pytorch/serve/blob/master/test/pytest/test_onnx.py#L49
You can inherit the base handler initialize method.
It worked. Thanks for your help.
from serve.
Related Issues (20)
- Serve multiple models with both CPU and GPU HOT 3
- How to modify torchserve’s Python runtime from 3.8.0 to 3.10 HOT 1
- TorchServe crashes in production with `WorkerThread - IllegalStateException error' HOT 4
- Unable to use build_image.sh to build the cu102 version of the image HOT 2
- Metrics collector crashes when NVIDIA MIGs are present HOT 1
- Server crashes in production with `WorkerThread - IllegalStateException error' HOT 1
- Whether the pre- and post-processing operations of batch processing are parallel HOT 1
- Update cpp/llamacpp to Llama 3 HOT 1
- Update LLM/llama2 to Llama3
- Update large_models/inferentia2/llama2 to Llama3
- Update large_models/tp_llama to llama3
- Update large_models/gpt_fast to llama3
- How to pass parameters from preprocessing to postprocessing when using micro-batch operations HOT 4
- Load model failed - error: Worker died HOT 6
- Docker regression failure: test_handler_traceback_logging.py
- Exchange Llama2 against Llama3 in HF_accelerate example
- CUDA out of Memory with low Memory Utilization (CUDA error: device-side assert triggered) HOT 5
- If micro_batch_size of micro-batch is set to 1, then model inference is still batch processing? HOT 1
- question to model inference optimization HOT 1
- Duplicate base_neuronx_continuous_batching_handler.py HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from serve.