Torchserve Error: number of batch response mismatched about serve HOT 9 CLOSED

@rajeshmore1 Could you please share the logs and if possible, the handler you are using?
Also, you said you are using without GPU, any specific reason to use the pytorch/torchserve:latest-gpu image?

from serve.

If you are running on intel cpu, please refer to this link to improve performance

https://github.com/pytorch/serve/tree/master/examples/intel_extension_for_pytorch

from serve.

We used pytorch/torchserve:latest-gpu this base image because if we want to go for gpu in future, we don't need to update this. Is this base image causing a problem for getting good throughput (without gpu)? Please guide.

from serve.

@rajeshmore1 No, that should be fine. Can you please check the ipex example that I shared.

from serve.

The error happens here https://github.com/pytorch/serve/blob/master/ts/service.py#L161

I notice that you have many try/except statements . If a batch of 4 is being processed, the result must have 4 elements. Please make sure you are handling this

To benchmark torchserve and see the effect of num_workers , you can use this example to benchmark with different batch_sizes. concurrent requests, workers
https://github.com/pytorch/serve/tree/master/examples/benchmarking/resnet50

from serve.

What I understood is I need to update the handler script that will process the multiple requests let's say 4? Could you please provide sample scrip for that? Do I need to update the inference function? Could you please elaborate this solution?

Also I will try to implement the ipex example that you have provided and let you know.

from serve.

@rajeshmore1 You can refer to this https://github.com/pytorch/serve/blob/master/examples/image_classifier/near_real_time_video/near_real_time_video_handler.py

Basically, you need to make sure that if n requests are being batched, the output has a list of n elements

If this is not happening, problem is usually in the pre-process or post-process function. You can print the number of elements or use a debugger.

You can also follow this example to debug your handler with a debugger

https://github.com/pytorch/serve/tree/master/examples/image_classifier/resnet_18#debug-torchserve-backend

from serve.

Thank you. We are working on it.

from serve.

please re-open if you need more assistance

from serve.