Expose ONNXRuntime memory arena shrinkage option as configurable parameter. about onnxruntime_backend HOT 4 CLOSED

Resolved.

from onnxruntime_backend.

@pranavsharma Thanks for implementing this but can arena shrink be smarter? Currently, user should specify where the model is (e.g. cpu:0), but actually triton knows where the model is better than user. I suggest user just specify this value as 0 or 1, and triton translates it to cpu:0, gpu:0, gpu:1 etc.

from onnxruntime_backend.

@zeruniverse We would like to mirror the configs from the ORT framework here as much as possible. This has a few advantages.

1. It allows us to point to existing documentation that is being continually enriched.
2. Users can search for the config in https://github.com/Microsoft/onnxruntime/issues or get perf recommendations and when a config is mentioned they won't have to think how to translate the ORT framework string to ORT backend config.
3. It avoids the unnecessary translation of configs from ORT backend to ORT framework invocations. This allows for easy reproducibility of ORT framework related issues. You can paste the exact config and file an issue.

When you configure Triton to run with ORT backend and CPU, you've to mention the instance kind to be CPU any way. I realize you've to repeat it in this string but the minor inconvenience is a tradeoff against the advantages listed above.

from onnxruntime_backend.

@pranavsharma thanks for your reply! Yes, I understand for CPU, user can just specify cpu:0. The problem is when there’re multiple GPUs(say 2) and each GPU has one instance of model X. In this case, instance 1 should have value gpu:0 and instance 2 should have gpu:1. I don’t know if user would be able to configure this since I assume parameter settings apply to all instances

from onnxruntime_backend.