Comments (4)
Now that I've been working with this a little more, I see why this is an issue. Ragged batches are going to be pretty common and we need to do something on our side to do batching. I'm running an experiment that requires generating ~200 tokens and the 6B model takes like 22 seconds on an A10 without batching (~110ms/token sounds too high, not sure what's going on). If batching is enabled and ragged batches exist, then triton processes each request in the batch serially but doesn't return till it's done processing the entire batch. So I see a latency of ~120s because of the fake "batching". I don't see why it does that but that's been my observation.
Here's a log to show that:
https://pastebin.com/36WqBC3F
Btw the current copilot_proxy code calls generate() in the main thread, so there's no concurrency there either. I've fixed that on my local version for now and will send a PR soon.
from fauxpilot.
I have a possibly dumb question -- is there a benefit of doing this? I thought Triton already supports dynamic batching, so why would proxy code need to handle this?
from fauxpilot.
#
Max number of tokens the model can handle
MAX_MODEL_LEN = 2048
FYI, When I actually tested the maximum length of the token using Codegen model, it was possible to 2025 instead of 2048.
from fauxpilot.
I have a possibly dumb question -- is there a benefit of doing this? I thought Triton already supports dynamic batching, so why would proxy code need to handle this?
Oh, you may be right! I should do some perf testing to see if there's any benefit to sending the batch ourselves vs letting Triton batch the inputs. I may have been confused when I read this issue (triton-inference-server/fastertransformer_backend#23) because looking more closely it only applies to ragged batches.
from fauxpilot.
Related Issues (20)
- Maybe add windows/etc installer all-in-one in this project's 'releases'.
- 400 Bad Request when file has around 100 lines of code HOT 3
- C# support! HOT 2
- Hello all. The comments above have been very helpful in setting up the Copilot extension. I managed to get it to work with my instance and figured I would combine the steps I used (this is for Windows. Linux installation is similar, just different locations):
- It was working fine before... HOT 1
- Support for AMD GPUs HOT 1
- Triton doesnt exist anymore I think? HOT 3
- K8s deployment (via helm chart) HOT 2
- Caught signal 11 (Segmentation fault: address not mapped to object at address (nil)) HOT 1
- why my response are all !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! HOT 3
- Can I merge images of triton and client into one?eg fastertransformer_backend get content_fetch <fastertransformer&client>in CMakeLists ? HOT 1
- help me HOT 1
- What is the comparison of these model in huggingface? HOT 2
- Python Backend: "Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0" HOT 2
- [promptlib] proxy {"cause":{}} HOT 1
- ollama HOT 2
- Company Proxy HOT 1
- is documentation outdated?
- Jetbrains Support
- RTX 4060 Unsupported Message
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fauxpilot.