Comments (6)
That's a pretty interesting use-case. I think we could avoid the camera->system copy by using the Tensor(void *data, TypeRef ty)
constructor to make a tensor backed by the camera memory, and then bind that tensor to an input Variable.
from glow.
@bertmaher I originally introduced the Tensor(void *data, TypeRef ty)
constructor exactly for integrating with other 3rd party frameworks and the like, where the tensor itself is allocated and manged outside Glow.
But of course, this constructor requires that the payload is at least in the same address space and has the same memory layout, alignment, padding, etc. There could be use-cases, where it is not the case. E.g. the payload of the tensor is in a different address space (e.g. GPU memory, etc) or has a different memory layout (e.g. padding, alignment, row vs column major, etc).
Such use-cases may indeed need more than just this constructor.
from glow.
@jsubag Jacob, how do graphic drivers solve this problem? When I program cuda/opencl code I don't need to allocate user-space buffers with the driver. I understand that in some cases the driver would need to copy the data. Do you know what latency will this copy introduce?
from glow.
@nadavrot On GPUs there's usually a copy between system memory and the GPU memory but here there are usages that may require an additional copy.
Consider the case of an application analyzing images taken from a camera on the same host machine. Typically, the captured images are written to a specific location designated by the camera driver. In the current Glow design this data will be copied to a Tensor used as an input (backed by system memory), then the backend will be able to copy it a second time to the GPU.
The first copy can be removed by exposing the right mechanism so that the application can initiate only the "second" copy from camera output buffer to the GPU resource.
There are other mechanisms in the GPU domain such as sharing EGL resources (potentially saving all copies) but i think those require a tighter handshake between the components & drivers.
Additionally, some GPUs require the system memory to be copied from be pinned and aligned.
If the memory backing the tensor isn't complying with these that can incur a third copy from the tensor system memory to a another system memory location that complies with the pinning/alignment requirements (this is usually the result of DMA requirements).
Latency numbers vary across different hardware - but if we're talking about copying a few MB in system memory on a modern computer that shouldn't take more than a millisecond.
However, if your input/output are larger and your workload is latency sensitive this can be the difference between reaching your target frame-rate and missing it.
So the more we can save the better :)
from glow.
@opti-mix For the cases you mention there should be a mechanism to trigger data transfer between the backend specific address space/layout/etc. and host memory space (maybe similar to GPU-style map/unmap).
from glow.
@jsubag This issue is related to #1334
from glow.
Related Issues (20)
- Request release - This pull request was **exported** from Phabricator. Differential Revision: [D39456245](https://www.internalfb.com/diff/D39456245) HOT 1
- model-compiler missing HOT 1
- BatchedReduceAdd and BatchedReduceProd multi-axis support
- Where Glow invokes llvm ? HOT 2
- build_Debug error HOT 1
- Multiple tensorflow-lite models in one application
- Glow cmsis support on aarch64
- Typed pointers support will be dropped in LLVM 17 HOT 1
- Compile error when building glow HOT 6
- Error occurred during the compilation of Glow
- FAILED: bin/image-classifier during building glow
- Unable to access glow model HOT 3
- Adding a library for a new backend, CMSIS. HOT 1
- Adding Support for ELU Activation functions
- Evaluate Profile-Guided Optimization (PGO)
- How to build glow for Buildroot based Linux?
- Sunsetting Glow Active Development HOT 1
- Error building on Mac Ventura
- Cannot open connection when downloading pb file
- Is the Glow still under development and maintenance? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from glow.