Comments (2)
Just took a look at this. There was a change to guard the use of that callback based on CUPTI API VERSION.
#792 that enables this above CUPTI API version >=17
Just checking the headers however.
CUDA 11.7.1 (and 11.7.0) do not have the CUPTI_RUNTIME_TRACE_CBID_cudaLaunchKernelExC_v11060 callback
https://gitlab.com/nvidia/headers/cuda-individual/cupti/-/blob/cuda-11.7.1/cupti_runtime_cbid.h?ref_type=tags
CUPTI API Version 17
https://gitlab.com/nvidia/headers/cuda-individual/cupti/-/blob/cuda-11.7.1/cupti_version.h?ref_type=tags#L104
And,
CUDA 11.8.0 does have CUPTI_RUNTIME_TRACE_CBID_cudaLaunchKernelExC_v11060 callback
https://gitlab.com/nvidia/headers/cuda-individual/cupti/-/blob/cuda-11.8.0/cupti_runtime_cbid.h?ref_type=tags#L440
CUPTI API Version 18
https://gitlab.com/nvidia/headers/cuda-individual/cupti/-/blob/cuda-11.8.0/cupti_version.h?ref_type=tags#L105
I'll add a fix to update the define. @moonbucks for a local fix you can change the '>=' part in /home/user/pytorch/third_party/kineto/libkineto/src/CuptiActivity.cpp:247 to 18 and it probably will work. Let us know
from kineto.
Changing 17 to 18 solved the problem. Thanks for your help!
from kineto.
Related Issues (20)
- Why PyTorch TensorBoard Profiler (Deprecated) HOT 3
- Fail to profile CUDA activities when ProfilerActivity.CPU is not enabled HOT 3
- The results captured in "DIFF" view are incomplete compared to those in "NORMAL" view HOT 1
- Question about how to run "make test" correctly? HOT 2
- CUPTI symbols are undefined after libkineto build HOT 1
- The trace json file in the "Input Dims" field of aten::conv2d only has input shape and kernel shape
- TB_Plugin_CI failing with AttributeError: module 'mpmath' has no attribute 'rational'
- [Plugin-Bug]The Operators of baseline-run and exp-run are showed in a misaligned order HOT 1
- How to add customized metadata with on demand profiling ? HOT 5
- [RFC] Support XPU Backend With PTI-sdk in Kineto HOT 3
- [Discussion] Which clock should we be using for timestamps? HOT 2
- GPU traces fail when using PyTorch lightning due to square braces in traceName HOT 2
- Support memory profiling feature from on-demand path
- Roctracer crashes when number of samples too high
- TypeError: bad operand type for unary -: 'NoneType' HOT 4
- [Synchronization events] Missing StreamWait event in cases
- KeyError: <torch_tb_profiler.profiler.node.OperatorNode object at 0x7f4a45dc3e80> HOT 1
- Module View cannot show device time HOT 5
- CUDA time difference between print function and Profiler TensorBoard
- 【Feature Request】Add Process Status Check Before Profiling to Handle Non-Running Training Tasks
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kineto.