Comments (3)
Hi @xunsongh,
At Graphcore we are looking to make changes to Kineto to support an IPU profiling backend.
We want to add another activity - e.g. IPUActivityAPI
but we don't want to compile time link this to Kineto/PyTorch. i.e. We would like to register IPUActivityAPI
(or perhaps IPUActivityProfiler
) at run-time with Kineto.
We had a few questions relating to this RFC:
- We understand that this RFC is a plan to add an
XPUActivityAPI
withinCuptiActivityProfiler
- extendingCuptiActivityProfiler
to support the XPU with additional XPU specific members and paths. This will be an addition to the existing CUPTI and Roctracer code paths. On its own, this change will add XPU support, but does not provide a generic way to add new profiling backends as you described in #647. Is our understanding correct? - What was the main difficulty with #647 and was there any investigation into how this could be broken into smaller changes?
- You state that "To my knowledge, Kineto is planning to separate CUPTI and Roctracer from the main profiler." Is there any more information about this to give regarding details and timeline? Will this introduce a system to register profiling backends?
Thanks in advance for taking the time to go over these questions and for any answers you can provide.
from kineto.
Hi @harryPorter828 ,
Thank you for your attention to this RFC and for taking the time to read and respond. However, after an online discussion in April, our XPU team has decided to follow the advice of the Kineto developers. We will adopt the form of a Kineto plugin, registering the XPU profiler as a relatively independent child profiler to the existing main profiler of Kineto. This approach avoids making changes to the main profiler of Kineto and does not involve directly pushing XPU-related integration code into the Kineto code repository. Instead, this code will be placed in PyTorch. Moreover, this plugin-based implementation is not directly related to what I previously proposed and what you mentioned in #647; they can even be considered two separate topics.
As developers on the XPU side, our primary goal is to enable PyTorch XPU users to utilize a profiler supported by Kineto without changing their usage habits, meaning with minimal changes to the front-end code. To achieve this, we are willing to first adapt to all existing designs and code logic of Kineto, reaching this goal in a way that is acceptable to all parties. In the future, we are open to participating in the structural reconstruction of Kineto, continuing to develop it into a more open, general, and user-friendly library. #647 was merely a conceptual idea towards this goal, and through extensive practice over this period, we have become aware of various issues with the original plan.
As for the current this RFC, it has been rejected by Meta's meeting, so we do not intend to discuss or explain it further. Once we complete and submit our design based on the Kineto plugin, I will close this RFC to avoid any misunderstandings.
Thank you.
from kineto.
Hi @xunsongh,
Thank you for your response.
Would you be able to elaborate on the Kineto plugin solution (which is different from #647) that you have mentioned? It was our understanding that a plugin type architecture does not yet exist in Kineto or as part of PyTorch profiler. Do you have a timescale for submitting the design for this new approach and will it be in the form of an RFC?
Regarding your point for 'continuing to develop it [Kineto] into a more open, general and user-friendly library', we would like to make it clear that we have similar goals and would be open to collaborating on these changes in the long run. However, we appreciate that adhering to existing Kineto designs may make sense for now.
from kineto.
Related Issues (20)
- The trace json file in the "Input Dims" field of aten::conv2d only has input shape and kernel shape
- TB_Plugin_CI failing with AttributeError: module 'mpmath' has no attribute 'rational'
- [Plugin-Bug]The Operators of baseline-run and exp-run are showed in a misaligned order HOT 1
- How to add customized metadata with on demand profiling ? HOT 7
- [Discussion] Which clock should we be using for timestamps? HOT 2
- GPU traces fail when using PyTorch lightning due to square braces in traceName HOT 2
- Support memory profiling feature from on-demand path
- Roctracer crashes when number of samples too high
- TypeError: bad operand type for unary -: 'NoneType' HOT 4
- [Synchronization events] Missing StreamWait event in cases
- KeyError: <torch_tb_profiler.profiler.node.OperatorNode object at 0x7f4a45dc3e80> HOT 1
- Module View cannot show device time HOT 5
- CUDA time difference between print function and Profiler TensorBoard
- 【Feature Request】Add Process Status Check Before Profiling to Handle Non-Running Training Tasks
- Upgrade to CUDA 12.4 is causing segfaults in 4 Range Profiler Tests
- [BUG] Number of communication kernels don't match between workers in run
- Train process is blocked when kineto is processing traceEvents HOT 1
- Update libfmt in kineto
- IN Build CUPTI_RUNTIME_TRACE_CBID_cudaLaunchKernelExC_v11060, not found HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kineto.