Comments (10)
Possibly a dumb question, but is there any reason to prefer NtQuerySystemInformation over GetProcessMemoryInfo? Is it because you're expecting the WS monitored process list to be large and NtQuerySystemInformation ends up being cheaper? Or is it that the numbers returned are not actually the same in all cases? I guess with chrome.exe, the process list could indeed be quite large.
From a quick test on my Win7 machine, users in the administrators group cannot access csrss.exe processes PROCESS_QUERY_LIMITED_INFORMATION, which NtQuerySystemInformation ignores, so there's that advantage. But enabling SeDebugPrivilege for the UIforETW process token allows access to those processes, too.
There's possibly lots of other exciting and useful info that is only available through NTQSI, I just don't know if this is one of them.
from uiforetw.
Whatever works I will be happy with. Getting pagefile usage information is a bit of a black art it seems so it is quite likely that there are better ways. https://codereview.chromium.org/1181263005 shows some work I did in Chromium to get private working set data more efficiently than my normal methods and that took much crazy research and spelunking.
from uiforetw.
Sorry for the delay, my weekends and nights suddenly got busy.
Looking briefly at that Chromium issue, I guess it depends on what data you actually want to record. From the struct definition from ProcessHacker, I believe the only memory information that is not available from a fully documented, single API call are VirtualSize, PeakVirtualSize, WorkingSetPrivateSize, and HardFaultCount. The VirtualSize ones aren't necessarily interesting for the processes in this list, and HardFaultCount is probably redundant with other ETW data.
Since SYSTEM_PROCESS_INFORMATION::WorkingSetPrivateSize exists, "Private WS" could possibly be moved out of just being reported with expensive monitoring. (In a quick test, it seemed like SYSTEM_PROCESS_INFORMATION lined up with both UIforETW and PDH's "Working Set - Private" counter were reporting for that number, but VMMap reported a slightly larger number.)
Also, I should have looked at the source code for SampleWorkingSets earlier since you obviously already know about GetProcessMemoryInfo. Woops.
Since NTQSI(SystemProcessInformation) returns similar data as the toolhelp snapshot, my initial step would be to write a class that internally switches between the two methods of enumerating the processes so that UIforETW "gracefully degrades" if it is running against an unknown ntdll, or do you think that is overkill? I'd probably still write some helpers to manage the iteration since internally dealing with NextEntryOffset's pointer arithmetic and UNICODE_STRING is ugly and "uncommon", but deferring the switching until some later time.
from uiforetw.
I'd have to test and see if WorkingSetPrivateSize gives the numbers that I want. It would certainly be useful to be able to get that efficiently. HardFaultCount would also be useful if it was a process-lifetime total. And PrivatePageCount could be useful, except that I don't know what it measures.
PSS is not available of course.
For gracefully degrading keep in mind that ETW only usefully works on Windows 7 and above.
I would potentially be interested in ways of getting more memory information, cheaper. In particular, getting the amount of private data for a process that has been paged out would be pretty sweet. I won't have time to do any work on this for a while, but I'm generally willing to accept pull requests (as long as the CLA is signed - see the CONTRIBUTING file).
from uiforetw.
HardFaultCount appears to be process lifetime, or if it is not, it has a weird time that it resets back to zero. PrivatePageCount appears to be mislabeled as it exactly matches PrivateUsage from PROCESS_MEMORY_COUNTERS_EX (which is documented as being the same as PageFileUsage). This comparison is on Win10 64bit.
On gracefully degrading, I'm more worried about structure definition changes in future operating systems / service packs. Maybe it doesn't actually matter, since the user community is likely small, so it can be broken for a short period of time.
Agreed that PSS would still have to be expensive-WS only. When I get some spare time, I'm hoping to write something that displays the different numbers for working-set related items. It won't be pretty, but hopefully it will be useful. I'll leave a note here if it's ready before you've done your tests.
from uiforetw.
Sounds good. And, just to make it perfectly clear, it is normal and expected to add to or modify the working set ETW events in order to present the information ideally. I should probably have added a second working set event for when the expensive data is not available rather than calling the old one with zeroes - maximum clarity to analysts is the goal.
from uiforetw.
Okay, good. Also, how nitpicky do you want to be about the definition of "working set"? If large page and AWE allocation information were added, should that information be recorded in a new ETW working set event? Or should it be something entirely separate from "working set"?
from uiforetw.
To avoid confusion it is probably best to stick to the standard definition of working set, and use extra columns for large page and AWE allocations. I thought that AWE was pretty rare now that 64-bit is widely available. Accounting for large pages somewhere seems like a good idea - I didn't realize that they don't get counted as part of the regular working set.
The extra AWE and large-data columns could either be always present or a different event could be used when needed - whatever shows up in the UIforETW default Generic Events table the best.
from uiforetw.
The ETW provider Microsoft-Windows-Kernel-Memory
has a keyword KERNEL_MEM_KEYWORD_WS_SWAP
("0x80"). Here there are some events that occur when data are paged out/paged in:
<template tid="WorkingSetOutSwapStartArgs">
<data name="ProcessId" inType="win:UInt32"/>
</template>
<template tid="WorkingSetOutSwapStopArgs">
<data name="ProcessId" inType="win:UInt32"/>
<data name="Status" inType="win:HexInt32"/>
<data name="PagesProcessed" inType="win:UInt32"/>
</template>
<template tid="WorkingSetInSwapStopArgs">
<data name="ProcessId" inType="win:UInt32"/>
<data name="Status" inType="win:HexInt32"/>
</template>
<template tid="WorkingSetOutSwapStartArgs_V1">
<data name="ProcessId" inType="win:UInt32"/>
<data name="Flags" inType="win:HexInt32"/>
</template>
<template tid="WorkingSetOutSwapStopArgs_V1">
<data name="ProcessId" inType="win:UInt32"/>
<data name="Status" inType="win:HexInt32"/>
<data name="PagesProcessed" inType="win:Pointer"/>
<data name="WriteCombinePagesProcessed" inType="win:Pointer"/>
<data name="UncachedPagesProcessed" inType="win:Pointer"/>
<data name="CleanPagesProcessed" inType="win:Pointer"/>
</template>
from uiforetw.
Interesting. The page faults graph already shows data being paged in, but data being paged out would be a bit interesting.
However what I really want is to know how much data is currently paged out. Knowing how much paged out data (in pagefile.sys mostly, or maybe by file) there is per process, sampled occasionally, would be awesome. The paging out may have happened before the trace was recorded so I'm not sure that recording data about data being paged out during the trace is particularly meaningful.
from uiforetw.
Related Issues (20)
- Could UIForETW help in a case of getting only the necessary CPU/Process data? HOT 3
- Do you know how to simulate win10 `perfmon.exe` to monitor the disk operation of a process? HOT 3
- No data and empty presets in new WPA even after 'Copy Startup Profiles' HOT 2
- Broken File I/O recording? HOT 7
- Feature Request for Remote start/stop HOT 2
- UIforETW shortcut doesn't launch from within a folder named 'UI for ETW' HOT 5
- -stackwalk: Unknown flag 'HandleCreate' HOT 3
- Cannot fine exe file HOT 5
- UI for ETW version 1.56 - Locks up when I choose to Start Tracing. HOT 2
- Security Policy violation Binary Artifacts HOT 26
- 需要翻译!need translation! HOT 1
- Windows Performance Analyzer stops at 99%
- This not maybe issue, but I'm asking for help, someone who know how read results from UIforETW HOT 1
- UIforETW does not start if windows 10 sdk is not in default path HOT 15
- Record CPU Core Frequency HOT 1
- ETW Only showing 1 process HOT 4
- VirusTotal reports some detections in etwpackage1.58.zip HOT 7
- Need help! HOT 4
- App is currently showing a spurious update notification HOT 1
- Chrome special keyword flags are outdated
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from uiforetw.