asonje / pat Goto Github PK
View Code? Open in Web Editor NEWPerformance Analysis Tool
License: Other
Performance Analysis Tool
License: Other
I am trying to setup PAT on my rhel7 system with all the dependencies installed as stated in the github link. But i am facing below issue. I am not much of an expert in python and hence couldn't figure out the solution for the same. I have cloned the setup PAT github yesterday ie 12 june 2019.
Error seen at step :
cd matplotlib
[root@matplotlib]# ls -lrt
total 224
-rwxr-xr-x 1 root root 3905 Jun 11 07:08 README.rst
-rwxr-xr-x 1 root root 777 Jun 11 07:08 MANIFEST.in
drwxr-xr-x 2 root root 4096 Jun 11 07:08 LICENSE
-rwxr-xr-x 1 root root 11529 Jun 11 07:08 INSTALL.rst
-rwxr-xr-x 1 root root 932 Jun 11 07:08 build_alllocal.cmd
-rwxr-xr-x 1 root root 1281 Jun 11 07:08 azure-pipelines.yml
drwxr-xr-x 2 root root 4096 Jun 11 07:08 ci
drwxr-xr-x 12 root root 4096 Jun 11 07:08 doc
drwxr-xr-x 29 root root 4096 Jun 11 07:08 examples
drwxr-xr-x 5 root root 4096 Jun 11 07:08 extern
-rwxr-xr-x 1 root root 33371 Jun 11 07:08 matplotlibrc.template
drwxr-xr-x 4 root root 4096 Jun 11 07:08 lib
-rwxr-xr-x 1 root root 10285 Jun 11 07:08 setup.py
-rwxr-xr-x 1 root root 26629 Jun 11 07:08 setupext.py
-rwxr-xr-x 1 root root 2360 Jun 11 07:08 setup.cfg.template
drwxr-xr-x 4 root root 4096 Jun 11 07:08 requirements
-rwxr-xr-x 1 root root 425 Jun 11 07:08 pytest.ini
-rwxr-xr-x 1 root root 1574 Jun 11 07:08 tests.py
drwxr-xr-x 4 root root 4096 Jun 11 07:08 src
-rwxr-xr-x 1 root root 591 Jun 11 07:08 tox.ini
drwxr-xr-x 2 root root 4096 Jun 11 07:08 tools
drwxr-xr-x 8 root root 4096 Jun 11 07:08 tutorials
-rwxr-xr-x 1 root root 62731 Jun 11 07:08 versioneer.py
[root@matplotlib]#
Error:
[root@matplotlib]# python setup.py install
File "setup.py", line 175
for opt in [*Distribution.display_option_names, 'help'])
^
SyntaxError: invalid syntax
[root@matplotlib]#
Please help out.
Regards
Harsha
I am using ./pat run to collect node status data in computer cluster when trying to process big dataset. After the job is done, i want to convert the data with ./pat-post-process.py.
But, I get an error when trying to process the collected data using ./pat-post-process.py. It says ValueError: invalid literal for float(): 91,00.
I think this is because the data using "," not "." for decimal number. The first error showed when trying to process cpu data (with cpu_module.py). So, i change "," in cpustat file to "." and run ./pat-post-process.py. It passed the error.
But, this error affected with disk_module.py and net_module.py, too. And again i change "," to "." in memstat and iostat file to pass the error. It works well.
I'm using Python 2.7.12 (default, Nov 19 2016, 06:48:10) with matplotlib 1.5.1, and xlsxwriter 0.9.6. I hope this issue will be solved later.
When the following conditions are met at the same time, the result directory is deleted.
WORKER_SCRIPT_DIR
and MASTER_SCRIPT_DIR
is the same value.ALL_NODES
includes the master node (the master node is also a worker node).I think the above situation is likely to occur because the default INSTRUCTIONS in README.txt suggests ALL_NODES
can include the master node, and the default values of WORKER_SCRIPT_DIR
and MASTER_SCRIPT_DIR
are the same.
To avoid this, we propose following corrections. (just examples)
WORKER_SCRIPT_DIR
and MASTER_SCRIPT_DIR
Regards,
I have established a 10-node spark cluster and deploy PAT on the nodes.
However, after I conducted the word count workload, no KPIs were collected (i.e. the corresponding files are all empty) though it could show the logs during working.
It didn't report any WARN or ERROR, either. Could you please give me some help with such a strange situation?
I synced the time of 7 worker nodes and executed PAT on them.
The raw data shows that the time is alligned:
[hostname@sr563 instruments]$ head -2 sr537/cpustat
HostName TimeStamp CPU %user %nice %system %iowait %steal %idle
sr537 1560497986 all 0.04 0.00 0.21 0.06 0.00 99.69
[hostname@sr563 instruments]$ head -2 sr538/cpustat
HostName TimeStamp CPU %user %nice %system %iowait %steal %idle
sr538 1560497987 all 0.08 0.00 0.25 0.26 0.00 99.40
[hostname@sr563 instruments]$ head -2 sr539/cpustat
HostName TimeStamp CPU %user %nice %system %iowait %steal %idle
sr539 1560497986 all 0.07 0.00 0.11 0.00 0.00 99.82
However, after drawing the graph in Excel using macro, the data of different servers are perfectly stagger, you can see 7 similar shapes parallel:
Could you add a feature and configuration that would allow specifying the user name? That way with SSH_KEY and the proposed USER config, the tool will be able to specify the rest of this.
From this_user@localhost:
ssh -i ~/.ssh/id_rsa THAT_user@remotehost
Not every linux distribution comes with gawk as the default awk link, but gawk is required because of "strftime" used in e.g. PAT/WORKER_scripts/instruments.
If gawk is not the default awk, pat will silently fail and not produce any results.
On debian 7 only /usr/bin/nawk and /usr/bin/mawk are installed.
Please place a hint in readme.md pointing out this issue/requirement for gawk or make the awk binary path a configurable parameter in config.
best regards
Michael
We have been using some of this tooling and want to continue (and contribute) :) It would be useful to add a LICENCE to clarify.
Some devices may have never been used. I want to ignore them. How could we specify the device? Thank you!
I am seeing below error while processing the data. Please do help out. I do see that "Timestamp" is present in the data.
-bash-4.2# ./pat-post-process.py
---- You currently have Python 3.6.8 (default, Apr 21 2019, 08:11:10)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
---- You currently have matplotlib 3.1.0
---- You currently have xlsxwriter 1.1.8
---- You have all required dependencies
---- PAT-post-processing script will start automatically
Started processing on Fri Jun 28 07:11:07 2019
Traceback (most recent call last):
File "./pat-post-process.py", line 417, in
main()
File "./pat-post-process.py", line 411, in main
cluster = make_cluster(root[3].text)
File "./pat-post-process.py", line 234, in make_cluster
new_node = Node(node_path)
File "./pat-post-process.py", line 123, in init
self.cpu_obj = cpu_module.Cpu(self.node_file_paths[0])
File "/home/hs/PAT-tool/PAT/PAT-post-processing/cpu_module.py", line 47, in init
self.avg_array = self.extract_data()
File "/home/hs/PAT-tool/PAT/PAT-post-processing/cpu_module.py", line 56, in extract_data
self.ts_index = self.title_line.index("TimeStamp")
ValueError: 'TimeStamp' is not in list
-bash-4.2#
-bash-4.2# ls -lrt ./results/2019-06-28-06-58-39/instruments/dl380-rhel7.perflab.hp.com-22/
total 28912
-rw-r--r--. 1 root root 305959 Jun 28 06:57 vmstat
-rw-r--r--. 1 root root 3280828 Jun 28 06:57 netstat
-rw-r--r--. 1 root root 1085800 Jun 28 06:57 memstat
-rw-r--r--. 1 root root 92692 Jun 28 06:57 jvms
-rw-r--r--. 1 root root 1885334 Jun 28 06:57 iostat
-rw-r--r--. 1 root root 20596100 Jun 28 06:57 cpustat
-rw-r--r--. 1 root root 2344610 Jun 28 06:58 perfout
-bash-4.2# pwd
-bash-4.2# ls -lrt ./results/2019-06-28-06-58-39/instruments/dl380-rhel7.perflab.hp.com-22/
total 28912
-rw-r--r--. 1 root root 305959 Jun 28 06:57 vmstat
-rw-r--r--. 1 root root 3280828 Jun 28 06:57 netstat
-rw-r--r--. 1 root root 1085800 Jun 28 06:57 memstat
-rw-r--r--. 1 root root 92692 Jun 28 06:57 jvms
-rw-r--r--. 1 root root 1885334 Jun 28 06:57 iostat
-rw-r--r--. 1 root root 20596100 Jun 28 06:57 cpustat
-rw-r--r--. 1 root root 2344610 Jun 28 06:58 perfout
-bash-4.2# more ./results/2019-06-28-06-58-39/instruments/dl380-rhel7.perflab.hp.com-22/vmstat
HostName TimeStamp r b swpd free buff cache si so bi bo in cs us sy id wa st
dl380-rhel7.perflab.hp.com 1561717035 17 0 0 139352288 2312 46485300 0 0 0 0 0 0 0 0 100 0 0
dl380-rhel7.perflab.hp.com 1561717036 8 0 0 139073568 2312 46495468 0 0 0 572 29497 257744 2 2 96 0 0
dl380-rhel7.perflab.hp.com 1561717037 9 0 0 139051152 2312 46495476 0 0 0 0 25406 617070 1 3 96 0 0
dl380-rhel7.perflab.hp.com 1561717038 2 0 0 138954560 2312 46495620 0 0 0 0 23985 624872 1 3 96 0 0
dl380-rhel7.perflab.hp.com 1561717039 4 0 0 138841664 2312 46495692 0 0 0 0 23738 626724 1 3 96 0 0
dl380-rhel7.perflab.hp.com 1561717040 4 0 0 138730416 2312 46495692 0 0 0 52 24933 618301 1 3 96 0 0
dl380-rhel7.perflab.hp.com 1561717041 5 0 0 138687008 2312 46496684 0 0 0 188 19339 619724 1 3 96 0 0
dl380-rhel7.perflab.hp.com 1561717042 4 0 0 138507712 2312 46497940 0 0 0 0 21742 616269 1 3 96 0 0
dl380-rhel7.perflab.hp.com 1561717043 1 0 0 138482336 2312 46498076 0 0 0 0 24117 619453 1 3 96 0 0
dl380-rhel7.perflab.hp.com 1561717044 5 0 0 138482080 2312 46497884 0 0 0 0 24306 631239 1 3 96 0 0
Regards
Harsha
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.