asonje / pat Goto Github PK

I am trying to setup PAT on my rhel7 system with all the dependencies installed as stated in the github link. But i am facing below issue. I am not much of an expert in python and hence couldn't figure out the solution for the same. I have cloned the setup PAT github yesterday ie 12 june 2019.

Error seen at step :
cd matplotlib
[root@matplotlib]# ls -lrt
total 224
-rwxr-xr-x 1 root root 3905 Jun 11 07:08 README.rst
-rwxr-xr-x 1 root root 777 Jun 11 07:08 MANIFEST.in
drwxr-xr-x 2 root root 4096 Jun 11 07:08 LICENSE
-rwxr-xr-x 1 root root 11529 Jun 11 07:08 INSTALL.rst
-rwxr-xr-x 1 root root 932 Jun 11 07:08 build_alllocal.cmd
-rwxr-xr-x 1 root root 1281 Jun 11 07:08 azure-pipelines.yml
drwxr-xr-x 2 root root 4096 Jun 11 07:08 ci
drwxr-xr-x 12 root root 4096 Jun 11 07:08 doc
drwxr-xr-x 29 root root 4096 Jun 11 07:08 examples
drwxr-xr-x 5 root root 4096 Jun 11 07:08 extern
-rwxr-xr-x 1 root root 33371 Jun 11 07:08 matplotlibrc.template
drwxr-xr-x 4 root root 4096 Jun 11 07:08 lib
-rwxr-xr-x 1 root root 10285 Jun 11 07:08 setup.py
-rwxr-xr-x 1 root root 26629 Jun 11 07:08 setupext.py
-rwxr-xr-x 1 root root 2360 Jun 11 07:08 setup.cfg.template
drwxr-xr-x 4 root root 4096 Jun 11 07:08 requirements
-rwxr-xr-x 1 root root 425 Jun 11 07:08 pytest.ini
-rwxr-xr-x 1 root root 1574 Jun 11 07:08 tests.py
drwxr-xr-x 4 root root 4096 Jun 11 07:08 src
-rwxr-xr-x 1 root root 591 Jun 11 07:08 tox.ini
drwxr-xr-x 2 root root 4096 Jun 11 07:08 tools
drwxr-xr-x 8 root root 4096 Jun 11 07:08 tutorials
-rwxr-xr-x 1 root root 62731 Jun 11 07:08 versioneer.py
[root@matplotlib]#

Error:
[root@matplotlib]# python setup.py install
File "setup.py", line 175
for opt in [*Distribution.display_option_names, 'help'])
^
SyntaxError: invalid syntax
[root@matplotlib]#

Please help out.

Regards
Harsha

I am using ./pat run to collect node status data in computer cluster when trying to process big dataset. After the job is done, i want to convert the data with ./pat-post-process.py.

But, I get an error when trying to process the collected data using ./pat-post-process.py. It says ValueError: invalid literal for float(): 91,00.

I think this is because the data using "," not "." for decimal number. The first error showed when trying to process cpu data (with cpu_module.py). So, i change "," in cpustat file to "." and run ./pat-post-process.py. It passed the error.

But, this error affected with disk_module.py and net_module.py, too. And again i change "," to "." in memstat and iostat file to pass the error. It works well.

I'm using Python 2.7.12 (default, Nov 19 2016, 06:48:10) with matplotlib 1.5.1, and xlsxwriter 0.9.6. I hope this issue will be solved later.

When the following conditions are met at the same time, the result directory is deleted.

• WORKER_SCRIPT_DIR and MASTER_SCRIPT_DIR is the same value.
• ALL_NODES includes the master node (the master node is also a worker node).

I think the above situation is likely to occur because the default INSTRUCTIONS in README.txt suggests ALL_NODES can include the master node, and the default values of WORKER_SCRIPT_DIR and MASTER_SCRIPT_DIR are the same.

To avoid this, we propose following corrections. (just examples)

• Add the notes in README.txt to avoid
• Modify the default values of WORKER_SCRIPT_DIR and MASTER_SCRIPT_DIR
• Modify the runall script to work correct.

Regards,

I have established a 10-node spark cluster and deploy PAT on the nodes.

However, after I conducted the word count workload, no KPIs were collected (i.e. the corresponding files are all empty) though it could show the logs during working.

It didn't report any WARN or ERROR, either. Could you please give me some help with such a strange situation?

I synced the time of 7 worker nodes and executed PAT on them.
The raw data shows that the time is alligned:

[hostname@sr563 instruments]$ head -2 sr537/cpustat
HostName TimeStamp CPU %user %nice %system %iowait %steal %idle
sr537 1560497986 all 0.04 0.00 0.21 0.06 0.00 99.69
[hostname@sr563 instruments]$ head -2 sr538/cpustat
HostName TimeStamp CPU %user %nice %system %iowait %steal %idle
sr538 1560497987 all 0.08 0.00 0.25 0.26 0.00 99.40
[hostname@sr563 instruments]$ head -2 sr539/cpustat
HostName TimeStamp CPU %user %nice %system %iowait %steal %idle
sr539 1560497986 all 0.07 0.00 0.11 0.00 0.00 99.82

However, after drawing the graph in Excel using macro, the data of different servers are perfectly stagger, you can see 7 similar shapes parallel:

Could you add a feature and configuration that would allow specifying the user name? That way with SSH_KEY and the proposed USER config, the tool will be able to specify the rest of this.

From this_user@localhost:
ssh -i ~/.ssh/id_rsa THAT_user@remotehost

Not every linux distribution comes with gawk as the default awk link, but gawk is required because of "strftime" used in e.g. PAT/WORKER_scripts/instruments.
If gawk is not the default awk, pat will silently fail and not produce any results.

On debian 7 only /usr/bin/nawk and /usr/bin/mawk are installed.

Please place a hint in readme.md pointing out this issue/requirement for gawk or make the awk binary path a configurable parameter in config.

best regards
Michael

We have been using some of this tooling and want to continue (and contribute) :) It would be useful to add a LICENCE to clarify.

Some devices may have never been used. I want to ignore them. How could we specify the device? Thank you!

I am seeing below error while processing the data. Please do help out. I do see that "Timestamp" is present in the data.

-bash-4.2# ./pat-post-process.py
---- You currently have Python 3.6.8 (default, Apr 21 2019, 08:11:10)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
---- You currently have matplotlib 3.1.0
---- You currently have xlsxwriter 1.1.8
---- You have all required dependencies
---- PAT-post-processing script will start automatically
Started processing on Fri Jun 28 07:11:07 2019
Traceback (most recent call last):
File "./pat-post-process.py", line 417, in
main()
File "./pat-post-process.py", line 411, in main
cluster = make_cluster(root[3].text)
File "./pat-post-process.py", line 234, in make_cluster
new_node = Node(node_path)
File "./pat-post-process.py", line 123, in init
self.cpu_obj = cpu_module.Cpu(self.node_file_paths[0])
File "/home/hs/PAT-tool/PAT/PAT-post-processing/cpu_module.py", line 47, in init
self.avg_array = self.extract_data()
File "/home/hs/PAT-tool/PAT/PAT-post-processing/cpu_module.py", line 56, in extract_data
self.ts_index = self.title_line.index("TimeStamp")
ValueError: 'TimeStamp' is not in list
-bash-4.2#

-bash-4.2# ls -lrt ./results/2019-06-28-06-58-39/instruments/dl380-rhel7.perflab.hp.com-22/
total 28912
-rw-r--r--. 1 root root 305959 Jun 28 06:57 vmstat
-rw-r--r--. 1 root root 3280828 Jun 28 06:57 netstat
-rw-r--r--. 1 root root 1085800 Jun 28 06:57 memstat
-rw-r--r--. 1 root root 92692 Jun 28 06:57 jvms
-rw-r--r--. 1 root root 1885334 Jun 28 06:57 iostat
-rw-r--r--. 1 root root 20596100 Jun 28 06:57 cpustat
-rw-r--r--. 1 root root 2344610 Jun 28 06:58 perfout
-bash-4.2# pwd

-bash-4.2# ls -lrt ./results/2019-06-28-06-58-39/instruments/dl380-rhel7.perflab.hp.com-22/
total 28912
-rw-r--r--. 1 root root 305959 Jun 28 06:57 vmstat
-rw-r--r--. 1 root root 3280828 Jun 28 06:57 netstat
-rw-r--r--. 1 root root 1085800 Jun 28 06:57 memstat
-rw-r--r--. 1 root root 92692 Jun 28 06:57 jvms
-rw-r--r--. 1 root root 1885334 Jun 28 06:57 iostat
-rw-r--r--. 1 root root 20596100 Jun 28 06:57 cpustat
-rw-r--r--. 1 root root 2344610 Jun 28 06:58 perfout
-bash-4.2# more ./results/2019-06-28-06-58-39/instruments/dl380-rhel7.perflab.hp.com-22/vmstat
HostName TimeStamp r b swpd free buff cache si so bi bo in cs us sy id wa st
dl380-rhel7.perflab.hp.com 1561717035 17 0 0 139352288 2312 46485300 0 0 0 0 0 0 0 0 100 0 0
dl380-rhel7.perflab.hp.com 1561717036 8 0 0 139073568 2312 46495468 0 0 0 572 29497 257744 2 2 96 0 0
dl380-rhel7.perflab.hp.com 1561717037 9 0 0 139051152 2312 46495476 0 0 0 0 25406 617070 1 3 96 0 0
dl380-rhel7.perflab.hp.com 1561717038 2 0 0 138954560 2312 46495620 0 0 0 0 23985 624872 1 3 96 0 0
dl380-rhel7.perflab.hp.com 1561717039 4 0 0 138841664 2312 46495692 0 0 0 0 23738 626724 1 3 96 0 0
dl380-rhel7.perflab.hp.com 1561717040 4 0 0 138730416 2312 46495692 0 0 0 52 24933 618301 1 3 96 0 0
dl380-rhel7.perflab.hp.com 1561717041 5 0 0 138687008 2312 46496684 0 0 0 188 19339 619724 1 3 96 0 0
dl380-rhel7.perflab.hp.com 1561717042 4 0 0 138507712 2312 46497940 0 0 0 0 21742 616269 1 3 96 0 0
dl380-rhel7.perflab.hp.com 1561717043 1 0 0 138482336 2312 46498076 0 0 0 0 24117 619453 1 3 96 0 0
dl380-rhel7.perflab.hp.com 1561717044 5 0 0 138482080 2312 46497884 0 0 0 0 24306 631239 1 3 96 0 0

Regards
Harsha