anthemiusguo / aspersa Goto Github PK

Maciek created a tool that can pause and resume any process.  This could be
run from 'stalk' as pause/resume:

> > #!/usr/bin/perl -w
> > use POSIX ":sys_wait_h";
> > use DBI;
> > system('/etc/rc.d/lvm_snapshot start');
> > $pid= fork();
> > if ($pid == 0)
> > {
> >   $t= time();
> >   exec('/bin/tar -C /mnt/dbbackup -c -z -v -f
> > /var/storage/'.$t.'.tar.gz mysql');
> > }
> > $was_running= 1;
> > $running= 1;
> > while(1)
> > {
> >   if ($running == 0 && $was_running == 1) { kill 19, $pid;
> > $was_running= 0; print "BACKUP STOPPED!\n"; }
> >   if ($running == 1 && $was_running == 0) { kill 18, $pid;
> > $was_running= 1; print "BACKUP RESUMED!\n"; }
> >   my $dbh= DBI->connect("DBI:mysql:host=localhost", 'user', 'pass');
> >   if (!$dbh) { $running= 0; next; };
> >   my $sth= $dbh->prepare('SHOW GLOBAL STATUS LIKE "Threads_connected"');
> >   $sth->execute();
> >   my $row= $sth->fetchrow_arrayref();
> >   if ($$row[1] < 50) { $running= 1; }
> >   if ($$row[1] > 150) { $running= 0; }
> >   $sth->finish();
> >   $dbh->disconnect();
> >   undef($dbh);
> >   $kid = waitpid($pid, WNOHANG);
> >   if ($kid > 0) { last; }
> >   sleep(1);
> > }
> > system('/etc/rc.d/lvm_snapshot stop');

Move mk-log-server from Maatkit into Aspersa as a Bash script.

Make ioprofile aggregate by the "all" keyword, so you can see the system calls 
a process makes, not aggregated by file/pid.  Change the output in this case: 
print the time and system call one per line, sort by time desc (or whatever the 
-c and -a are), and include a total.

Percona customer issue 11354 has samples of the commands and output from tw_cli.

Import initial code from Maatkit.  See Maatkit issue:
http://code.google.com/p/maatkit/issues/detail?id=923

See Greg Sabino Mullane's 'boxinfo' tool on bucardo.org for more inspiration.

in addition to generating and aggregating, I want to be able to use pmp to save 
a file full of stacktraces.  (By default it should be /tmp/aspersa, and should 
be deleted after running.)

Also, /tmp/aspersa.awk needs to be deleted after running.

I have a sample file where ioprofile claims that fsync() on 
/var/log/mysqld/error.log is a big time consumer.  This is fd=1.  In the strace 
file, I can see that the fsyncs are really on fd=11, which is the iblogfile.

Great tool.

I get this error in the disk schedulers and queue size section.

It also fails the detect my Smart Array P400i RAID controler (HP DL360g5 
server), but I will try to make a patch for that.

# Mounted Filesystems ########################################
  Filesystem                    Size Used Type  Opts          Mountpoint
  /dev/cciss/c0d0p1              97M  22% ext3  (rw)          /boot
  /dev/mapper/VolGroup00-LogVol00127G  52% ext3  (rw)          /
  none                          1.8G   1% binfmt_misc (rw)          
/var/lib/xenstored
  none                          1.8G   1% tmpfs (rw)          
/var/lib/xenstored
  tmpfs                         1.8G   0% tmpfs (rw)          /dev/shm
# Disk Schedulers And Queue Size #############################
cat: /sys/block/[h,s]d*/queue/scheduler: No such file or directory
cat: /sys/block/[h,s]d*/queue/nr_requests: No such file or directory
     [h,s]d* |
# LVM Volumes ################################################
  LV       VG         Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  LogVol00 VolGroup00 -wi-ao 130.78G
  LogVol01 VolGroup00 -wi-ao   5.81G
# RAID Controller ############################################
  Controller | No RAID controller detected

The content of /sys/block:
[root@xen-01 ~]# ls -l /sys/block
total 0
drwxr-xr-x 7 root root 0 May 12 13:45 cciss!c0d0
drwxr-xr-x 4 root root 0 Feb 19 08:17 dm-0
drwxr-xr-x 4 root root 0 Feb 19 08:17 dm-1
drwxr-xr-x 5 root root 0 Feb 19 08:18 md0
drwxr-xr-x 4 root root 0 Feb 19 08:17 ram0
drwxr-xr-x 4 root root 0 Feb 19 08:17 ram1
drwxr-xr-x 4 root root 0 Feb 19 08:17 ram10
drwxr-xr-x 4 root root 0 Feb 19 08:17 ram11
drwxr-xr-x 4 root root 0 Feb 19 08:17 ram12
drwxr-xr-x 4 root root 0 Feb 19 08:17 ram13
drwxr-xr-x 4 root root 0 Feb 19 08:17 ram14
drwxr-xr-x 4 root root 0 Feb 19 08:17 ram15
drwxr-xr-x 4 root root 0 Feb 19 08:17 ram2
drwxr-xr-x 4 root root 0 Feb 19 08:17 ram3
drwxr-xr-x 4 root root 0 Feb 19 08:17 ram4
drwxr-xr-x 4 root root 0 Feb 19 08:17 ram5
drwxr-xr-x 4 root root 0 Feb 19 08:17 ram6
drwxr-xr-x 4 root root 0 Feb 19 08:17 ram7
drwxr-xr-x 4 root root 0 Feb 19 08:17 ram8
drwxr-xr-x 4 root root 0 Feb 19 08:17 ram9

I am not sure what directory should be used insted of [h,s]d*

Please add the following to the summary sections

# Query cache
- show hits / inserts ratio
- show lowmem prunes

# InnoDB
- in addition to log file size, show checkpoint age, or the difference between 
these
- show history list length
- show # open read views, # open trx, and age (seconds) of oldest trx
- note if any trx have very large # of undo records

$ sh summary 
# Aspersa System Summary Report ##############################
Current date | 2010-03-06 13:51:02 UTC (local TZ: EST -0500)
    Hostname | shilo.local
      Uptime | 6 days,  8:35, 3 users, load averages: 0.25 0.19 0.24
     Release | 
grep: /proc/cpuinfo: No such file or directory
Architecture | 32-bit CPU, 64-bit OS
getconf: no such configuration parameter `GNU_LIBPTHREAD_VERSION'
   Threading | 
# Processor ##################################################
cp: /proc/cpuinfo: No such file or directory
  Processors |        0 physical, 0 cores, 0 virtual, no hyperthreading
      Speeds | 
      Models | 
      Caches | 
# Memory #####################################################
summary: line 172: free: command not found
summary: line 49: [: val: integer expression expected
summary: line 52: [: val: integer expression expected
summary: line 55: [: val: integer expression expected
summary: line 60: export: `=0.00k': not a valid identifier
       Total | 
summary: line 180: + : syntax error: operand expected (error token is " ")

Here is a template for the HP Smart Array controller. I have not 
implemented parsing of the output from hpacucli.

612d611
<    ls -l /sys/block > /tmp/aspersa2
621,622d619
<    elif grep -q 'cciss' /tmp/aspersa2; then
<       controller='HP Smart Array'
643,648d639
<    elif [ "${controller}" = "HP Smart Array" ]; then
<       if [ -x /usr/sbin/hpacucli ]; then
<          if /usr/sbin/hpacucli ctrl all show config > /tmp/aspersa; then
<             cat /tmp/aspersa
<          fi
<       fi


The result looks like this:

# RAID Controller ############################################
  Controller | HP Smart Array

Smart Array P400i in Slot 0 (Embedded)    (sn: PH73MU7325     )

   array A (SAS, Unused Space: 0 MB)


      logicaldrive 1 (136.7 GB, RAID 1, OK)

      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 146 GB, OK)
      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 146 GB, OK)

# Virtualization #############################################

mext2 was born from efforts to make things like command-line option parsing 
work on all platforms.  I actually think that we can parse options manually, 
the same way e.g. pmp does it, and rip the mext code out and replace it by some 
awk code like what mysql-summary does, and we'll have a great tool.

What steps will reproduce the problem?
1. Use the following architecture as listed I get this error producing your
60 character hashline

     SELinux | Disabled
./summary: line 47: printf: -v: invalid option
printf: usage: printf format [arguments]
  Processors | 2 physical, 2 cores, 4 virtual, has hyperthreading

     46 section () {
     47    printf -vline "#_%-60s" "$1_"
     48    line="${line// /#}"
     49    printf "%s\n" "${line//_/ }"
     50 }


What is the expected output? What do you see instead?

     SELinux | Disabled
# Processor ##################################################


What version of the product are you using? On what operating system?

     Release | Red Hat Enterprise Linux ES release 4 (Nahant Update 4)
      Kernel | 2.6.18-028stab056
Architecture | 32-bit CPU, 32-bit OS
   Threading | NPTL 2.5
 Compiled by | GNU CC version 4.1.2 20071124 (Red Hat 4.1.2-41).
     SELinux | Disabled


Please provide any additional information below.

What steps will reproduce the problem?
1.

:;  /bin/sh summary 
summary: line 591: syntax error near unexpected token `('
summary: line 591: `   join <(df -hP|sort) <(mount|sort) > /tmp/aspersa'


What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?
Centos 5.3.

Please provide any additional information below.

issue opened at the request of xaprb.  For reference from irc:

16:30:18 < xaprb> HCoyote, I think I can avoid that idiom.  I'll just
"df|sort>file" 
                  and then "mount|sort - file > otherfile"

On http://code.google.com/p/aspersa/wiki/Main?tm=2

you have a mysql-summary command, but the summary output, as well as no details 
of the summary command.

$ ./mext -r -- mysqladmin ext -i10 -c3
getopt: illegal option -- n
Usage: ./mext [-r] -- commandline
   -r     Relative: subtract each column from the previous column.
   --     Option separator; after this, all arguments are passed
          to the program whose output mext will columnize.
mext columnizes repeated output from a program like mysqladmin extended.
Example: 'mext -r -- mysqladmin ext -i10 -c3'
You can also work with data from a file:
Example: 'mext -r -- cat mysqladmin-output.txt'
If this doesn't work on your system, maybe mext2 will help

Show /proc/scsi/scsi in a digested format.

http://aspersa.googlecode.com/svn/trunk/align

"Finally, it goes through and deides"

Create a script to prompt the user for hosts, groups, users, checks and so
on -- and create a templated set of Nagios config files based on some kind
of skeleton structure.  This is a very vague idea at this point and needs a
lot more thought and a good spec.

I want a little utility that will rewrite iozone's output (sort of
pivot-table it) so it is easier to paste into an email, and easier to read.

Running as a normal user I get.

  Swappiness | ./summary: line 576: sysctl: command not found

The issue is that while sysctl is on the system, it's not in the path.

$ which sysctl
/usr/bin/which: no sysctl in
(/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/home/rbradfor/bi
n:/usr/local/sbin)
$ ls /sbin/sysctl 
/sbin/sysctl


The solution here is to do several things.  Redirecting stderr to /dev/null
is no first trying to get possible information

1. Add common extra locations such as  /sbin and /usr/sbin to the PATH
within the script execution.
2. Pre-check commands, e.g.   [ -z `which sysctl` ] && # Do something to
try and find it before ignoring command
3. Add a pre-check if you need to run command as root.  e.g. some commands
need root permissions, e.g. ethtool, mii-tool

I do this.

USER_ID=`id -u`
[ "${USER_ID}" -eq 0 ] && ROOT_USER="Y"

If any command must be run by root, then I have a rootcmd wrapper function,
that skips over if it's not root and gives a warning.


If nobody gets to this before me, I can refactor some of my own monitoring
code to support this.  Just a tad busy at present.

Grab strace + lsof, and turn the result into a tabular display of file 
descriptor, read/pread/write/pwrite/open/close, and the count and time and size 
of each operation in the cells of the table.  At the far right, print the 
filename and size of the file.  Example:

  lsof | grep mysql > files;
  strace -f -e read,write,pread,pwrite,open -p 2362 -T -xx -s 8 > strace 2>&1
  grep -o 'pread([0-9]*' strace | cut -d'(' -f2 | sort |uniq -c | sort -rn
  grep 7792uW files

Currently summary detects a MegaSAS as a MegaRAID, but it's not the same
thing.  I also need to learn more about the tools that inspect these
controllers.  I think it's "megasas" which has a similar syntax to MegaCli64.

On Debian, the split() function in awk doesn't work the same way it does 
elsewhere, leading to lines pretty-printed like this:

port                       = 12345  = 
socket                     = /tmp/12345/mysql_sandbox12345.sock = 
pid-file                   = /tmp/12345/data/mysql_sandbox12345.pid = 

This is hard to test because it doesn't fail on most awk variants.

What steps will reproduce the problem?


./summary

      Uptime | 447 days, 17:22,  1 user,  load average: 1.56, 1.43, 1.42
Invalid string keyword: chassis-type
Valid string keywords are:
  bios-vendor
  bios-version
  bios-release-date
  system-manufacturer
  system-product-name
  system-version
  system-serial-number
  baseboard-manufacturer
  baseboard-product-name
  baseboard-version
  baseboard-serial-number
  baseboard-asset-tag
  chassis-manufacturer
  chassis-version
  chassis-serial-number
  chassis-asset-tag
  processor-manufacturer
  processor-version
      System | Supermicro PDSMU v0123456789 ()


Line 466 is not valid on this architecture

    463    if [ "${vendor}" ]; then
    464       product="$(dmidecode -s system-product-name | sed 's/ *$//g')"
    465       version="$(dmidecode -s system-version | sed 's/ *$//g')"
    466       chassis="$(dmidecode -s chassis-type | sed 's/ *$//g')"
    467       system="${vendor} ${product} v${version} (${chassis})"
    468       name_val "System" "${system%%[ ]}";
    469    fi



What is the expected output? What do you see instead?

The output is fine, this case just needs to be handled as an error and
discarded.

      System | Supermicro PDSMU v0123456789 ()


What version of the product are you using? On what operating system?

     Release | CentOS release 5.2 (Final)
      Kernel | 2.6.18-92.1.22.el5
Architecture | 64-bit CPU, 64-bit OS


Please provide any additional information below.

$ cat flashcache_stats
read_hits=1 read_hit_percent=6 write_hits=0 write_hit_percent=0
dirty_write_hits=0 dirty_write_hit_percent=0 replacement=0
write_replacement=0 write_invalidates=0 read_invalidates=1
pending_enqueues=0 pending_inval=0 metadata_dirties=15 metadata_cleans=0
cleanings=0 no_room=0 front_merge=0 back_merge=0 nc_pid_adds=0
nc_pid_dels=0 nc_pid_drops=0 nc_expiry=0

Idea: make a tool that can repeatedly read such a file.  On the first read
it parses the name= part and makes a column header line.  On each
subsequent read it outputs incremental stats.

   * alert if non-standard my.cnf location
   * show size of slow log
   * look for incorrectly rotated slow / error logs with lsof
   * show read/write percentage (number of queries, and number of rows
     from Handler)

There are a couple of tools that don't check for and remove temp files they 
use: pmp, mext

sh mext  -r -- mysqladmin -u root -p -c3 -i10 ext

fails with

expr: non-numeric argument
sort: open failed: /tmp/mext_temp_file{0..1}: No such file or directory
sort: open failed: /tmp/mext_temp_file: No such file or directory
mv: `/tmp/mext_temp_file' and `/tmp/mext_temp_file' are the same file
expr: non-numeric argument
expr: non-numeric argument
awk: line 1: syntax error at or near )

However, I don't have the output of mysqladmin, so I have no clue what's wrong.

Make the 'collect' script automatically purge samples that are older than a 
specified time period.  This way it'll both avoid running out of disk space and 
keep collecting samples for diagnosing problems.

Add a tool that will generate a quick, readable, diffable, wiki-able,
email-able report of the system's configuration, hardware, and load
characteristics.

Add info on the following:

# default 60, can be very low, shouldn't be higher than default.
/proc/sys/net/ipv4/tcp_fin_timeout
/proc/sys/net/ipv4/ip_local_port_range

Aggregate TCP connections by remote IP, local port, local IP, and TCP status.
netstat -antp | grep 3306 | awk '{print $5}' | cut -d: -f1 | sort | uniq -c
| sort -rn

Check for errors in ifconfig.

Use ethtool or mii-tool to discover interfaces and print their model and
speed, and attached IP addresses.  (It might also be available in dmesg).

Report on scheduler

Hey there,

Many kudos for your scripts! I'm planning to use 'summary' in production and am 
making some changes to accommodate company specifics.

e.g. you can now run it in one go:
wget -qO- http://github.com/true/true-public-scraps/raw/master/summary |bash

and it reports Ubuntu versions more specifically than Debian-based.

Let me know if I'm violating any licenses (think not as it's bsd style) or 
policies of yours in doing this:
http://github.com/true/true-public-scraps/compare/708d964996b7955567b1...master

I want an iostat tool that can not only show stats separately for reads and 
writes (iostat doesn't always), but also label things non-confusingly, and can 
read the diskstat stuff that 'collect' saves, in 3 different ways.

Make a tool that can analyze InnoDB status and show interesting things about it 
that are hard to see by manual inspection, such as:

- Number of semaphores blocked, and what the hottest ones are.
- What are the queries from the blocked threads, if they're query threads?
- What thread(s) are blocking other threads, and what are their queries?
- Automate Yasufumi's analysis technique from his 2010 UC slides.

# Table cache ################################################
                Size | 0
               Usage | 0%

     15 for file in /tmp/aspersa /tmp/aspersa2; do
     16    if [ -e "${file}" ]; then
     17       echo "For security reasons, ${file} must not exist.";
     18       exit 1;
     19    fi
     20    if ! touch "${file}"; then
     21       echo "I can't make my temp file ${file}";
     22       exit 1;
     23    fi
     24 done
...
    292 #
########################################################################
    293 # All done.  Signal the end so it's explicit.
    294 #
########################################################################
    295 rm -f /tmp/aspersa*;
    296 section The_End

I have this case where /etc/my.cnf exists but mysqld has been started with 
another config file.

mysql-summary will just check if /etc/my.cnf exists, if not, it tries 
/etc/mysql/my.cnf.
So it displayed wrong configuration settings for me.

I've made a little patch, that checks with `ps` what value there is in 
--defaults-file.
This will work when running multiple mysql instances or having some special 
configuration setup.

There are probably some more issues like when having renamed the mysqld binary 
(because multiple versions are in place), but this one will work in most cases.

lines  script
610:   # TODO: look for /var/log/dmesg first.
611:   dmesg | grep '^scsi[0-9]' > /tmp/aspersa
612:   if grep -qi ': .*megaraid' /tmp/aspersa; then


Check:
root@server:~# dmesg | grep '^scsi[0-9]'
root@server:~# dmesg | grep 'scsi[0-9]'
[    3.375696] scsi1 : ahci
[    3.375821] scsi2 : ahci
[    3.375887] scsi3 : ahci
[    3.375946] scsi4 : ahci
[    3.376008] scsi5 : ahci
[    3.376068] scsi6 : ahci
[    3.572429] scsi0 : aacraid

Maybe a better solution: 
dmesg | grep 'scsi[0-9] :'

Cheers,
Jan

0 259401555 flashcache stats:
        reads(0), writes(1656159)
        read hits(0), read hit percent(0)
        write hits(0) write hit percent(0)
        dirty write hits(0) dirty write hit percent(0)
        replacement(0), write replacement(18516)
        write invalidates(0), read invalidates(0)
        pending enqueues(0), pending inval(0)
        metadata dirties(1406024), metadata cleans(31874)
        cleanings(31874), no room(249254) front merge(29156) back merge(1253)
        nc_pid_adds(0), nc_pid_dels(0), nc_pid_drops(0) nc_expiry(0)

Nice tool! 

I guess the output speaks for itself, so I'll just get to that:

[root@cassandra4 ~]# ./ioprofile  -s 10 -p 31118
Sat Oct  9 22:44:52 BST 2010
./ioprofile: line 381: kill: (10812) - No such process
....

I downloaded the tool from trunk at 10:40 pm BST, so presumably it's the latest 
version, running on RHEL 5.5.

I suppose strace exits cleanly after receiving SIGINT on line 379, so the 
subsequent kill fails.  
It's harmless and I can't see any side effects from it, so it's more an FYI 
really.

The summary tool really should not get into specific things like mysql
summaries.  But, we could make summary do something like this: prompt "do
you want to see a mysql summary?" and if so, wget and execute the
summary-mysql tool.

It's nonsensical to count something like Max_used_connections per second, or 
Open_tables per second, because it isn't a counter, it is a gauge (in RRDTool 
terms), or as Deva says, it's a "fact", not a counter.  Omit these.

Make the 'summary' tool support omreport.  Percona customer case 12612 has a 
good example of a server where this would be useful.

In case lsof doesn't work: make it use fd_NNNNN for filenames it can't figure 
out.

Default format is still too verbose -- make it support a super-compact format.

On the attached file, mext shows

baron@kanga:~$ mext -r -- cat /tmp/m
Handler_commit                                0      0
Handler_delete                                0      0
Handler_discover                              0      0
Handler_prepare                               0      0
Handler_read_first                            0      0
Handler_read_key                              0      2
Handler_read_next                             0      0
Handler_read_prev                             0      2
Handler_read_rnd                              0      0
Handler_read_rnd_next                         5     16
Handler_rollback                              0      0
Handler_savepoint                             0      0
Handler_savepoint_rollback                    0      0
Handler_update                                0      0
Handler_write                                18     15

I want it to show

baron@kanga:~$ mext -r -- cat /tmp/m
Handler_commit                                0      0      0
Handler_delete                                0      0      0
Handler_discover                              0      0      0
Handler_prepare                               0      0      0
Handler_read_first                            0      0      0
Handler_read_key                              0      2      2
Handler_read_next                             0      0    236
Handler_read_prev                             0      2      0
Handler_read_rnd                              0      0      0
Handler_read_rnd_next                         5     16     16
Handler_rollback                              0      0      0
Handler_savepoint                             0      0      0
Handler_savepoint_rollback                    0      0      0
Handler_update                                0      0      0
Handler_write                                18     15     15

The downloads page provides wget commands for all except summary which has
a description. Guessing it's called summary is lucky, should be in the docs.

wget http://aspersa.googlecode.com/svn/trunk/summary