Giter VIP home page Giter VIP logo

ioprof's Introduction

DISCONTINUATION OF PROJECT

This project will no longer be maintained by Intel.

Intel has ceased development and contributions including, but not limited to, maintenance, bug fixes, new releases, or updates, to this project.

Intel no longer accepts patches to this project.

If you have an ongoing need to use this project, are interested in independently developing it, or would like to maintain patches for the open source software community, please create your own fork of this project.

Contact: [email protected] Linux I/O Profiler (ioprof)

The Linux I/O profiler (ioprof) is a tool that provides significant insight into I/O workloads while remaining easy to use. It reports the following information:

  • I/O Histogram - Great for determining size of hot data for SSD caching
  • I/O Heatmap - Useful visualization to "see" where the hot data resides
  • I/O Size Stats - IOPS and bandwidth stats, which is useful for mixed workloads
  • Top Files (opt) - Can ID top accessed files in EXT3/EXT4 filesystems
  • Zipf Theta - An estimate of Zipfian distribution theta

The tool is recommended to be used to further analyze I/O intensive workloads after running tools like iostat, since blktrace/blkparse can affect performance.

It is intended to be stable enough to use to profile production systems and makes every attempt to minimize resource utilization. In additon, the trace file is self-contained and can be offloaded for analysis on a separate system.

Contents

  • README - This file
  • LICENSE - GPLv2 license
  • ioprof.pl - The script

Dependencies

Perl v5.x and Perl Core Library

Requires the following tools:

  • fdisk
  • blktrace
  • blkparse

Optional PDF report requires:

  • gnuplot
  • pdf2latex
  • terminal png

Design

The tool currently groups statistics into 1MB "buckets" to provide relatively accurate results, while minimizing system resources.

TODO:

  • Confirm XFS filesystem tracing
  • Add option to specifiy output file name
  • Add option to specify temp directory
  • Improve file mapping performance

Maintainers

Benjamin Donie [email protected]

ioprof's People

Contributors

bjdonie avatar rdower avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ioprof's Issues

Tar creation error for small traces

ioprof fails when creating traces < 3 seconds long

~/ioprof> ioprof.py -m trace -d /dev/nvme0n1  -r 2
Mapping files to block locations                
Creating tarball nvme0n1.tar
tar -cf nvme0n1.tar blk.out.nvme0n1.*.gz fdisk.nvme0n1  &> /dev/null
ERROR: failed to tarball nvme0n1.tar

(this only happens on the python version)

Please specify dependencies packages name or git repo

Hi,

Trying to use ioprof. I've been able to obtain a tar file from a trace, but can't create the report.

./ioprof.pl (2.0.5) Unpacking sda.tar. This may take a minute. Use of uninitialized value $total_lbas in scalar chomp at ./ioprof.pl line 1985. Argument "" isn't numeric in multiplication (*) at ./ioprof.pl line 1988. Use of uninitialized value $total_lbas in multiplication (*) at ./ioprof.pl line 1988. Use of uninitialized value $total_lbas in printf at ./ioprof.pl line 1990. lbas: 0 sec_size: 0 total: 0.00 GiB Use of uninitialized value $total_lbas in multiplication (*) at ./ioprof.pl line 1992. Time to parse. Please wait... Finished parsing files. Now to analyze Done correlating files to buckets. Now time to count bucket hits.

I suspect I miss some of the 'Optional dependencies' ?

I couldn't find what you refer to specifically in the optional PDF deps :

Optional PDF report requires:

gnuplot
pdf2latex
terminal png

What are 'pdf2latex' and 'terminal png' ? Packages names ? Can you point to git repos for those ?

Thanks !

Error in post-processing tar file

Apologies for opening a second issue so quickly.

The perl version of ioprof works correctly in "trace" mode. However, when I post-process the tar file, I get the following error message:

ioprof.pl -m post -t sdb1.tar
ioprof.pl (2.0.5)
Unpacking sdb1.tar. This may take a minute.
Argument "bytes" isn't numeric in multiplication (*) at /home/laytonjb/NVIDIA/IO_PROFILE/ioprof/ioprof.pl line 1988.
lbas: 5860531087 sec_size: 0 total: 0.00 GiB
Time to parse. Please wait...
Finished parsing files. Now to analyze
Done correlating files to buckets. Now time to count bucket hits.

Illegal modulus zero at /home/laytonjb/NVIDIA/IO_PROFILE/ioprof/ioprof.pl line 883.

Just in case, line 1988 is the following:

1988 $total_capacity_GiB = $total_lbas * $sector_size / $GiB;

Line 883 is the following:

883 while (($i % $xwidth) != 0) { $i++; $buffer .= "0 "; }

My system is Ubuntu 18.04. uname -r is "5.3.0-53-generic". (BTW - same system for the previous issue I opened).

Observation:

I'm trying to determine if ioprof can capture mmap() file IO. I'm wondering if the issue is that it didn't capture any IO so the post processing fails (no write or read bytes on the device). I can share the mmap() test code if needed (it's really, simple).

Thanks for the help!

Jeff

Error when using Python version of ioprof

I tried running the Python version of ioprof and I got an error.

The command line is:

ioprof.py -m trace -d /dev/sdb1 -r 3

The error is:

VERSION: 1.0.0.1
/usr/sbin/blktrace
/usr/bin/blkparse
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
0 % done (3 seconds left)No input files found for blk.out.sdb1.0
Mapping files to block locations
Creating tarball sdb1.tar
tar -cf sdb1.tar blk.out.sdb1.*.gz fdisk.sdb1 &> /dev/null
tar: blk.out.sdb1.0.blkparse.gz: Cannot open: No such file or directory
tar: Exiting with failure status due to previous errors
FINISHED tracing: sdb1.tar
Please use this file with ioprof.py -m post -t sdb1.tar to create a report

The Perl version works correctly.

Thanks for developing the tool!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.