sflkit's Introduction

SFLKit: A Workbench for Statistical Fault Localization

SFLKit (https://dl.acm.org/doi/10.1145/3540250.3558915) is an out-of-the-box library and tool for statistical fault localization. Statistical fault localization aims at detecting execution features that correlate with failures, such as whether individual lines are part of the execution.

Language Support

SFLKit supports currently Python 3 but we plan on releasing further language support.

Installation

You need to navigate to the root directory of SFLKit and run

pip install sflkit

If you have a separate Python 2 and Python 3 on your machine you may need to run

pip3 install sflkit

Execution

To execute SFLKit you need to create a config file matching your needs.

Config

[target]
path=/path/to/the/subject
language=Python|C                       ; The programming language used

[events]
events=Event(,Event)*                   ; The events to investigate, overwritten by predicates.
predicates=Predicate(,Pridcate)*        ; The predicates to investigate, overwrites events.
metrics=Metric(,Metric)*                ; The metrics used for investigation
passing=/path(,path)*                   ; The event files of passing runs, if a dir is provided
                                        ; all files inside the tree will be treated as event files
failing=/path(,path)*                   ; The event files of failing runs, if a dir is provided
                                        ; all files inside the tree will be treated as event files

[instrumentation]
path=/path/to/the/instrumented/subject
exclude=file(,file)*                    ; Files to exclude from the instrumentation, should be a python re pattern

[test]
runner=TestRunner                       ; The testrunner class, None if no run needed

This is the specification of the config file.

Usage

The general usage of SFLKit is

usage: sflkit [-h] [--debug] -c CONFIG {instrumentation,analyze} ...

A workbench for statistical fault localization python programs and in the future other programs.

optional arguments:
  -h, --help            show this help message and exit
  --debug               the debug flag to activate debug information
  -c CONFIG, --config CONFIG
                        path to the config file

command:
  The framework allows for the execution of various commands.

  {instrumentation,analyze}
                        the command to execute
    instrumentation     execute the instrumentation of subject
    analyze             execute the analysis of the collected predicates

If you have adopted a config file for your investigations you need to execute

sflkit -c path/to/your/config instrument

to instrument the project defined by the file.

After the instrumentation, you can run your tests or experiments. But keep in mind to preserve the EVENTS_PATH file for each failing and passing run.

If you want to analyze your runs you need to execute

sflkit -c path/to/your/config analyze

which produces an output with the suggested code locations for the analysis objects and metrics defined in the config file.

Citing SFLKit

You can cite SFLKit as following

@inproceedings{10.1145/3540250.3558915,
  author = {Smytzek, Marius and Zeller, Andreas},
  title = {SFLKit: a workbench for statistical fault localization},
  year = {2022},
  isbn = {9781450394130},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3540250.3558915},
  doi = {10.1145/3540250.3558915},
  abstract = {Statistical fault localization aims at detecting execution features that correlate with failures, such as whether individual lines are part of the execution. We introduce SFLKit, an out-of-the-box workbench for statistical fault localization. The framework provides straightforward access to the fundamental concepts of statistical fault localization. It supports five predicate types, four coverage-inspired spectra, like lines, and 44 similarity coefficients, e.g., TARANTULA or OCHIAI, for statistical program analysis.  
  SFLKit separates the execution of tests from the analysis of the results and is therefore independent of the used testing framework. It leverages program instrumentation to enable the logging of events and derives the predicates and spectra from these logs. This instrumentation allows for introducing multiple programming languages and the extension of new concepts in statistical fault localization. Currently, SFLKit supports the instrumentation of Python programs. It is highly configurable, requiring only the logging of the required events.},
  booktitle = {Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering},
  pages = {1701–1705},
  numpages = {5},
  keywords = {similarity coefficient, spectrum-based fault localization, statistical debugging, statistical fault localization},
  location = {<conf-loc>, <city>Singapore</city>, <country>Singapore</country>, </conf-loc>},
  series = {ESEC/FSE 2022}
}

sflkit's People

Contributors

Stargazers

Watchers

sflkit's Issues

Exception has occurred: KeyError

middle.py:
def middle(x, y, z):
m = z
if y < z:
if x < y:
m = y
elif x < z:
m = y # bug
else:
if x > y:
m = y
elif x > z:
m = x
return m

tmp.py:
import sflkitlib.lib

def middle(x, y, z):
sflkitlib.lib.add_line_event(0)
m = z
sflkitlib.lib.add_line_event(1)
if y < z:
sflkitlib.lib.add_line_event(2)
if x < y:
sflkitlib.lib.add_line_event(3)
m = y
else:
sflkitlib.lib.add_line_event(4)
if x < z:
sflkitlib.lib.add_line_event(5)
m = y
else:
sflkitlib.lib.add_line_event(6)
if x > y:
sflkitlib.lib.add_line_event(7)
m = y
else:
sflkitlib.lib.add_line_event(8)
if x > z:
sflkitlib.lib.add_line_event(9)
m = x
sflkitlib.lib.add_line_event(10)
return m

working.py:
import enum
import importlib
import inspect
import os
import shutil
import middle

from IPython.display import HTML

from sflkit.color import ColorCode
from sflkit import instrument_config, analyze_config
from sflkit.config import Config

class TestResult(enum.Enum):

def repr(self):
    return self.value

PASS = 'PASS'
FAIL = 'FAIL'

def test(function, x, y, z, expected):
try:
if function(x, y, z) == expected:
return TestResult.PASS
else:
return TestResult.FAIL
except BaseException:
return TestResult.FAIL

def test_middle(x, y, z, expected):
return test(middle, x, y, z, expected)

test_middle(3, 2, 1, expected=2)
test_middle(3, 1, 2, expected=2)
test_middle(2, 1, 3, expected=2)

source = inspect.getsource(middle)
print(source)

middle_py = 'middle.py'
tmp_py = 'tmp.py'

with open(middle_py, 'w') as fp:
fp.write(source)

def test_middle_import(x, y, z, expected):
from middle import middle
return test(middle, x, y, z, expected)

test_middle_import(3, 2, 1, expected=2), test_middle_import(3, 1, 2, expected=2), test_middle_import(2, 1, 3, expected=2)

language='Python'
predicates='Line'
metrics='Tarantula'
passing='event-files/0,event-files/1'
failing='event-files/2'

def get_config():
return Config.create(path=middle_py, working=tmp_py, language=language, predicates=predicates, metrics=metrics, passing=passing, failing=failing)

def instrument(out=True):
instrument_config(get_config())
if out:
with open(tmp_py, 'r') as fp:
print(fp.read())

instrument()

def test_tmp(x, y, z, expected):
import tmp
importlib.reload(tmp)
tmp.sflkitlib.lib.reset()
try:
return test(tmp.middle, x, y, z, expected)
finally:
tmp.sflkitlib.lib.dump_events()
del tmp

event_files = 'event-files'

def run_tests():
if os.path.exists(event_files):
shutil.rmtree(event_files)
os.mkdir(event_files)
os.environ['EVENTS_PATH'] = os.path.join(event_files, '0')
test_tmp(3, 2, 1, expected=2)
os.environ['EVENTS_PATH'] = os.path.join(event_files, '1')
test_tmp(3, 1, 2, expected=2)
os.environ['EVENTS_PATH'] = os.path.join(event_files, '2')
test_tmp(2, 1, 3, expected=2)

def analyze():
run_tests()
print(Config)
return analyze_config(get_config())

results = analyze()

results

def sfl():
instrument(out=False)
results = analyze()
code = ColorCode(results[predicates.upper()][metrics])
return HTML(code.code(middle_py, source, color=True, suspiciousness=True))

sfl()

I wanted to test the library's functionality on a demo case
An exception KeyError is thrown
sflkit version 0.2.16
The same thing happens when using it through the terminal

Exception:
Traceback (most recent call last):
File "C:\SFLkitpr\working.py", line 98, in
results = analyze()
File "C:\SFLkitpr\working.py", line 96, in analyze
return analyze_config(get_config())
File "..\sflkit_init_.py", line 46, in analyze_config
analyzer.analyze()
File "..\sflkit\analysis\analyzer.py", line 32, in analyze
self._analyze(event_file)
File "..\sflkit\analysis\analyzer.py", line 24, in _analyze
for event in event_file.load():
File "..\sflkit\model\event_file.py", line 34, in load
yield event.load_next_event(self._file_pointer, self.mapping.mapping)
File "..\sflkitlib\events\event.py", line 587, in load_next_event
event = events[read_int(stream, int.from_bytes(test, ENDIAN))]
KeyError: 0

Recommend Projects

uds-se / sflkit Goto Github PK

sflkit's Introduction

SFLKit: A Workbench for Statistical Fault Localization

Language Support

Installation

Execution

Config

Usage

Citing SFLKit

sflkit's People

Contributors

Stargazers

Watchers

Forkers

sflkit's Issues

Exception has occurred: KeyError

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent