Giter VIP home page Giter VIP logo

verificarlo / verificarlo Goto Github PK

View Code? Open in Web Editor NEW
61.0 8.0 25.0 13.98 MB

A tool for debugging and assessing floating point precision and reproducibility.

License: Other

Shell 6.82% Fortran 5.02% C 46.80% R 0.10% C++ 4.02% Makefile 1.60% Python 30.83% M4 2.61% Dockerfile 0.19% Gnuplot 0.03% Modula-3 0.01% SourcePawn 0.01% JavaScript 0.51% HTML 1.47% Cython 0.01%
montecarlo-arithmetic mpfr precision floating-point tool numerical-analysis llvm mixed-precision

verificarlo's Introduction

Verificarlo v2.0.0

A tool for debugging and assessing floating point precision and reproducibility.

verificarlo logo

GitHub Release Build Status Docker Pulls DOI Coverity License

Installation

To install Verificarlo please refer to the installation documentation. Verificarlo supports both amd64 and aarch64 architectures.

Using Verificarlo through its Docker image

A docker image is available at https://hub.docker.com/r/verificarlo/verificarlo/. This image uses the latest release version of Verificarlo and includes support for Fortran. It uses llvm-7 and gcc-7.

Example of usage with Monte Carlo arithmetic:

$ cat > test.c <<HERE
#include <stdio.h>
int main() {
  double a = 0;
  for (int i=0; i < 10000; i++) a += 0.1;
  printf("%0.17f\n", a);
  return 0;
}
HERE

$ docker pull verificarlo/verificarlo
$ docker run -v "$PWD":/workdir verificarlo/verificarlo \
   verificarlo-c test.c -o test
$ docker run -v "$PWD":/workdir -e VFC_BACKENDS="libinterflop_mca.so" \
   verificarlo/verificarlo ./test
999.99999999999795364
$ docker run -v "$PWD":/workdir -e VFC_BACKENDS="libinterflop_mca.so" \
   verificarlo/verificarlo ./test
999.99999999999761258

Usage

To automatically instrument a program with Verificarlo you must compile it using the verificarlo --linker=<linker> command, where <linker> depends on the targeted language:

  • verificarlo --linker=clang for C
  • verificarlo --linker=clang++ for C++
  • verificarlo --linker=flang for Fortran

Verificarlo uses the linker clang by default.

You can also use the provided wrappers to call verificarlo with the right linker:

  • verificarlo-c for C
  • verificarlo-c++ for C++
  • verificarlo-f for Fortran

First, make sure that the verificarlo installation directory is in your PATH.

Then you can use the verificarlo-c, verificarlo-f and verificarlo-c++ commands to compile your programs. Either modify your makefile to use verificarlo as the compiler (CC=verificarlo-c, FC=verificarlo-f, and CXX=verificarlo-c++) and linker (LD=verificarlo --linker=<linker>) or use the verificarlo command directly:

   $ verificarlo-c program.c -o ./program

If you are trying to compile a shared library, such as those built by the Cython extension to Python, you can then also set the shared linker environment variable (LDSHARED='verificarlo --linker=<linker> -shared') to enable position-independent linking.

When invoked with the --verbose flag, verificarlo provides a detailed output of the instrumentation process.

It is important to include the necessary link flags if you use extra libraries. For example, you should include -lm if you are linking against the math library.

Branch instrumentation

Verificarlo can instrument floating point comparison operations. By default, comparison operations are not instrumented and default backends do not make use of this feature. If your backend requires instrumenting floating-point comparisons, you must call verificarlo with the --inst-fcmp flag.

FMA instrumentation

Verificarlo can instrument Fused Multiply-Add (FMA) operations. By default, FMA operations are not instrumented and default backends do not make use of this feature. If your backend requires instrumenting FMA operations, you must call verificarlo with the --inst-fma flag.

Cast instrumentation

Verificarlo can also instrument cast operations. By default, cast operations are not instrumented and default backends do not make use of this feature. If your backend requires instrumenting cast operations, you must call verificarlo with the --inst-cast flag.

Examples and Tutorial

The tests/ directory contains various examples of Verificarlo usage.

A tutorial is available.

Backends

Verificarlo includes different numerical backends. Please refer to the backends' documentation.

Inclusion/exclusion options

To include only certain parts of the code in the analysis or exclude parts of the code from instrumentation please refer to inclusion/exclusion options documentation.

Pinpointing numerical errors with Delta-Debug

To pinpoint numerical errors please refer to the Delta-Debug documentation.

VPREC Function Instrumentation

A function instrumentation pass enables VPREC exploration and optimization at the function granularity level. Please refer to the VPREC Function Instrumentation documentation.

Postprocessing

Verificarlo includes a set of postprocessing tools to help analyze Verificarlo results and produce high-level reports.

Interflop user call instrumentation

Verificarlo provides the ability to call low-level backend functions directly through the interflop_call function. Please refer to the Interflop user call instrumentation documentation.

How to cite Verificarlo

If you use Verificarlo in your research, please cite one of the following papers:

Thanks! ✨

Discussion Group

For questions, feedback, or discussions about Verificarlo, you can use the Discussions section on our GitHub project page.

License

This file is part of the Verificarlo project,
under the Apache License v2.0 with LLVM Exceptions.
SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception.
See https://llvm.org/LICENSE.txt for license information.

Copyright (c) 2019-2024 Verificarlo Contributors

Copyright (c) 2018 Universite de Versailles St-Quentin-en-Yvelines

Copyright (c) 2015 Universite de Versailles St-Quentin-en-Yvelines CMLA, Ecole Normale Superieure de Cachan

verificarlo's People

Contributors

avaquet avatar chdenis avatar ddavid42 avatar gkiar avatar killianbabilotte avatar mistoan avatar ojamond avatar pablooliveira avatar purplepachyderm avatar redfoxp avatar roblim1 avatar scemama avatar sholde avatar titeup avatar vincentalba avatar yohanchatelain avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

verificarlo's Issues

UI: add --preset= to vprec backend

To make vprec more user friendly, we could add a simple --preset= option that loads pre-configured settings for precision and range. We could therefore support --preset=bfloat16, --preset=tensorflow32, --preset=binary16, --preset=fp24, etc.

There is a good overview of available formats here.

Unexpected behavior for VERIFICARLO_PRECISION=0

I open a ticket for the behavior of VERIFICARLO_PRECISION when is set to 0.
We expected to have all bits noised but it turns out that no noise is added.
When verbose mode is activated, verificarlo outputs :

VERIFICARLO_PRECISION invalid value provided, defaulting to default

ability to provide arguments in environment variable without spaces?

Hi!

I'm currently trying to use Verificarlo inside a Singularity container, but there is a bug in their stack that prevents me from providing environment variables on the command-line that contain spaces (see https://github.com/sylabs/singularity/issues/5108). While they work on this, I was wondering if there is any way to specify args for verificarlo that would be compatible with this restriction?

Ideally I'd be using options equivalent to libinterflop_mca.ieee --mode rr.

thanks!

verificarlo fails compiling rotg.c in openblas (fp80)

This seems related to fp80 support,

$ cd OpenBLAS/interface
$ verificarlo -c -O2 -DMAX_STACK_ALLOC=2048 -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DNO_LAPACK -DNO_LAPACKE -DNO_AVX512 -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=4 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8.dev\" -DASMNAME=srotg -DASMFNAME=srotg_ -DNAME=srotg_ -DCNAME=srotg -DCHAR_NAME=\"srotg_\" -DCHAR_CNAME=\"srotg\" -DNO_AFFINITY -I.. -I. -UDOUBLE  -UCOMPLEX rotg.c -o srotg.o

Call parameter type does not match function signature!
  %5 = tail call x86_fp80 @fabsl(x86_fp80 %2) #3
 float  %8 = call float %7(x86_fp80 %5, x86_fp80 %6)
Both operands to FCmp instruction are not of the same type!
  %10 = fcmp oeq float %8, x86_fp80 0xK00000000000000000000

test_backends strange behavior on RR mode with MPFR MCA backend

When running test_backend, which compares MPFR and QUAD backends with 50 random samples accross all basic FP operations, on RR mode at precision 23, the MPFR backend reports a +inf error.

Checking + float
clang: warning: -lm: 'linker' input unused
clang: warning: -lm: 'linker' input unused
Checking at PRECISION 3
ok for precision 3
Checking at PRECISION 13
ok for precision 13
Checking at PRECISION 23
ERROR: 24.3144 inf
ERROR: 24.3926 inf
ERROR: 23.8912 inf
ERROR: 24.4896 inf
MAX > THRESHOLD
error

vprec bug for negative exponent

With the last fix we've done in the vprec backend test (PR in progress), we can confirm that the weird behavior appearing in Allan experiment was coming from a bug in vprec. We need to investigate.
An example of this behavior appear in the following log (mpfr has the correct answer):
float: MODE=OB RANGE=2 PRECISION=3 OP=+
Relative error too high: a=0x1.ff10ddbd996a2p-7 b=0x1.137d3527080f0p-2
mpfr=0.25 vprec=0.3125 error=0.25 (2.0 b=2)

MPI support

Hello,
I prototyped an mpi support (no specific feature related to mpi operation, just compile the binaries with verificarlo) based on mpich script that seems to work pretty well on my HPC use case. This should be discuss during our next meeting to figure out the best way to integrate it (software and license).
I'll pass the current version to Yohan to test it with its app to have a broader validation.
For MPI operation instrumentation, the question remains open. Shall we use interposition? shall we instrument the mpi librairies? etc.

veritracer launch should stop with prefix-dir error

Example

$ rm -rf 1 10 2 3 4 5 6 7 8 9
$ veritracer launch --j 10 --binary=muller --prefix-dir=exp
[Errno 17] File exists: 'exp'
$ ls
1  10  2  3  4  5  6  7  8  9  backtrace.dat ...

I would expect that when veritracer launch fails with a prefix-dir error; no experiments are launched. Yet, trace directories 1 to 10 have been created (outside of prefix-dir).

Some troubles with a small example using the Boost C++ library

Hi,

First of all thank you for sharing Verificarlo, that is really a promising project!

I just successfully installed it using your debian package

I also run with success the tests (test_tchebychev, test_...) that come with the source.

I just tried something more complicated, a special function of the Boost C++ library, but I got some troubles...

I wanted to test the Boost digamma function

My small C++ code is

#include <boost/math/special_functions/digamma.hpp>
#include <iostream>

int
main()
{
  std::cout << "\n result " << boost::math::digamma(0.1);
}

I try to compile it with:

verificarlo oneExample.cpp -o oneExample

But I get the following error messages:

Unsupported operand type: x86_fp80
Unsupported operand type: x86_fp80
Unsupported operand type: x86_fp80
Unsupported operand type: x86_fp80
Unsupported operand type: x86_fp80
Unsupported operand type: x86_fp80
Unsupported operand type: x86_fp80
Unsupported operand type: x86_fp80
Call parameter type does not match function signature!
x86_fp80 0xK3FFF8000000000000000
 float  %11 = call float %10(x86_fp80 0xK3FFF8000000000000000, x86_fp80 %9)
Stored value type does not match pointer operand type!
  store float %11, x86_fp80* %2, align 16
 x86_fp80Call parameter type does not match function signature!
  %12 = load x86_fp80, x86_fp80* %2, align 16
 float  %16 = call float %15(x86_fp80 %12, x86_fp80 %14)
Stored value type does not match pointer operand type!
  store float %16, x86_fp80* %remainder, align 16
...
 x86_fp80LLVM ERROR: Broken function found, compilation aborted!
/usr/bin/verificarlo: command failed:
/usr/lib/llvm-3.7/bin/opt -S  -load /usr/lib/x86_64-linux-gnu/libvfcinstrument.so -vfclibinst   oneExample.1.ll -o oneExample.2.ll

My question, as a new Verificarlo user, is:
1/ Do I missed something (in my compile command etc...)?
or
2/ Verificarlo is not yet fully fonctional and I have to wait until it supports libs like Boost?
or
3/ Is it a Verificarlo bug?

  • Vincent

ps: I also posted this question in the google group but maybe here is a better place.

Newer LLVM support

Currently verificarlo support LLVM up to version 3.8.1.
LLVM current version is 6.0, and most modern linux distrubution use version 4.0+.
Do you think that verificarlo can be updated to support newer llvm ?

Thanks.

buggy test_vprec_backend

It was not testing negative exponent because of minor bugs, and it is a good opportunity to improve it.

I linked the corresponding PR.

test_cancellation is broken

The test returns ok but it should not. Actually, the directory OUTPUT does not exist so test.sh fails and the diff too. When I create the directory, the diff commands fails and I got an error.

LLVM 8?

I tried installing Verificarlo with LLVM 8.0.1 and 9.0, both compiled with source, but get the following error. See config_vfc_llvm8.log. Verificarlo with LLVM 4.0.1 (source) works for me.

PYTHONPATH is not set correctly for ddebug whch causes vfc_ddebug to fail

For vfc_debug to work, $PYTHONPATH needs to contain the path to the location where the verificarlo python packages are installed.
This is not set correctly during setup.
The automatic test 5 test_ddebug_integration fails, but isn't detected by the integration tests, as the path is set correctly in the docker image.
A temporary fix is to export manually the path to the environment.

ieee mode is not ieee...

Hello guys,

Testing on nas omp I realised that the so-called ieee mode is not giving the same result as the original program.
The probelem seems related to the fact that in the current version it is a mode of the mpfr backend.
Float are convered to mpfr type and noise is set to zero. and then the rounding happen to produce the final result. During the process, the correct iee rounding is lost.
A quasi trivial way to fix it would be to promote the MCA MODE IEEE to a backend.
i.e. instrument all the FP and the instrumentation function are just doing the original application.
The good point is that this will constitute a reference backend to test and validate the front-ends neutrality.
We discussed it on the phone with Pablo, and one of us will do it.

Performance of MCA backends: MPFR faster than QUAD

I compared the performance of MCA backends MPFR and QUAD and it seems that MPFR is faster than QUAD while I expected the oposite.
Here are the results of the comparison for the test_fortran_NAS, time ./test.sh:
libinterflop_ieee.so 1.7s
libinterflop_mca.so 23.7s
libinterflop_mca_mpfr.so 18.5s
So MPFR seems to be 30% faster.
I attached the test.gz

-- config --
Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz

Add verbose mode

During compilation, verificarlo outputs a lot of debug information. A --verbose flag should be added so this information is only provided when requested.

SGP4 bug by using verificarlo

I've tried to compile and use SGP4 (https://www.danrw.com/sgp4/) with verificarlo.
CMakeLists.txt
When i use verifiicarlo to compile the project with CMake, i have a lot of warning flags on an unused include path and then the compilation fail because the linking with the c++ standard librairy is not done.
I've used the folowing sccript to make the compilation possible (i don't know if this script is correct but the compilation is now ok, i print options to check that the inlcude path is correctly separate than other options).
verificarlo.txt
After that if i launch the runtest executable with the backend libinterflop_ieee.so results are corrupted by a nan and they are not usable.

Error with llvm-3.3 and llvm-3.4

Verificarlo crash during the execution with llvm-3.3 and llvm-3.4.
It seems that a function has been introduced since llvm-3.5 and breaks the execution.
I will provide a full report.

rewrite test_backends using Chi-2 test

Instead of using an arbitrary threshold when comparing sampling distributions from our two MCA implementations, we could instead do a Chi-2 test to check that they follow the same distributions.

Improve performance in verificarlo.in.in

It should be possible to improve slightly the performance of verificarlo's build by controlling better the optimizations flags during the library and wrapper's build.

  • In particular, we should ensure that backends are optimized (-O2 at least).
  • We could leverage LTO support in clang to inline vfc_calls from .vfcwrapper.c
  • Instead of using text .ll we could use bytecode .bc format which would speed up compilation

Bug in static mode

I tried to install verificarlo on my computer following the README.md instruction. After running the installcheck I've got one test failure on the "test_static_mode". The file test.log said :

".vfcwrapper.o : Dans la fonction « vfc_init » :
/usr/local/bin/../include/vfcwrapper.c:(.text+0x210): avertissement : Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
test: Cannot load backend libinterflop_ieee.so: dlopen error
libinterflop_ieee.so: cannot open shared object file: No such file or directory "

No support for vector operations longer than the archictecture's vector width

The backend for the double4 gives wrong results.
In the following minimal example test-vecto.zip, I add the value 2 to a vector of 0 for different vector sizes.
For the double4 type, we have arbitrary values.

VFC_BACKENDS="libinterflop_ieee.so --debug" ./a.out
Info [verificarlo]: loaded backend libinterflop_ieee.so
Info [interflop_ieee]: Decimal 2 + 0 -> 2
Info [interflop_ieee]: Decimal 2 + 0 -> 2
float2[0]=2.000000
float2[1]=2.000000

Info [interflop_ieee]: Decimal 2 + 0 -> 2
Info [interflop_ieee]: Decimal 2 + 0 -> 2
double2[0]=2.000000
double2[1]=2.000000

Info [interflop_ieee]: Decimal 2 + 0 -> 2
Info [interflop_ieee]: Decimal 2 + 0 -> 2
Info [interflop_ieee]: Decimal 2 + 0 -> 2
Info [interflop_ieee]: Decimal 2 + 0 -> 2
float4[0]=2.000000
float4[1]=2.000000
float4[2]=2.000000
float4[3]=2.000000

Info [interflop_ieee]: Decimal 6.95293e-310 + 2.08323e-317 -> 6.95293e-310
Info [interflop_ieee]: Decimal 0 + 6.95293e-310 -> 6.95293e-310
Info [interflop_ieee]: Decimal 0 + 6.95293e-310 -> 6.95293e-310
Info [interflop_ieee]: Decimal 6.90958e-310 + 2.07517e-317 -> 6.90958e-310
double4[0]=0.000000
double4[1]=0.000000
double4[2]=0.000000
double4[3]=0.000000

When I compile with clang the code is working.

float2[0]=2.000000
float2[1]=2.000000

double2[0]=2.000000
double2[1]=2.000000

float4[0]=2.000000
float4[1]=2.000000
float4[2]=2.000000
float4[3]=2.000000

double4[0]=2.000000
double4[1]=2.000000
double4[2]=2.000000
double4[3]=2.000000

I suspect the compiler to pass the vector by reference since it is to wide.
So we have a pointer instead of a value.

random rounding backend

Hello all,

I started a while ago (and almost finished) a random rounding backend to explore wether or not it can efficiently (perf and accuracy) replace the quad/double backend for standard precision noise and if the statistic are still converging to the same values.
I expect some diffrences, but when used in a debugging purpose, that should do the trick.
I am also working on the administrative aspect of releasing this contribution together with the rebuild quad/double backend taking effectively into ccount special values and denormal numbers.
I'll keep you posted.

log info in cancellation backend

Hello,
Just to warn you on the lack of "\n" at the end of the log info in interflop_cancellation.c at the line 209.
logger_info("interflop_cancellation: loaded backend with tolerance = %d \n", TOLERANCE);

Get Zenodo DOI?

It would be nice to have a citable DOI for each version of the code (alongside the pre-print/soon to be published paper), to more easily identify the specific build that is used in a given dependency. I noticed you just made a version, and Zenodo can automatically hook in creating a fresh DOI each time to you do this. Instructions pasted below, in case you're unfamiliar with this integration:

https://guides.github.com/activities/citable-code/

configure failure during checking can compile with and link with LLVM(all)...

Hello,

the configure test failed to compile the simple main test.
I used LLVM 4.0.1 compiled with gcc 4.8.5.
It seems that some flags are missing.

I corrected it changing line 104 of file m4/ax_llvm.m4
104 LDFLAGS="$LDFLAGS -stdlib=libstdc++ -pthread -ltinfo -ldl"

Nevertheless, I'm not sure this is the best way to correct this "bug".
Thanks for your work,
Hugo

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.