ctuning / ck-math Goto Github PK

Collective Knowledge packages for various mathematical libs to be plugged into portable and customizable CK research workflows:

Home Page: https://github.com/ctuning/ck/wiki/Portable-workflows

License: BSD 3-Clause "New" or "Revised" License

Batchfile 5.93% Shell 10.05% Roff 0.02% Python 21.68% Jupyter Notebook 24.07% C 4.37% C++ 33.74% Makefile 0.14%

ck-math's People

Contributors

Stargazers

Watchers

Forkers

kayuri dvefremov dsavenko zyzhou1028 maria-18-git winminecv dividiti arjunsuresh krai

ck-math's Issues

lib-lapack-3.4.2 on Odroid

$ ck install package:lib-lapack-3.4.2
The compilation ends fine.
However, I got the following error:

Setting up environment for installed package ...
  (full path = /home/flavio/CK_TOOLS/lapack-3.4.2-linux-32)

  Software entry found: lib.lapack (4069972dfce0f22d)
CK error: [package] software not found in a specified path (/home/flavio/CK_TOOLS/lapack-3.4.2-linux-32)!

$ ls /home/flavio/CK_TOOLS/lapack-3.4.2-linux-32

I found the lib as expected

cmake/ libblas.a liblapack.a libtmglib.a pkgconfig/

Installing package:lib-clblas-master-universal fails when libOpenCL.so is not in a standard path

Probably need to set OPENCL_LIBRARIES.

$ ck install package:lib-clblas-master-universal 
...
-- Boost version: 1.64.0
-- Found the following Boost libraries:
--   program_options
-- Boost_PROGRAM_OPTIONS_LIBRARY: /home/anton/CK_TOOLS/lib-boost-1.64.0-gcc-6.3.0-linux-64/install/lib/libboost_program_options.a
-- CMAKE_C_COMPILER_MACHINE: aarch64-linux-gnu
-- AutoGemm-PreCompile: no kernels to be pre-compiled.
-- OPENCL_VERSION = 1.2
-- clBLAS will NOT depend on /home/anton/CK_TOOLS/lib-clblas-master-gcc-6.3.0-linux-64/obj/include/AutoGemmKernelBinaries/AutoGemmKernelBinariesPreCompiled.h
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
OPENCL_LIBRARIES (ADVANCED)
    linked by target "AutoGemm_Tools_Profile" in directory /home/anton/CK_TOOLS/lib-clblas-master-gcc-6.3.0-linux-64/src/src/library
    linked by target "AutoGemm_Tools_Test" in directory /home/anton/CK_TOOLS/lib-clblas-master-gcc-6.3.0-linux-64/src/src/library
    linked by target "clBLAS" in directory /home/anton/CK_TOOLS/lib-clblas-master-gcc-6.3.0-linux-64/src/src/library
    linked by target "tune" in directory /home/anton/CK_TOOLS/lib-clblas-master-gcc-6.3.0-linux-64/src/src/library/tools/tune
    linked by target "make-ktest" in directory /home/anton/CK_TOOLS/lib-clblas-master-gcc-6.3.0-linux-64/src/src/library/tools/ktest

-- Configuring incomplete, errors occurred!
See also "/home/anton/CK_TOOLS/lib-clblas-master-gcc-6.3.0-linux-64/obj/CMakeFiles/CMakeOutput.log".
See also "/home/anton/CK_TOOLS/lib-clblas-master-gcc-6.3.0-linux-64/obj/CMakeFiles/CMakeError.log".
Error: configuring failed!

opencv-3.2.0 installation Odroid

$ ck install package:lib-opencv-3.2.0

Error:

Length: 82033498 (78M) [application/zip]
Saving to: '3.2.0.zip'

3.2.0.zip               12%[==>                        ]   9.52M  2.74MB/s    in 4.0s    


Cannot write to '3.2.0.zip' (Success).
Error: downloading package failed!
CK error: [package] package installation failed!```

Add dependency on Fortran compiler to OpenBLAS 0.2.18+

Building OpenBLAS failed on a machine with only gcc-6 and g++-6 installed, but not gfortran-6. Interestingly, the machine had gfortran-5 installed (aliased as gfortran) but OpenBLAS clearly wanted to use everything from the "GNU tools 6 series"... And so did CK, because the CK environment of the failed build had the following variables set:

CK_F90=gfortran-6
CK_F95=gfortran-6
CK_FC=gfortran-6

Installing gfortran-6 solved the issue. However, it's clear that OpenBLAS has a dependency on a Fortran compiler (perhaps even specifically gfortran) which should be specified explicitly in the package's metadata. Unfortunately, no Fortan compiler description is available in CK yet, so this issue will have to be stalled until that's resolved.

Should the number of OpenBLAS threads be limited by the number of big cores on big.LITTLE systems?

On Odroid-XU3 with 4 "big" Cortex-A15 cores and 4 "LITTLE" Cortex-A7 cores, OpenBLAS configures itself with the maximum number of threads equal to the total number of cores, i.e. 8 (-DMAX_CPU_NUMBER=8). I'd say it's probably better to configure OpenBLAS to limit the maximum threads to the number of big cores only. Any thoughts?

Support newer versions of OpenCV built from source

The versions of OpenCV we currently support are quite old:

$ ck search package --tags=lib,opencv,channel-stable
ck-math:package:lib-opencv-3.4.5
ck-math:package:lib-opencv-3.3.0
ck-math:package:lib-opencv-2.4.13.2
ck-math:package:lib-opencv-3.4.3
ck-math:package:lib-opencv-3.2.0

The latest releases are 3.4.10 and 4.3.0. (Intel used 4.1.2 in the MLPerf Inference v0.5 submission.)

Adding support for 3.4.10 should be quite easy:

Test you can build OpenCV 3.4.3 on your system:

$ ck install package --tags=lib,opencv,v3.4.3

Fork repo:ck-math to your GitHub account e.g. i_want_to_help.
Switch your local repo:ck-math to your new remote fork and create a branch there: e.g.

$ cd `ck find repo:ck-math`
$ git remote set-url origin [email protected]:i_want_to_help/ck-math
$ git checkout -b opencv-3.4.10

Create a copy of the OpenCV 3.4.3 package:

$ ck cp ck-math:package:lib-opencv-3.4.3 ck-math:package:lib-opencv-3.4.10

Modify the copy to replace "3.4.3" with "3.4.10" everywhere in meta.json.
Test you can build the new package:

$ ck install package --tags=lib,opencv,v3.4.10

Commit the change and push your branch to your remote.
Issue a Pull Request from your remote to https://github.com/ctuning/ck-math.
Repeat on another system (Windows, MacOS?)
Try the same with the latest OpenCV 4.x version.

Two suggestion

What do you think of:
https://github.com/hfp/libxsmm
https://github.com/01org/mkl-dnn

Update package:lib-clblast-master-universal and package:lib-clblast-master-universal-tune

Need to refresh the patches e.g.

Hunk #2 succeeded at 164 (offset 12 lines).                                                                                                   [0/24827]
patching file src/database/kernels/padtranspose.hpp
Hunk #1 succeeded at 69 (offset 6 lines).
Hunk #2 succeeded at 132 (offset 12 lines).
Hunk #3 succeeded at 163 (offset 12 lines).
Hunk #4 succeeded at 294 (offset 86 lines).
patching file src/database/kernels/transpose.hpp
Hunk #1 succeeded at 69 (offset 6 lines).
Hunk #2 succeeded at 133 (offset 12 lines).
Hunk #3 succeeded at 164 (offset 12 lines).
Hunk #4 succeeded at 221 (offset 18 lines).
patching file src/database/kernels/xaxpy.hpp
Hunk #1 succeeded at 69 (offset 6 lines).
Hunk #2 succeeded at 164 (offset 12 lines).
patching file src/database/kernels/xdot.hpp
Hunk #1 succeeded at 62 (offset 6 lines).
Hunk #2 succeeded at 139 (offset 12 lines).
patching file src/database/kernels/xgemm.hpp
Hunk #2 FAILED at 120.
Hunk #3 succeeded at 157 (offset 6 lines).
Hunk #4 succeeded at 214 (offset 6 lines).
1 out of 4 hunks FAILED -- saving rejects to file src/database/kernels/xgemm.hpp.rej
patching file src/database/kernels/xgemm_direct.hpp
Hunk #2 FAILED at 85.
Hunk #3 succeeded at 111 (offset 6 lines).
Hunk #4 succeeded at 143 (offset 6 lines).
1 out of 4 hunks FAILED -- saving rejects to file src/database/kernels/xgemm_direct.hpp.rej
patching file src/database/kernels/xgemv.hpp
Hunk #2 succeeded at 145 (offset 6 lines).
patching file src/database/kernels/xgemv_fast.hpp
Hunk #2 succeeded at 145 (offset 6 lines).
patching file src/database/kernels/xgemv_fast_rot.hpp
Hunk #2 succeeded at 96 (offset 6 lines).
Hunk #3 succeeded at 117 (offset 6 lines).
patching file src/database/kernels/xger.hpp
Hunk #1 succeeded at 69 (offset 6 lines).
Hunk #2 succeeded at 152 (offset 12 lines).
patching file src/routines/level3/xtrsm.cpp

Boost gets tag "compiled-by-unknown_host-compiler"

I have noticed that when Boost is installed as a Caffe dependency (and possibly standalone), in addition to compiled-by-gcc and and compiled-by-gcc-<version tags, it gets a weird compiled-by-unknown_host-compiler tag:

$ ck show env --tags=boost,lib
Env UID:         Target OS: Bits: Name:         Version:             Tags:

49673bc810e4d8e9   linux-64    64 Boost library 1.67.0-min-for-caffe 64bits,boost,channel-stable,compiled-by-gcc,compiled-by-gcc-8.1.0,compiled-by-unknown_host-compiler,host-os-linux-64,lib,min-for-caffe,target-os-linux-64,v1,v1.67,v1.67.0,v1.67.0-min-for-caffe,v1.67.0.0,v1.67.0.0.0,v1.67.0.0.0.0
002ceccabd454515   linux-64    64 Boost library 1.67.0-min-for-caffe 64bits,boost,channel-stable,compiled-by-llvm,compiled-by-llvm-5.0.0-3~16.04.1,compiled-by-unknown_host-compiler,host-os-linux-64,lib,min-for-caffe,target-os-linux-64,v1,v1.67,v1.67.0,v1.67.0-min-for-caffe,v1.67.0.0,v1.67.0.0.0,v1.67.0.0.0.0
c0aefcb48a2b39f7   linux-64    64 Boost library 1.67.0-min-for-caffe 64bits,boost,channel-stable,compiled-by-gcc,compiled-by-gcc-7.3.0,compiled-by-unknown_host-compiler,host-os-linux-64,lib,min-for-caffe,target-os-linux-64,v1,v1.67,v1.67.0,v1.67.0-min-for-caffe,v1.67.0.0,v1.67.0.0.0,v1.67.0.0.0.0
90bbc584d0260b58   linux-64    64 Boost library 1.66.0-min-for-caffe 64bits,boost,channel-stable,compiled-by-gcc,compiled-by-gcc-6.4.0,compiled-by-unknown_host-compiler,host-os-linux-64,lib,min-for-caffe,target-os-linux-64,v1,v1.66,v1.66.0,v1.66.0-min-for-caffe,v1.66.0.0,v1.66.0.0.0,v1.66.0.0.0.0
508a59286c69b492   linux-64    64 Boost library 1.66.0               64bits,boost,channel-stable,compiled-by-gcc,compiled-by-gcc-6.4.0,compiled-by-unknown_host-compiler,host-os-linux-64,lib,target-os-linux-64,v1,v1.66,v1.66.0

Add OpenBLAS 0.2.19 package

@xianyi released OpenBLAS 0.2.19 on 1-Sep-2016. Should add a new CK package.

Ideally, it should reuse the installation script from the OpenBLAS 0.2.18 package, rather than copy and modify it. (It has some AArch64 changes coming.) For an example, see CK-Caffe where the GPU versions (e.g. lib-caffe-bvlc-master-cuda) depend on the CPU version (lib-caffe-bvlc-master-cpu).

"ck install package:lib-boost-1.66.0" fails with ICC 18.0.1

$ ck detect soft:compiler.icc
$ ck install package:lib-boost-1.66.0
...
Cleaning ...

Executing extra script ...
Building Boost.Build engine with toolset gcc... tools/build/src/engine/bin.linuxx86_64/b2
Detecting Python version... 2.7
Detecting Python root... /usr
Unicode/ICU support for Boost.Regex?... not found.
Generating Boost.Build configuration in project-config.jam...

Bootstrapping is done. To build, run:

    ./b2
    
To adjust configuration, edit 'project-config.jam'.
Further information:

   - Command line help:
     ./b2 --help
     
   - Getting started guide: 
     http://www.boost.org/more/getting_started/unix-variants.html
     
   - Boost.Build documentation:
     http://www.boost.org/build/doc/html/index.html


Building Boost (can take a long time) ...
/home/anton/CK_REPOS/ck-math/package/lib-boost-1.62.0/scripts.linux/install.sh: line 32: /home/anton/CK_TOOLS/lib-boost-1.66.0-icc-18.0.1-linux-64/install/user-config.jam: No such file or directory
icc.jam: No such file or directory
/home/anton/CK_TOOLS/lib-boost-1.66.0-icc-18.0.1-linux-64/boost_1_66_0/tools/build/src/build/toolset.jam:43: in toolset.using
ERROR: rule "icc.init" unknown in module "toolset".
/home/anton/CK_TOOLS/lib-boost-1.66.0-icc-18.0.1-linux-64/boost_1_66_0/tools/build/src/build-system.jam:461: in process-explicit-toolset-requests
/home/anton/CK_TOOLS/lib-boost-1.66.0-icc-18.0.1-linux-64/boost_1_66_0/tools/build/src/build-system.jam:527: in load
/home/anton/CK_TOOLS/lib-boost-1.66.0-icc-18.0.1-linux-64/boost_1_66_0/tools/build/src/kernel/modules.jam:295: in import
/home/anton/CK_TOOLS/lib-boost-1.66.0-icc-18.0.1-linux-64/boost_1_66_0/tools/build/src/kernel/bootstrap.jam:139: in boost-build
/home/anton/CK_TOOLS/lib-boost-1.66.0-icc-18.0.1-linux-64/boost_1_66_0/boost-build.jam:17: in module scope
Error: b2 make failed!
CK error: [package] package installation failed!

Create package lib-openblas-0.2.20-universal

https://github.com/xianyi/OpenBLAS/releases/tag/v0.2.20

Program armcl-classification-mobilenet fails to predict when some versions of ArmCL is used

When armcl-classification-mobilenet is compiled against ArmCL 18.05 it can predict images succesfully. For example for option 0) image-jpeg-0001 (1aaaa23c44e588f9) it gives

      0.9341 - "oxygen mask (692)"
      0.0264 - "book jacket (922)"
      0.0175 - "harmonica (594)"
      0.0030 - "loupe (634)"
      0.0029 - "beaker (439)"

But when compiled against 18.01 or 18.08 it gives zero predictions for the same image:

      0.0000 - "toilet tissue (1000)"
      0.0000 - "ear (999)"
      0.0000 - "bolete (998)"
      0.0000 - "hen-of-the-woods (997)"
      0.0000 - "earthstar (996)"

Seems it is not because of version ArmCL but something is wrong with the program itself. Another program ck-request-asplos18-mobilenets-armcl-opencl:program:mobilenets-armcl-opencl gives the same results being compiled against all those versions of ArmCL.

Broken link for lib-eigen-src-3.3.4

The package lib-eigen-src-3.3.4 cannot be downloaded with ck because the link in the meta.cm, "PACKAGE_URL": "http://bitbucket.org/eigen/eigen/get" seems to no longer be active.

Create package:lib-clblast-1.0.1

https://github.com/CNugteren/CLBlast/releases/tag/1.0.1

Installing package:lib-clblast-master-universal-tune fails

Patching source directory ...
/home/anton/CK_REPOS/ck-math/package/lib-clblast-master-universal-tune/patch.linux/patch-firefly
can't find file to patch at input line 5
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/src/database/kernels/xgemm.hpp b/src/database/kernels/xgemm.hpp
|index 8246b91..740f7c7 100644
|--- a/src/database/kernels/xgemm.hpp
|+++ b/src/database/kernels/xgemm.hpp
--------------------------
File to patch:

A workaround is to overwrite the patch file with nothing:

$ echo "" > /home/anton/CK_REPOS/ck-math/package/lib-clblast-master-universal-tune/patch.linux/patch-firefly

(Can't remove it completely as otherwise the install script fails. Perhaps removing the whole patch.linux is called for.)

Create package:lib-clblas-2.12

https://github.com/clMathLibraries/clBLAS/releases/tag/v2.12

boost install.sh fails to write user-config.jam

When building lib-boost-1.64.0 for host on Ubuntu 16.04 I get the following issue reported:
Building Boost (can take a long time) ...
/home/stutay01/CK/ck-math/package/lib-boost-1.64.0/scripts.linux/install.sh: line 34: /HDD/Benchmarking/CK/local/env/06453390c34751aa/install/user-config.jam: No such file or directory

Looks like it fails to write the user-config.jam file. The install.sh script looks like it needs some directory creation prior to writing the script.

Same applies for the other platform install scripts.

I have a program which requires boost to be built with -fPIC and at the moment this flag is not set due to the write failure. Let me know if you require any more information

Makefile improvements for lib-acl-* packages

Hi,

lib-acl-* packages use Makefiles and custom python/bash/batch scripts. This may be further improved so, that there will be less Python/bash/batch code and more Makefiles code (ideally, no custom Python). What can be done:

Find files to compile (*.cpp) in Makefile (probably, using $(wildcard ..)). This includes compiling different sets of sources depending on compilation options like USE_NEON and USE_OPENCL (including special handling of kernels' *.cl files for OpenCL).
Either copy the library headers to install/include from Makefile, or don't copy them at all and instead point *_INCLUDE variable(s) into the respective directories inside of src. For now, headers are copied in bash/batch script.
Compile the main lib and core lib in one run of Make, instead of two runs.
Once the above is done, we may get rid of bash/batch scripts completely, and just call Makefile with necessary parameters.
Make lib-acl-opencl-* package a simple alias for lib-acl-* package with different set of compilation parameters. Currently, we can't do this because of 1 and 2.