oap-project / oap-mllib Goto Github PK

View Code? Open in Web Editor NEW

20.0 5.0 12.0 32.67 MB

Optimized Spark package to accelerate machine learning algorithms in Apache Spark MLlib.

License: Apache License 2.0

C++ 23.82% Java 20.06% Scala 52.47% Makefile 0.35% Shell 3.28% C 0.03%

oap-mllib's Issues

[Scalability] Scalability test for large number of nodes

Migrate CI from Travis to GitHub Actions

Due to the license change of Travis, we would like to migrate CI to GitHub Actions and remove Travis

[PIP] Use "spark.plugins" to plug in oap-mllib into MLlib

[PIP] Enable MKL as default linear algebra lib when fall back

Branch "branch-1.1-spark-3.x" build failure

[Release] No performance benefit when using Intel-MLlib to run ALS algorithm.

I run ALS algorithm with Intel-MLlib and no performance benefit obtained. The configuration of ALS.conf is showed below:

And the configuration of spark is showed below:

[PIP] Update to oneAPI (incl. oneCCL) to 2021.2 and Rework examples for validation

Update to oneAPI (incl. oneCCL) to 2021.2
Remove hibench examples and support Scala & PySpark examples only
Use conf/env to setup envs
Reorg CI stuffs in dev/
Enable cluster testing for CI
Change to clang++ for native code building from gcc
Update README

[Optimization] Optimize memory footprint for data conversion

Porting Kmeans and PCA to new oneCCL API

Latest oneCCL API has been changed since beta version, need to port to the new API.
The first two algos will be Kmeans and PCA

Optimize oneCCL port detecting

[Release] Error when installing intel-oneapi-dal-devel-2021.1.1 intel-oneapi-tbb-devel-2021.1.1

#  sh dev/install-build-deps-centos.sh
Installing oneAPI components ...
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: ftp.sjtu.edu.cn
 * epel: epel.mirror.angkasa.id
 * extras: ftp.sjtu.edu.cn
 * updates: mirror.lzu.edu.cn
oneAPI/signature                                                                                                                                                                                                                                                |  287 B  00:00:00
oneAPI/signature                                                                                                                                                                                                                                                | 1.5 kB  00:00:00 !!!
Resolving Dependencies
--> Running transaction check
---> Package intel-oneapi-dal-devel.x86_64 0:2021.1.1-79 will be installed
---> Package intel-oneapi-dal-devel-2021.1.1.x86_64 0:2021.1.1-79 will be installed
--> Processing Dependency: intel-oneapi-dal-2021.1.1 for package: intel-oneapi-dal-devel-2021.1.1-2021.1.1-79.x86_64
--> Processing Dependency: intel-oneapi-common-licensing for package: intel-oneapi-dal-devel-2021.1.1-2021.1.1-79.x86_64
--> Processing Dependency: intel-oneapi-common-vars for package: intel-oneapi-dal-devel-2021.1.1-2021.1.1-79.x86_64
--> Processing Dependency: intel-oneapi-condaindex for package: intel-oneapi-dal-devel-2021.1.1-2021.1.1-79.x86_64
--> Processing Dependency: intel-oneapi-dal-common-devel-2021.1.1 for package: intel-oneapi-dal-devel-2021.1.1-2021.1.1-79.x86_64
---> Package intel-oneapi-tbb-devel.x86_64 0:2021.1.1-119 will be installed
---> Package intel-oneapi-tbb-devel-2021.1.1.x86_64 0:2021.1.1-119 will be installed
--> Processing Dependency: intel-oneapi-tbb-common-devel-2021.1.1 for package: intel-oneapi-tbb-devel-2021.1.1-2021.1.1-119.x86_64
--> Processing Dependency: intel-oneapi-tbb-2021.1.1 for package: intel-oneapi-tbb-devel-2021.1.1-2021.1.1-119.x86_64
--> Running transaction check
---> Package intel-oneapi-common-licensing-2021.1.1.noarch 0:2021.1.1-60 will be installed
---> Package intel-oneapi-common-vars.noarch 0:2021.2.0-195 will be installed
---> Package intel-oneapi-condaindex.x86_64 0:2021.2.0-94 will be installed
---> Package intel-oneapi-dal-2021.1.1.x86_64 0:2021.1.1-79 will be installed
--> Processing Dependency: intel-oneapi-compiler-dpcpp-cpp-runtime for package: intel-oneapi-dal-2021.1.1-2021.1.1-79.x86_64
--> Processing Dependency: intel-oneapi-dal-common-2021.1.1 for package: intel-oneapi-dal-2021.1.1-2021.1.1-79.x86_64
---> Package intel-oneapi-dal-common-devel-2021.1.1.noarch 0:2021.1.1-79 will be installed
---> Package intel-oneapi-tbb-2021.1.1.x86_64 0:2021.1.1-119 will be installed
--> Processing Dependency: intel-oneapi-tbb-common-2021.1.1 for package: intel-oneapi-tbb-2021.1.1-2021.1.1-119.x86_64
---> Package intel-oneapi-tbb-common-devel-2021.1.1.noarch 0:2021.1.1-119 will be installed
--> Running transaction check
---> Package intel-oneapi-compiler-dpcpp-cpp-runtime.x86_64 0:2021.2.0-610 will be installed
--> Processing Dependency: intel-oneapi-compiler-dpcpp-cpp-runtime-2021.2.0 for package: intel-oneapi-compiler-dpcpp-cpp-runtime-2021.2.0-610.x86_64
---> Package intel-oneapi-dal-common-2021.1.1.noarch 0:2021.1.1-79 will be installed
---> Package intel-oneapi-tbb-common-2021.1.1.noarch 0:2021.1.1-119 will be installed
--> Running transaction check
---> Package intel-oneapi-compiler-dpcpp-cpp-runtime-2021.2.0.x86_64 0:2021.2.0-610 will be installed
--> Processing Dependency: intel-oneapi-compiler-shared-runtime-2021.2.0 for package: intel-oneapi-compiler-dpcpp-cpp-runtime-2021.2.0-2021.2.0-610.x86_64
--> Processing Dependency: intel-oneapi-common-licensing-2021.2.0 for package: intel-oneapi-compiler-dpcpp-cpp-runtime-2021.2.0-2021.2.0-610.x86_64
--> Processing Dependency: intel-oneapi-tbb-2021.2.0 for package: intel-oneapi-compiler-dpcpp-cpp-runtime-2021.2.0-2021.2.0-610.x86_64
--> Running transaction check
---> Package intel-oneapi-common-licensing-2021.2.0.noarch 0:2021.2.0-195 will be installed
---> Package intel-oneapi-compiler-shared-runtime-2021.2.0.x86_64 0:2021.2.0-610 will be installed
--> Processing Dependency: intel-oneapi-compiler-shared-common-runtime-2021.2.0 for package: intel-oneapi-compiler-shared-runtime-2021.2.0-2021.2.0-610.x86_64
--> Processing Dependency: intel-oneapi-openmp-2021.2.0 for package: intel-oneapi-compiler-shared-runtime-2021.2.0-2021.2.0-610.x86_64
---> Package intel-oneapi-tbb-2021.2.0.x86_64 0:2021.2.0-357 will be installed
--> Processing Dependency: intel-oneapi-tbb-common-2021.2.0 for package: intel-oneapi-tbb-2021.2.0-2021.2.0-357.x86_64
--> Running transaction check
---> Package intel-oneapi-compiler-shared-common-runtime-2021.2.0.noarch 0:2021.2.0-610 will be installed
---> Package intel-oneapi-openmp-2021.2.0.x86_64 0:2021.2.0-610 will be installed
--> Processing Dependency: intel-oneapi-openmp-common-2021.2.0 for package: intel-oneapi-openmp-2021.2.0-2021.2.0-610.x86_64
---> Package intel-oneapi-tbb-common-2021.2.0.noarch 0:2021.2.0-357 will be installed
--> Running transaction check
---> Package intel-oneapi-openmp-common-2021.2.0.noarch 0:2021.2.0-610 will be installed
--> Processing Conflict: intel-oneapi-compiler-shared-common-runtime-2021.2.0-2021.2.0-610.noarch conflicts intel-oneapi-common-licensing < 2021.2.0
--> Processing Conflict: intel-oneapi-openmp-common-2021.2.0-2021.2.0-610.noarch conflicts intel-oneapi-common-licensing < 2021.2.0
--> Processing Conflict: intel-oneapi-compiler-dpcpp-cpp-runtime-2021.2.0-2021.2.0-610.x86_64 conflicts intel-oneapi-common-licensing < 2021.2.0
--> Processing Conflict: intel-oneapi-compiler-dpcpp-cpp-runtime-2021.2.0-2021.2.0-610.x86_64 conflicts intel-oneapi-tbb < 2021.2.0
--> Processing Conflict: intel-oneapi-tbb-2021.2.0-2021.2.0-357.x86_64 conflicts intel-oneapi-common-licensing < 2021.2.0
--> Processing Conflict: intel-oneapi-tbb-2021.2.0-2021.2.0-357.x86_64 conflicts intel-oneapi-tbb-common < 2021.2.0
--> Processing Conflict: intel-oneapi-openmp-2021.2.0-2021.2.0-610.x86_64 conflicts intel-oneapi-common-licensing < 2021.2.0
--> Processing Conflict: intel-oneapi-common-vars-2021.2.0-195.noarch conflicts intel-oneapi-common-licensing < 2021.2.0
--> Processing Conflict: intel-oneapi-compiler-shared-runtime-2021.2.0-2021.2.0-610.x86_64 conflicts intel-oneapi-common-licensing < 2021.2.0
--> Processing Conflict: intel-oneapi-tbb-common-2021.2.0-2021.2.0-357.noarch conflicts intel-oneapi-common-licensing < 2021.2.0
--> Finished Dependency Resolution
Error: intel-oneapi-compiler-shared-common-runtime-2021.2.0 conflicts with intel-oneapi-common-licensing-2021.1.1-2021.1.1-60.noarch
Error: intel-oneapi-openmp-common-2021.2.0 conflicts with intel-oneapi-common-licensing-2021.1.1-2021.1.1-60.noarch
Error: intel-oneapi-compiler-dpcpp-cpp-runtime-2021.2.0 conflicts with intel-oneapi-tbb-2021.1.1-2021.1.1-119.x86_64
Error: intel-oneapi-common-vars conflicts with intel-oneapi-common-licensing-2021.1.1-2021.1.1-60.noarch
Error: intel-oneapi-openmp-2021.2.0 conflicts with intel-oneapi-common-licensing-2021.1.1-2021.1.1-60.noarch
Error: intel-oneapi-compiler-dpcpp-cpp-runtime-2021.2.0 conflicts with intel-oneapi-common-licensing-2021.1.1-2021.1.1-60.noarch
Error: intel-oneapi-compiler-shared-runtime-2021.2.0 conflicts with intel-oneapi-common-licensing-2021.1.1-2021.1.1-60.noarch
Error: intel-oneapi-tbb-2021.2.0 conflicts with intel-oneapi-tbb-common-2021.1.1-2021.1.1-119.noarch
Error: intel-oneapi-tbb-2021.2.0 conflicts with intel-oneapi-common-licensing-2021.1.1-2021.1.1-60.noarch
Error: intel-oneapi-tbb-common-2021.2.0 conflicts with intel-oneapi-common-licensing-2021.1.1-2021.1.1-60.noarch
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

[Release] Merge #47 and prepare for OAP 1.1

Merge #47 and prepare for OAP 1.1

[ALS] Use user ID and item ID instead of matrix indices for ALS

Current ALS data input is using row index and column index to index rating in rating matrix. In real use cases, those row index and column index are usually using user ID and item ID and are represented with Integer or String. For example:

Use string as id

“User1”, “Item1”, 1.0
“User2”, “Item2”, 2.0

Or use integer as id

1234, 4567, 1.0
4321, 5678, 2.0

=> Row / column index for oneDAL

0, 1, 1.0
1, 1, 2.0

Other framework such as Spark MLlib ALS will handle this string/integer ID out of box. We need to do an extra data step to map from UserID/ItemID to row index/column index before calling DAL ALS and map back.

Also refer to oneapi-src/oneDAL#1514

[Core] NumericTable can't exceed 2G bytes

Should remove this restriction for oneDAL Java API.

Use getifaddrs to get host ips for oneCCL kvs

Use getifaddrs instead of "hostname -I"

[Doc]Update the documents for OAP 1.1.1

Using Intel-MLlib to run ALS algorithm sometimes meets hanging issue.

The same issue like #46. The frequency of this issue is not that high.

Fix third-party-programs-oneCCL.txt

@HongW2019 could you help to fix third-party-programs-oneCCL.txt according to latest one from oneCCL repo?

[Core] Use Barrier Execution Mode to schedule oneCCL ranks together

Barrier Execution Mode is only supported without dynamic allocation

[PIP] Refactor oneCCL and add an abstraction layer for collective operations

Implement Gather using oneccl all2all

[Core] Use std::shared_ptr instead of raw pointer

Add ALS with new oneCCL APIs

Fix oneCCL KVS port auto detect and improve logging

Fix oneCCL KVS port auto detect
Improve logging

Reorganize the source code for the new repository organization

We split the original single repository OAP project to a more flexible multi-repositories. OAP project becomes the top level organization at github.

This is the issue to track the work for reorganizing the source code structure.

Update PULL_REQUEST_TEMPLATE to simpler form

[Doc] Add explanation for spark.shuffle.reduceLocality.enabled

To close #70

[Tests] Add NaiveBayes test and refactors

Modify title check on PRs and automatic link to Issues below PRs

Modify title check on PRs and automatic link to Issues below PRs on Github Actions

[Coding Style] Add code style rules & scripts for Scala, Java and C++

Cannot compile Intel-mllib of master branch.

We try to compile Intel-mllib of master branch and meet the error like:

The maven command is "mvn clean package -DskipTests -Pspark-3.1.1" and we've successfully installed oneapi and oneccl.

[Release] Meet hang issue when running PCA algorithm.

In our automatic tests for OAP product, we found that using Intel-MLlib to run PCA and Kmeans algorithms often meet hanging issue which lead to block the whole workflow. The phenomenon is shown in the picture below：

[PIP] Misc improvements and refactor code

CI:

Fix CI tests bugs

Code Style:

Fix lint_scala bug
Apply styles for java, scala and c++
Add license header
rename all fit functions to train

Kmeans:

improve data conversion and cache & unpersist
refactor code for each profile

ALS:

Remove redundent code

OneCCL:

improve error message output

[Core] Support oneAPI 2021.3

[NaiveBayes] Fix isOAPEnabled and add multi-version support

Fix isOAPEnabled, enable by default
Add multi-version support

And we will use build-helper-maven-plugin to add additional version specific source code to build.
Additionally, as part of this refactor, change maven-scala-plugin (old) to scala-maven-plugin (new)

[Release] Revert to old oneCCL and Prepare for OAP 1.1

[PIP] Support Spark 3.0.1 / 3.0.2 and upcoming 3.1.1

Currently we are only supporting Spark 3.0.0. Need to validate for new versions.

3.0.1
3.0.2
3.1.1

https://spark.apache.org/news/index.html

Auto detect KVS port for oneCCL to avoid port conflict

Organize documentation of different versions in website with Github Pages and Mkdocs

Put documents in ${repo}/docs
create gh-pages, and use Github Pages and Mkdocs to build different versions

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.