Giter VIP home page Giter VIP logo

hpdbscan's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

hpdbscan's Issues

segfault in sort_by_cell()

Hello!

Running HPDBSCAN results in segmentation fault.

Dataset: SCADI.
Converted to h5f: https://cloud.mail.ru/public/e7ux/HykrvAgmM

Platform LSF log with HPDBSCAN stdout:

Sender: LSF System <[email protected]>
...
Started at Sat Jul 10 21:30:59 2021
Terminated at Sat Jul 10 21:31:04 2021
Results reported at Sat Jul 10 21:31:04 2021

Your job looked like:

------------------------------------------------------------
# LSBATCH: User input
# this file was automaticly created by mpisubmit.pl script for edu-cmc-sqi20-05 #
#BSUB -n 1
#BSUB -W 00:15
#BSUB -o hpdbscan.%J.out
#BSUB -e hpdbscan.%J.err
OMP_NUM_THREADS=1 mpiexec ./hpdbscan -t 1 -i /home_edu/edu-cmc-sqi20/edu-cmc-sqi20-05/my/ClusterLog_HPDBSCAN/ClusterLog/test/scadi.h5 --input-dataset DBSCAN -o data.h5 --output-dataset CLUSTERS -e 1.3 -m 2

------------------------------------------------------------

Exited with exit code 139.

Resource usage summary:

    CPU time :                                   1.32 sec.
    Max Memory :                                 119 MB
    Average Memory :                             79.67 MB
    Total Requested Memory :                     -
    Delta Memory :                               -
    Max Swap :                                   -
    Max Processes :                              5
    Max Threads :                                13
    Run time :                                   9 sec.
    Turnaround time :                            5 sec.

The output (if any) follows:

Computing cell space
        Computing dimensions...[OK] in 0.000053
        Computing cells...     [OK] in 0.000047
        Sorting points...

PS:

Read file <hpdbscan.765884.err> for stderr output of this job.

Platfrom LSF error log with HPDBSCAN staderr:

[polus-c4-ib:94545:0:94545] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x3fab4042367150cd)
==== backtrace ====
 0 0x00000000000452c4 ucs_debug_cleanup()  ???:0
 1 0x0000000000013f30 GOMP_parallel()  ???:0
 2 0x00000000100554a0 SpatialIndex<double>::sort_by_cell()  /polusfs/home_edu/edu-cmc-sqi20/edu-cmc-sqi20-05/my/py/highly/gcc9/hpdbscan/include/spatial_index.h:200
 3 0x0000000010072224 SpatialIndex<double>::SpatialIndex()  /polusfs/home_edu/edu-cmc-sqi20/edu-cmc-sqi20-05/my/py/highly/gcc9/hpdbscan/include/spatial_index.h:593
 4 0x0000000010073498 HPDBSCAN::cluster<double>()  /polusfs/home_edu/edu-cmc-sqi20/edu-cmc-sqi20-05/my/py/highly/gcc9/hpdbscan/include/hpdbscan.h:373
 5 0x0000000010073ef0 HPDBSCAN::cluster()  /polusfs/home_edu/edu-cmc-sqi20/edu-cmc-sqi20-05/my/py/highly/gcc9/hpdbscan/include/hpdbscan.h:349
 6 0x000000001000719c main()  /polusfs/home_edu/edu-cmc-sqi20/edu-cmc-sqi20-05/my/py/highly/gcc9/hpdbscan/src/cli.cpp:94
 7 0x0000000000024980 generic_start_main.isra.0()  libc-start.c:0
 8 0x0000000000024b74 __libc_start_main()  ???:0
===================
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 0 on node polus-c4-ib exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

GDB core dump:

GNU gdb (GDB) Red Hat Enterprise Linux 8.3-3.el7
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "ppc64le-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./hpdbscan...
[New LWP 94545]
[New LWP 94550]
[New LWP 94546]
[New LWP 94547]

warning: .dynamic section for "/lib64/libsz.so.2" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/lib64/libstdc++.so.6" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/lib64/libgomp.so.1" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/lib64/libgcc_s.so.1" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/lib64/libaec.so.0" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/usr/lib64/libnuma.so.1" is not at the expected address (wrong library or version mismatch?)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `./hpdbscan -t 1 -i /home_edu/edu-cmc-sqi20/edu-cmc-sqi20-05/my/ClusterLog_HPDBS'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000010050974 in std::_Hashtable<unsigned long, std::pair<unsigned long const, std::atomic<unsigned long> >, std::allocator<std::pair<unsigned long const, std::atomic<un
signed long> > >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::
__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_find_before_node (__code=0, __k=@0x3ffffc13dbe8: 0, __n=0, this=0x3ffffc13dd28)
    at /opt/rh/devtoolset-9/root/usr/include/c++/9/bits/hashtable.h:1532
1532        _Hashtable<_Key, _Value, _Alloc, _ExtractKey, _Equal,
[Current thread is 1 (Thread 0x100000049500 (LWP 94545))]
Missing separate debuginfos, use: debuginfo-install glibc-2.17-196.el7_4.2.ppc64le hdf5-1.8.12-12.el7.ppc64le libaec-1.0.2-1.el7.ppc64le libgcc-4.8.5-44.el7.ppc64le libgomp-4.8.5
-44.el7.ppc64le libibcm-41mlnx1-OFED.4.1.0.1.0.43101.ppc64le libibverbs-41mlnx1-OFED.4.3.0.1.8.43101.ppc64le libmlx4-41mlnx1-OFED.4.1.0.1.0.43101.ppc64le libmlx5-41mlnx1-OFED.4.3
.0.2.1.43101.ppc64le libnl3-3.2.28-4.el7.ppc64le librdmacm-41mlnx1-OFED.4.2.0.1.3.43101.ppc64le librxe-41mlnx1-OFED.4.1.0.1.7.43101.ppc64le libstdc++-4.8.5-44.el7.ppc64le numactl
-libs-2.0.9-7.el7.ppc64le ucx-1.3.0-1.43101.ppc64le zlib-1.2.7-17.el7.ppc64le

bt:

(gdb) bt
#0  0x0000000010050974 in std::_Hashtable<unsigned long, std::pair<unsigned long const, std::atomic<unsigned long> >, std::allocator<std::pair<unsigned long const, std::atomic<unsigned long> > >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_find_before_node (__code=0, __k=@0x3ffffc13dbe8: 0, __n=0, this=0x3ffffc13dd28)
    at /opt/rh/devtoolset-9/root/usr/include/c++/9/bits/hashtable.h:1532
#1  std::_Hashtable<unsigned long, std::pair<unsigned long const, std::atomic<unsigned long> >, std::allocator<std::pair<unsigned long const, std::atomic<unsigned long> > >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_find_node (__c=0, __key=@0x3ffffc13dbe8: 0, __bkt=0, this=0x3ffffc13dd28)
    at /opt/rh/devtoolset-9/root/usr/include/c++/9/bits/hashtable.h:655
#2  std::__detail::_Map_base<unsigned long, std::pair<unsigned long const, std::atomic<unsigned long> >, std::allocator<std::pair<unsigned long const, std::atomic<unsigned long> > >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator[] (this=0x3ffffc13dd28, __k=@0x3ffffc13dbe8: 0)
    at /opt/rh/devtoolset-9/root/usr/include/c++/9/bits/hashtable_policy.h:697
#3  0x0000000010055154 in std::unordered_map<unsigned long, std::atomic<unsigned long>, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::atomic<unsigned long> > > >::operator[] (__k=<optimized out>, this=<optimized out>) at /opt/rh/devtoolset-9/root/usr/include/c++/9/bits/unordered_map.h:984
#4  SpatialIndex<double>::_ZN12SpatialIndexIdE12sort_by_cellEv._omp_fn.0(void) ()
    at /polusfs/home_edu/edu-cmc-sqi20/edu-cmc-sqi20-05/my/py/highly/gcc9/hpdbscan/include/spatial_index.h:204
#5  0x0000100000853f30 in GOMP_parallel () from /lib64/libgomp.so.1
#6  0x00000000100554a0 in SpatialIndex<double>::sort_by_cell (this=0x3ffffc13e000)
    at /polusfs/home_edu/edu-cmc-sqi20/edu-cmc-sqi20-05/my/py/highly/gcc9/hpdbscan/include/spatial_index.h:195
#7  0x0000000010072224 in SpatialIndex<double>::SpatialIndex (this=0x3ffffc13e000, data=..., epsilon=<optimized out>) at /opt/rh/devtoolset-9/root/usr/include/c++/9/ostream:621
#8  0x0000000010073498 in HPDBSCAN::cluster<double> (this=0x3ffffc13e4d0, dataset=..., threads=<optimized out>)
    at /opt/rh/devtoolset-9/root/usr/include/c++/9/bits/basic_ios.h:390
#9  0x0000000010073ef0 in HPDBSCAN::cluster (this=0x3ffffc13e4d0, path=..., dataset=..., threads=<optimized out>)
    at /polusfs/home_edu/edu-cmc-sqi20/edu-cmc-sqi20-05/my/py/highly/gcc9/hpdbscan/include/hpdbscan.h:349
#10 0x000000001000719c in main (argc=<optimized out>, argv=<optimized out>) at /polusfs/home_edu/edu-cmc-sqi20/edu-cmc-sqi20-05/my/py/highly/gcc9/hpdbscan/include/cxxopts.h:838

Tools:

$ mpic++ --version
g++ (GCC) 9.3.1 20200408 (Red Hat 9.3.1-2)

$ mpiexec --version
mpiexec (OpenRTE) 4.0.0

Memory issue

Great work, thanks.

I have an issue about the memory.
In the example, the features have small dimension, eg, 3.
When I try larger dimension, I will face out of memory issue.
e.g., I tried
data = np.random.rand(100, 300)
it takes more than 100G memory.

Do you have any idea about this issue ?
Thanks.

python integration problem

Hello,

I am trying to install your code but I encounter the following problem:


mkdir build && cd build && cmake .. && make
-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Checking for module 'mpi-c'
--   No package 'mpi-c' found
-- Could NOT find MPI_C (missing: MPI_C_LIB_NAMES MPI_C_HEADER_DIR MPI_C_WORKS) 
-- Checking for module 'mpi-cxx'
--   No package 'mpi-cxx' found
-- Could NOT find MPI_CXX (missing: MPI_CXX_LIB_NAMES MPI_CXX_HEADER_DIR MPI_CXX_WORKS) 
-- Could NOT find MPI (missing: MPI_C_FOUND MPI_CXX_FOUND) 
BUILDING WITHOUT MPI SUPPORT
-- Found OpenMP_C: -fopenmp (found suitable version "4.5", minimum required is "4.0") 
-- Found OpenMP_CXX: -fopenmp (found suitable version "4.5", minimum required is "4.0") 
-- Found OpenMP: TRUE (found suitable version "4.5", minimum required is "4.0")  
-- HDF5 C compiler wrapper is unable to compile a minimal HDF5 program.
-- Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5.so (found suitable version "1.10.7", minimum required is "1.8.10")  
-- Found SWIG: /usr/bin/swig4.0 (found version "4.0.2")  
-- Found PythonInterp: /home/fbenedet/my_env/bin/python (found version "3.10.12") 
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.10.so (found version "3.10.12") 
CMake Warning (dev) at /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:438 (message):
  The package name passed to `find_package_handle_standard_args` (NUMPY) does
  not match the name of the calling package (NumPy).  This can lead to
  problems in calling code that expects `find_package` result variables
  (e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
  cmake/FindNumPy.cmake:28 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:61 (FIND_PACKAGE)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found NUMPY: /home/fbenedet/my_env/lib/python3.10/site-packages/numpy/core/include  
-- Configuring done
-- Generating done
-- Build files have been written to: /media/fbenedet/My Passport/dbscan/hpdbscan-master/build
[ 16%] Building CXX object CMakeFiles/hpdbscan.dir/src/hpdbscan.cpp.o
[ 33%] Linking CXX shared library libhpdbscan.so
[ 33%] Built target hpdbscan
[ 50%] Building CXX object CMakeFiles/hpdbscan-bin.dir/src/cli.cpp.o
[ 66%] Linking CXX executable hpdbscan
[ 66%] Built target hpdbscan-bin
[ 83%] Building CXX object CMakeFiles/hpdbscan-binding.dir/hpdbscan_wrap.cpp.o
/media/fbenedet/My Passport/dbscan/hpdbscan-master/build/hpdbscan_wrap.cpp:3344:10: fatal error: numpy/arrayobject.h: No such file or directory
 3344 | #include <numpy/arrayobject.h>
      |          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/hpdbscan-binding.dir/build.make:76: CMakeFiles/hpdbscan-binding.dir/hpdbscan_wrap.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:139: CMakeFiles/hpdbscan-binding.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

I think it is related to the line 61 of CMakeLists.txt: FIND_PACKAGE(NumPy)
I do not understand why it is not working. I have, on my system, numpy in my virtual environment and I do find the arrayobject.h, but the make process cannot find it.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.