Comments (16)
JanuszL
commented on May 15, 2024
2
Hi,
If you extract it anywhere you need to run cmake with NVJPEG_ROOT_DIR argument which value points to dir where you have your nvJPEG.
Other way is to copy nvJPEG libs to /usr/local/lib/ and headers to /usr/local/include/.
from dali.
cliffwoolley
commented on May 15, 2024
2
It sounds like you didn't do a recursive clone of the git repo. Try this:
git submodule sync --recursive
git submodule update --init --recursive
from dali.
cliffwoolley
commented on May 15, 2024
1
I'm working on a PR that would make NVJPEG_ROOT_DIR (and similar vars for other dependencies) a bit more usable via ccmake. But yes you can already specify that with something like cmake ... -DNVJPEG_ROOT_DIR=/path/to/nvjpeg/ .
from dali.
shl666
commented on May 15, 2024
THXs for all replies.
I am working on a public server and do not have access to usr/local.
So I tried the first method:
$cmake -DBUILD_LMDB=ON -NVJPEG_ROOT_DIR=/home/smluo/GithubProject/cuda-linux64-nvjpeg ..
-- root:
CMake Error at /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
Could NOT find NVJPEG (missing: NVJPEG_INCLUDE_DIR NVJPEG_LIBRARY)
Call Stack (most recent call first):
/usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA_FAILURE_MESSAGE)
cmake/modules/FindNVJPEG.cmake:32 (find_package_handle_standard_args)
cmake/Dependencies.cmake:13 (find_package)
CMakeLists.txt:47 (include)
-- Configuring incomplete, errors occurred!
See also "/home/smluo/GithubProject/DALI-master/build/CMakeFiles/CMakeOutput.log".
See also "/home/smluo/GithubProject/DALI-master/build/CMakeFiles/CMakeError.log".Unfortunately, it does not work. It looks not reasonable. Or due to I am not familiar with cmake, wrote the command in a wrong way.
from dali.
JanuszL
commented on May 15, 2024
To pass variable to cmake you need to use -D_VARIABLE_NAME_. So try:
cmake -DBUILD_LMDB=ON -DNVJPEG_ROOT_DIR=/home/smluo/GithubProject/cuda-linux64-nvjpeg ..
from dali.
shl666
commented on May 15, 2024
@JanuszL Brilliant!!!!
However, some other errors comes out. It seems lack of CMakelist.txt files in DALI-master/third_party/googletest, DALI-master/third_party/benchmark and DALI-master/third_party/pybind11. And another Unkown error.
Following are details:
$cmake -DBUILD_LMDB=ON -DNVJPEG_ROOT_DIR=/home/smluo/GithubProject/cuda-linux64-nvjpeg ..
-- root: /home/smluo/GithubProject/cuda-linux64-nvjpeg
-- Found NVJPEG: /home/smluo/GithubProject/cuda-linux64-nvjpeg/include
-- Found NVJPEG (include: /home/smluo/GithubProject/cuda-linux64-nvjpeg/include, library: /home/smluo/GithubProject/cuda-linux64-nvjpeg/lib64/libnvjpeg_static.a)
CMake Error at cmake/Dependencies.cmake:59 (add_subdirectory):
The source directory
/home/smluo/GithubProject/DALI-master/third_party/googletest
does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
CMakeLists.txt:47 (include)
CMake Error at cmake/Dependencies.cmake:66 (add_subdirectory):
The source directory
/home/smluo/GithubProject/DALI-master/third_party/benchmark
does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
CMakeLists.txt:47 (include)
-- Found JPEG_TURBO: /home/smluo/anaconda3/include
-- Found JPEG_TURBO (include: /home/smluo/anaconda3/include, library: /home/smluo/anaconda3/lib/libturbojpeg.so)
-- Found OpenCV 2.4.9.1 (libs: opencv_core;opencv_imgproc;opencv_core;opencv_imgproc;opencv_highgui)
CMake Error at cmake/Dependencies.cmake:103 (add_subdirectory):
The source directory
/home/smluo/GithubProject/DALI-master/third_party/pybind11
does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
CMakeLists.txt:47 (include)
-- Found LMDB: /usr/include
-- Found LMDB /usr/include : /usr/lib/x86_64-linux-gnu/liblmdb.so
-- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so
-- Found Protobuf version 2 : /usr/include : /usr/lib/x86_64-linux-gnu/libprotobuf.so
-- Exclude libs 'libcudart_static.a:libnvjpeg_static.a:libnppc_static.a:libnppi_static.a:libculibos.a:libturbojpeg.a:libopencv_core.a:libopencv_imgproc.a:libopencv_imgcodecs.a:liblmdb.a'
-- Adding dependencies to dali: '/usr/local/cuda-8.0/lib64/libcudart_static.a;-lpthread;dl;/usr/lib/x86_64-linux-gnu/librt.so;/home/smluo/GithubProject/cuda-linux64-nvjpeg/lib64/libnvjpeg_static.a;/usr/local/cuda-8.0/lib64/libnppc_static.a;/usr/local/cuda-8.0/lib64/libnppi_static.a;/usr/local/cuda-8.0/lib64/libculibos.a;/home/smluo/anaconda3/lib/libturbojpeg.so;opencv_core;opencv_imgproc;opencv_core;opencv_imgproc;opencv_highgui;/usr/lib/x86_64-linux-gnu/liblmdb.so;/usr/lib/x86_64-linux-gnu/libprotobuf.so'
-- Adding dependencies to dali_test.bin: 'dali'
-- Adding dependencies to dali_benchmark.bin: 'dali'
CMake Error at dali/python/CMakeLists.txt:18 (pybind11_add_module):
Unknown CMake command "pybind11_add_module".
-- Configuring incomplete, errors occurred!
See also "/home/smluo/GithubProject/DALI-master/build/CMakeFiles/CMakeOutput.log".
See also "/home/smluo/GithubProject/DALI-master/build/CMakeFiles/CMakeError.log".from dali.
shl666
commented on May 15, 2024
@cliffwoolley Great it works!!!
$ cmake -DBUILD_LMDB=ON -DNVJPEG_ROOT_DIR=/home/smluo/GithubProject/cuda-linux64-nvjpeg -DJPEG_TURBO_ROOT_DIR=/home/smluo/anaconda3/envs/python35 ..
-- The CXX compiler identification is GNU 5.4.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The C compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Found PythonInterp: /home/smluo/anaconda3/envs/python35/bin/python (found version "3.5.5")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Found Git: /usr/bin/git (found version "2.7.4")
-- git Version: v0.0.0-dirty
-- Version: 0.0.0
-- Performing Test HAVE_CXX_FLAG_STD_CXX11
-- Performing Test HAVE_CXX_FLAG_STD_CXX11 - Success
-- Performing Test HAVE_CXX_FLAG_WALL
-- Performing Test HAVE_CXX_FLAG_WALL - Success
-- Performing Test HAVE_CXX_FLAG_WEXTRA
-- Performing Test HAVE_CXX_FLAG_WEXTRA - Success
-- Performing Test HAVE_CXX_FLAG_WSHADOW
-- Performing Test HAVE_CXX_FLAG_WSHADOW - Success
-- Performing Test HAVE_CXX_FLAG_WERROR
-- Performing Test HAVE_CXX_FLAG_WERROR - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC
-- Performing Test HAVE_CXX_FLAG_PEDANTIC - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS - Success
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32 - Failed
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL - Success
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WZERO_AS_NULL_POINTER_CONSTANT
-- Performing Test HAVE_CXX_FLAG_WZERO_AS_NULL_POINTER_CONSTANT - Success
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WD654
-- Performing Test HAVE_CXX_FLAG_WD654 - Failed
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY - Failed
-- Performing Test HAVE_CXX_FLAG_COVERAGE
-- Performing Test HAVE_CXX_FLAG_COVERAGE - Success
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX -- success
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX -- success
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK -- success
-- Found CUDA: /usr/local/cuda-8.0 (found suitable version "8.0", minimum required is "8.0")
-- Found NVJPEG: /home/smluo/GithubProject/cuda-linux64-nvjpeg/include (found suitable version "9.0", minimum required is "9.0")
CMake Warning at cmake/Dependencies.cmake:37 (message):
Using nvJPEG 9.0 together with CUDA 8.0 requires NVIDIA drivers compatible
with CUDA 9.0 or later
Call Stack (most recent call first):
CMakeLists.txt:47 (include)
-- Found OpenCV: /home/smluo/anaconda3/envs/python35/include;/home/smluo/anaconda3/envs/python35/include/opencv (found suitable version "3.4.1", minimum required is "2.0")
-- Found PythonLibs: /home/smluo/anaconda3/envs/python35/lib/libpython3.5m.so
-- pybind11 v2.3.dev0
-- Found LMDB: /usr/include (found suitable version "0.9.17", minimum required is "0.9")
-- Found Protobuf: /home/smluo/anaconda3/envs/python35/lib/libprotobuf.so;-pthread (found suitable version "3.5.1", minimum required is "2.0")
-- Enabling TensorFlow TFRecord file format support
-- Exclude libs 'libcudart_static.a:libnvjpeg_static.a:libnppc_static.a:libnppi_static.a:libculibos.a:libturbojpeg.a:libopencv_core.a:libopencv_imgproc.a:libopencv_highgui.a:libopencv_imgcodecs.a:liblmdb.a'
-- Adding dependencies to dali: '/usr/local/cuda-8.0/lib64/libcudart_static.a;-pthread;dl;/usr/lib/x86_64-linux-gnu/librt.so;/home/smluo/GithubProject/cuda-linux64-nvjpeg/lib64/libnvjpeg_static.a;/usr/local/cuda-8.0/lib64/libnppc_static.a;/usr/local/cuda-8.0/lib64/libnppi_static.a;/usr/local/cuda-8.0/lib64/libculibos.a;opencv_core;opencv_imgproc;opencv_imgcodecs;/usr/lib/x86_64-linux-gnu/liblmdb.so;/home/smluo/anaconda3/envs/python35/lib/libprotobuf.so'
-- Adding dependencies to dali_test.bin: 'dali'
-- Adding dependencies to dali_benchmark.bin: 'dali'
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- LTO enabled
-- Adding dependencies to backend_impl: 'dali'
-- Configuring done
-- Generating doneBut I still gets errors after I run bash $make -j"$(nproc)" install
There are too many warnings and two main errors.
[ 93%] Linking CXX executable python/nvidia/dali/test/dali_benchmark.bin
libdali.so: undefined reference to `tjDestroy'
libdali.so: undefined reference to `tjInitDecompress'
libdali.so: undefined reference to `tjDecompress2'
libdali.so: undefined reference to `nppiJpegDecodeGetScanDeadzoneSize'
collect2: error: ld returned 1 exit status
dali/CMakeFiles/dali_benchmark.bin.dir/build.make:266: recipe for target 'dali/python/nvidia/dali/test/dali_benchmark.bin' failed
make[2]: *** [dali/python/nvidia/dali/test/dali_benchmark.bin] Error 1
CMakeFiles/Makefile2:445: recipe for target 'dali/CMakeFiles/dali_benchmark.bin.dir/all' failed
make[1]: *** [dali/CMakeFiles/dali_benchmark.bin.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 93%] Building CXX object dali/CMakeFiles/dali_test.bin.dir/pipeline/operators/op_schema_test.cc.o
[ 94%] Building CXX object dali/CMakeFiles/dali_test.bin.dir/pipeline/operators/operator_factory_test.cc.o
[ 95%] Building CXX object dali/CMakeFiles/dali_test.bin.dir/pipeline/executor/executor_test.cc.o
[ 95%] Building CXX object dali/CMakeFiles/dali_test.bin.dir/pipeline/pipeline_test.cc.o
[ 96%] Building CXX object dali/CMakeFiles/dali_test.bin.dir/pipeline/op_graph_test.cc.o
[ 97%] Building CXX object dali/CMakeFiles/dali_test.bin.dir/test/dali_test.cc.o
[ 98%] Linking CXX shared module nvidia/dali/backend_impl.cpython-35m-x86_64-linux-gnu.so
[100%] Linking CXX executable python/nvidia/dali/test/dali_test.bin
libdali.so: undefined reference to `tjDestroy'
libdali.so: undefined reference to `tjInitDecompress'
libdali.so: undefined reference to `tjDecompress2'
libdali.so: undefined reference to `nppiJpegDecodeGetScanDeadzoneSize'
collect2: error: ld returned 1 exit status
dali/CMakeFiles/dali_test.bin.dir/build.make:551: recipe for target 'dali/python/nvidia/dali/test/dali_test.bin' failed
make[2]: *** [dali/python/nvidia/dali/test/dali_test.bin] Error 1
CMakeFiles/Makefile2:407: recipe for target 'dali/CMakeFiles/dali_test.bin.dir/all' failed
make[1]: *** [dali/CMakeFiles/dali_test.bin.dir/all] Error 2
[100%] Built target backend_impl
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2I only paste the errors.
I have totally no idea about them.
from dali.
cliffwoolley
commented on May 15, 2024
That one is a bit more of a mystery. From the CMake output I see you are using CUDA 8, support for which was a recent addition that I haven't personally tested at all; that could be the source of the NPPI symbol error. For TurboJpeg I see that CMake found an install in your Conda environment but wasn't able to tell what version of TurboJpeg (libjpeg-turbo) it was; that likely means it was libjpeg-turbo < 1.5 (whereas 1.5 is the minimum requirement and also the version that starts reporting its own version number in the way we presently expect; I'm working on detecting the older versions better).
It seems you could either try passing these two additional flags to CMake: -DBUILD_BENCHMARK=OFF -DBUILD_TEST=OFF or upgrade your libjpeg-turbo package and your CUDA.
But, backing up... is there a reason you can't use the pre-built wheels?
from dali.
JanuszL
commented on May 15, 2024
Hi,
CUDA 8.0 is not officially supported so you need to know what you are doing when you use it.
nppiJpegDecodeGetScanDeadzoneSize is used by nvJpeg, and the symbol itself is available in CUDA 9.x. If you want to proceed with CUDA 8.0 you need to disable nvJpeg.
Regarding libjpeg-turbo I see:
Call Stack (most recent call first):
CMakeLists.txt:47 (include)
As compilation itself succeed it means that you have libjpeg-turbo > 1.5 (or hacked header which may be one of the causes why it doesn't work).
Please rerun linking command with VERBOSE option
make VERBOSE=1
And check if libjpeg-turbo is properly linked there.
from dali.
JanuszL
commented on May 15, 2024
It seems that we can do nvJpeg to be independent form CUDA runtime. We will try to make that happen in next nvJpeg release.
from dali.
shl666
commented on May 15, 2024
@JanuszL @cliffwoolley THX for your replies.
For libjpeg-turbo, I installed it by using conda (this is the only way cuz I don't have root privileges on the remote server). I found its version (>1.5) could not identified for some reasons so I comment the 88th line of dali/cmake/CMakelist.txt. Its version could be identified later(I dont know why), so I undo this operation.
For CUDA, I updated it to 9.0 and tried to compile DALI again.
It gives the same feedback, except `nppiJpegDecodeGetScanDeadzoneSize' is defined this time.
(python35) [smluo@Node4 ~/GithubProject/dali/build]
$nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176
(python35) [smluo@Node4 ~/GithubProject/dali/build]
$make -j"$(nproc)" install
[ 0%] Built target libdali.map
[ 2%] Built target gtest
[ 3%] Built target DALI_PROTO
[ 4%] Built target CAFFE2_PROTO
[ 10%] Built target CAFFE_PROTO
[ 12%] Built target TF_PROTO
[ 21%] Built target benchmark
[ 79%] Built target dali
[ 80%] Linking CXX executable python/nvidia/dali/test/dali_benchmark.bin
[ 81%] Linking CXX shared module nvidia/dali/backend_impl.cpython-35m-x86_64-linux-gnu.so
[ 81%] Linking CXX executable python/nvidia/dali/test/dali_test.bin
libdali.so: undefined reference to `tjDecompress2'
libdali.so: undefined reference to `tjDestroy'
libdali.so: undefined reference to `tjInitDecompress'
collect2: error: ld returned 1 exit status
dali/CMakeFiles/dali_benchmark.bin.dir/build.make:215: recipe for target 'dali/python/nvidia/dali/test/dali_benchmark.bin' failed
make[2]: *** [dali/python/nvidia/dali/test/dali_benchmark.bin] Error 1
CMakeFiles/Makefile2:443: recipe for target 'dali/CMakeFiles/dali_benchmark.bin.dir/all' failed
make[1]: *** [dali/CMakeFiles/dali_benchmark.bin.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
libdali.so: undefined reference to `tjDestroy'
libdali.so: undefined reference to `tjInitDecompress'
libdali.so: undefined reference to `tjDecompress2'
collect2: error: ld returned 1 exit status
dali/CMakeFiles/dali_test.bin.dir/build.make:526: recipe for target 'dali/python/nvidia/dali/test/dali_test.bin' failed
make[2]: *** [dali/python/nvidia/dali/test/dali_test.bin] Error 1
CMakeFiles/Makefile2:405: recipe for target 'dali/CMakeFiles/dali_test.bin.dir/all' failed
make[1]: *** [dali/CMakeFiles/dali_test.bin.dir/all] Error 2
[ 82%] Built target backend_impl
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2Then I followed @cliffwoolley 's suggestion, added bash -DBUILD_BENCHMARK=OFF -DBUILD_TEST=OFF and DALI was installed successfully.
BTW, I still didn't get what's wrong with benchmark and test part.
from dali.
JanuszL
commented on May 15, 2024
Looking into your CMakeCache.txt file from #48 I don't see JPEG_TURBO_ROOT_DIR or JpegTurbo_* variables.
Can you try with latest version of code, we did few improvements to libjpeg-turbo finding process.
from dali.
shl666
commented on May 15, 2024
I commented find_package(JpegTurbo 1.5 REQUIRED) in cmake/Dependencies.cmake before, for avoid error about cannot identify version of JpegTurbo.
I have pulled the latest source code. Unfortunately, it does not work for my situation.
Following information may help:
I installed JpegTurbo by annaconda
conda install -c conda-forge libjpeg-turboAnd now when I type conda list I can see it:
libjpeg-turbo 1.5.90 0 conda-forgefrom dali.
JanuszL
commented on May 15, 2024
Try to install pkg-config as well:
conda install -c conda-forge pkg-config
If you comment out find_package(JpegTurbo 1.5 REQUIRED), then cmake is not able to properly find headers and so library and populate appropriate variables.
from dali.
shl666
commented on May 15, 2024
Nice! It works. It finds my JpegTurbo.
from dali.
JanuszL
commented on May 15, 2024
Good. Closing.
from dali.
Related Issues (20)
- Segmentation fault when using 'mixed' HOT 5
- Bbox Pruning Too Aggressive? HOT 5
- Indexing video with binary mask HOT 1
- source_info tensor not guaranteed to contain correct data HOT 1
- 16 bit gray scale Image read error HOT 1
- COCO Reader pixelwise_masks Emtpy Output HOT 7
- Dali on Jetson: nvidia.dali.fn.readers.video_resize is missing HOT 4
- Numpy reader test (GDS) HOT 4
- How to add a scalar value to the loader? HOT 1
- Can DALI be integrated into HuggingFace Trainer? HOT 9
- Bug in creating `TensorGPU` when `stream` key is `None` in CUDA array interface HOT 2
- Configure max image size HOT 3
- Webdataset reader behavior with many sources HOT 1
- ModuleNotFoundError: No module named 'nvidia.dali.python_function_plugin' HOT 3
- Speed up Dino with DALI HOT 3
- error using webdataset
- webdataset cannot stop cycling at end of epoch HOT 11
- Get audio data from external data sources and start iteration HOT 7
- Encountered some issues when using mirror flip HOT 2
- Can AArch64 Numba tests be re-renabled? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dali.