geodynamics / pylith_installer Goto Github PK
View Code? Open in Web Editor NEWInstaller code for PyLith.
License: MIT License
Installer code for PyLith.
License: MIT License
Add --developer
configure option to setup build and environment suitable for development.
Do not run make install
for PETSc
* dependencies
* include
* lib
* bin
* petsc
* pylith (plus pyre, nemesis, spatialdata)
* include
* lib
* bin
macOS no longer includes Python v2 so we need to add Python to the PyLith binary.
Add note about VS Code integration that you must open the folder of the top-lvel source directory of a repository. Verify header files, etc are found.
Create docker images with all dependencies (except pyre, nemesis, spatialdata, PETSc, and PyLith).
pylith-testenv-debian-stable-gcc8
(mpich)pylith-testenv-debian-testing-gcc9
(openmpi)pylith-testenv-debian-testing-clang8
(installer: openmpi)pylith-testenv-debian-testing-clang9
(installer: mpich)pylith-testenv-centos-7
(mpich)pylith-testenv-centos-8
(mpich)pylith-testenv-ubuntu-18.04
(openmpi)pylith-testenv-ubuntu-19.10
(mpich)pylith-testenv-fedora-30
(openmpi)pylith-testenv-fedora-31
(mpich)We use distutils.sysconfig.parse_makefile()
which was removed in Python v3.12 in packager/build.py
.
Write a simpleparse_makefile_vars()
function for this functionality. We just need PACKAGE
, VERSION
, and abs_top_srcdir variables
.
The MPI compiler wrappers embed the include path to the MPI install location. If a user has done a make install
and then tries to run make
after editing a header file, the build will see the installed header file, not the updated one.
The short-term workaround is to run make install
in the library subdirectory to install the header files, so that there is no difference between the installed header files and the edited ones.
We should separate the dependencies (including MPI) from the packages that are likely to be edited or need updating by the user (spatialdata, PETSc, PyLith).
Installing the netCDF4 Python module via pip fails with because the numpy
Python module is not found when the setup.py
script tries to import it.
The Python sys.path
is not getting the correct path inside the setup.py
script. It has the Python directory, but not the site-packages
directory under the Python directory. The workaround is to install via python3 setup.py install
rather than using pip.
The MPI wrappers cause problems when trying to develop code ina build created by the installer; see #9.
The solution is to install all of the dependencies that won't be changed to one install directory and spatialdata, PETSc, and PyLith (which won't be changed) to another install directory.
--enable-debugging
configure options. Trigger debugging flags for configure of spatialdata, PETSc, and PyLith.Add --debug
configure option to build with debugging support in PETSc, spatialdata, and PyLith. Keep other dependencies optimized.
Attempting to run ParaView installed using the paraview debian package results in errors associated with OpenGL libraries.
Potential options that need researching:
Installing (and configuring?) appropriate libraries in Docker container for software rendering.
Using x11docker to run a second X server. I believe the user must install this on the host, so this approach may be more work for the user.
/usr/bin/env
Use Ubuntu 20.04 w/USER=pylith-dev
Additional packages:
Install any dependencies built form source into /opt/dependencies
Create Docker volume for CIG code /opt/pylith
/opt
└── pylith
├── src
│ ├── pythia
│ ├── spatialdata
│ ├── petsc
│ └── pylith
├── build
│ ├── debug
│ │ ├── pythia
│ │ ├── spatialdata
│ │ └── pylith
│ └── opt
│ ├── pythia
│ ├── spatialdata
│ └── pylith
└── dest
├── debug
│ ├── bin
│ ├── include
│ ├── lib
│ └── share
└── opt
├── bin
├── include
├── lib
└── share
Use --filter=blob:none
when cloning.
Proj 6.3
CppProj 1.15.1
NetCDF 4.7.3
HDF5 1.10.6
MPICH 3.3.2
cmake 3.16.3
libssl and libcrypto for the Python hashtable module are not being built. We have copied them by hand from the build slave into the PyLith distribution. We should build these from source.
We need to check for any other dependencies that may be included indirectly (Python modules).
The current description of the Git workflow does not clearly separate setup, building, and updating.
Reorganize the description of the Git workflow into clearly distinguishable phases: setup, building, and updating.
docker run -ti --rm -u root registry.gitlab.com/cig-pylith/pylith_installer/pylith-devenv /bin/bash
# Verify ptrace setting needs updating
cat /proc/sys/kernel/yama/ptrace_scope
# If output is 1, then continue, if 0 then no need to change anything.
# Verify ptrace setting
cat /etc/sysctl.d/10-ptrace.conf
# Output should be 0
service procps restart
# Verify setting has changed
cat /proc/sys/kernel/yama/ptrace_scope
# Output should be 0
Add diagram to manual with detailed list of dependencies (based on installer).
Python depends on libcrypt and there are now multiple versions floating around, so we can't rely on the system library. Fedora seems to be the first Linux distribution to move to libcrypt.so.2.
See pypa/manylinux#305 for some detailed discussion of this issue and compatibility.
Create a Python script to build/manage developer source.
developer_helper
Python script--clone=[pythia, spatialdata, petsc, pylith, all]
--configure=[pythia, spatialdata, petsc, pylith, all]
--build=[pythia, spatialdata, petsc, pylith, all]
--pull=[pyre, nemesis, spatialdata, petsc, pylith, all]
--with-optimized [for configure, always include debugging symbols]
--show-config
--merge-from-upstream=[pythia, spatialdata, pylith]
--set-upstream=[pythia, spatialdata, petsc, pylith, all]
Show command or show and run command.
## Build functionality
* Clone
* Configure
* Build
* debug / optimized
## Git functionality
* Set upstream
* Merge from upstream (git pull upstream BRANCH merges to current branch)
* Clone
* Pull
## Config file
[base]
src_dir = /opt/pylith/src
build_dir = /opt/pylith/build
install_dir = /opt/pylith/dest
[petsc]
protocol = ssh/https
repo = https://...
branch = knepley/pylith
upstream = False
[pythia]
protocol = ssh/https
repo_url = https://
branch = main
upstream = False
[spatialdata]
protocol = ssh/https
repo_url = https://...
branch = main
upstream = False
[pylith]
protocol = ssh/https
repo_url = https://...
branch = knepley/feature-petsc-fe
upstream = https://github.com/geodynamics/pylith.git
Comment from Josimar on cig-short:
I am trying to build pylith-2.2.1 from source but the compilation stops at the h5py installation. My machine runs on macOS v. 10.13.6. Note that I was able to build it on a linux computer without any problems.
A summary of the error is below and the complete full set log is attached. Would you have any suggestions to fix this issue ?
Error summary:
Download error on https://pypi.org/simple/pkgconfig/: unknown url type: https -- Some packages may not be found!
Couldn't find index page for 'pkgconfig' (maybe misspelled?)
Download error on https://pypi.org/simple/: unknown url type: https -- Some packages may not be found!
No local packages or working download links found for pkgconfig
Traceback (most recent call last):
File "setup.py", line 167, in <module>
cmdclass = CMDCLASS,
File "build/bdist.macosx-10.13-x86_64/egg/setuptools/__init__.py", line 128, in setup
File "build/bdist.macosx-10.13-x86_64/egg/setuptools/__init__.py", line 123, in _install_setup_requires
File "build/bdist.macosx-10.13-x86_64/egg/setuptools/dist.py", line 504, in fetch_build_eggs
File "build/bdist.macosx-10.13-x86_64/egg/pkg_resources/__init__.py", line 774, in resolve
File "build/bdist.macosx-10.13-x86_64/egg/pkg_resources/__init__.py", line 1057, in best_match
File "build/bdist.macosx-10.13-x86_64/egg/pkg_resources/__init__.py", line 1069, in obtain
File "build/bdist.macosx-10.13-x86_64/egg/setuptools/dist.py", line 571, in fetch_build_egg
File "build/bdist.macosx-10.13-x86_64/egg/setuptools/command/easy_install.py", line 667, in easy_install
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pkgconfig')
make[2]: *** [h5py] Error 1
make[1]: *** [installed_h5py] Error 2
make: *** [installed_pylith] Error 2
To ensure we use the correction Python for running gmsh (python
may not exist, we need to change the #!/usr/bin/env python
to #!/usr/bin/env nemesis
, like we do for all Python scripts in the binary package.
OS X El Capitan removed support for DYLD_LIBRARY_PATH. As a result we use otool to generate link paths relative to the executable. We currently use the Python update_darwinlinking.py
script to do this manually after running the packager/build.py
script.
We should integrate this updating of link paths into the build.py
script.
Note: We must update the link paths for all module .so
files within the Python eggs as well.
Level of difficulty: Easy; this is a straightforward integration of one Python script into another plus some additional automation of shell commands.
PETSc does not get installed correctly when using the v3.0.3-0 installer. make install
and make check
are not getting run.
The fix is to have the installed_petsc
target make both build_petsc
and petsc
rather than having make build_petsc
as part of the petsc
target.
Packages added for master / v2.2.1rc1 fail.
Need to pinpoint under what conditions these build failures occur.
Comment from Josimar regarding building v2.2.1 from source and trying to build the openssl libraries.
I solved this problem. Fundamentally, in my case, during the building process the openssl libraries were not being linked at all during the python installation part. Thus, I modified the existing pylith 2.2.1 Makefile so that it finds the correct openssl libraries.
In summary, if anyone else finds similar issues in the future, here are the steps:
Install openssl (I used homebrew)
Run configure as normal for the pylith installation
Before running "make", open the Makefile and add two extra paths to the LDFLAGS and CPPFLAGS. For my case, I had to add:
a) LDFLAGS: -L/usr/local/opt/openssl/lib
b) CPPFLAGS: -I/usr/local/opt/openssl/include
I am under the impression that the default path for pylith to look for openssl is /usr/local/ssl, which in my case does not exist.
Note that simply configuring pylith-2.2.1 using enable-openssl did not solve my problem.
Build PyLith Linux binaries using oldest LTS CentOS version for maximum compatibility. CentOS glibc version lags behind other Linux distributions (Debian, etc).
Installation of the Proj.4 datums requires unzip. When installing Proj.4, the configure script should check for unzip.
The installer uses the compiler specified by CC to build Python. The installer will detect and use mpicc
and mpicxx
for building anything that needs MPI.
If a user configures the installer with CC=mpicc CXX=mpicxx
, it will generally cause problems building Python because the Python configure script includes:
# ICC needs -fp-model strict or floats behave badly
case "$CC" in
*icc*)
BASECFLAGS="$BASECFLAGS -fp-model strict"
;;
esac
As a result, the Python configure thinks it is being built with an Intel compiler and it adds -fp-model strict
to the CFLAGS
.
The solution is to configure the installer with CC=NON_MPI_C_COMPILER and CXX=NON_MPI_C++_COMPILER. For example, CC=gcc CXX=g++
and let the installer detect mpicc
and mpicxx
that are on the path for building anything that needs MPI.
Python3 includes venv
for virtual environments. We should add a configure option to create a virtual environment (default=yes, which would create a virtual environment in the installation directory).
For CIG codes (pythia, spatialdata, petsc, and PyLith) split installed_PACKAGE
into
fetch-PACKAGE
configure-PACKAGE
build-PACKAGE
install-PACKAGE
check-PACKAGE
This would permit user/developers to run just the steps they need after modifying the code.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.