pyiron / docker-stacks Goto Github PK

View Code? Open in Web Editor NEW

This project forked from max-hassani/pyiron-docker-stack

3.0 4.0 1.0 1.25 MB

Ready-to-run Docker images containing pyiron applications

Home Page: https://hub.docker.com/u/pyiron

License: BSD 3-Clause "New" or "Revised" License

Dockerfile 49.45% Shell 19.63% Python 30.92%

pyiron docker

docker-stacks's Introduction

Docker Stacks

Here, you find the necessary files for building various pyiron docker images. All of these images can be pulled from docker-hub. We provide following flavors based on the main pyiron modules:

Image Name	Derived From	Additional Dependencies	Pull Command
pyiron/base	jupyter/base-notebook	pyiron_base	`docker pull pyiron/base`
pyiron/md	pyiron/base	LAMMPS, pyiron, NGLview	`docker pull pyiron/md`
pyiron/pyiron	pyiron/md	SPHInX, GPAW	`docker pull pyiron/pyiron`
pyiron/experimental	pyiron/base	TEMMETA, pyprismatic, match-series, pyxem, pystem	`docker pull pyiron/experimental`
pyiron/continuum	pyiron/md	damask, sqsgenerator, fenics	`docker pull pyiron/continuum`
pyiron/potentialworkshop	pyiron/pyiron	atomicrex, calphy, pyiron_contrib, pyiron_gpl, python-ace, runner	`docker pull pyiron/potentialworkshop`
pyiron/mpie_cmti	pyiron/pyiron	atomicrex, calphy, fitsnap, pyiron_contrib, pyiron_gpl, pyiron_gui, pyiron_workflow, python-ace, pytorch, runner, tensorflow	`docker pull pyiron/mpie_cmti`

By deriving the images from each other the size of Docker layers is reduced to a minimum. The images also include some example notebooks to get you started.

Execute Docker Container

Running one of these container and spawning a Jupyter server from within will provide you with a ready-to-start environment for using pyiron. If you like a simple Jupyter notebook, run

docker run -i -t -p 8888:8888 <image name> /bin/bash -c "source /opt/conda/bin/activate; jupyter notebook --notebook-dir=/home/jovyan/ --ip='*' --port=8888"

replace <image_name> with respective image you want to use, e.g. pyiron/md. If you prefer to use Jupyter lab, run

docker run -i -t -p 8888:8888 <image_name> /bin/bash -c "source /opt/conda/bin/activate; jupyter lab --notebook-dir=/home/jovyan/ --ip='*' --port=8888"

These commands do a number of things:

docker run <image_name> spawns a container based on image <image_name>. In case the image isn't already on your system, it will be downloaded. Also, if not further specified, the latest tag will be assumed and outdated local versions may be updated.
-i -t: the container is spanwed in "interactive mode" by allocating a pseudo-tty (-t).
-p 8888:8888: port 8888 of the container instance is forwarded to port 8888 of the host.
<image name>: the image's name.
/bin/bash: inside the container, a bash shell is started.
-c "source /opt/conda/bin/activate; jupyter notebook --notebook-dir=/home/jovyan/ --ip='*' --port=8888": the shell executes the command inside the quotation marks:
- source /opt/conda/bin/activate: activate the conda environment
- jupyter notebook or jupyter lab: start a Jupyter server running a notebbok/lab. Do this in the user's (jovyan) home-directory (--notebook-dir=/home/jovyan/) and allow connections from any IP address (--ip='*') on port 8888 (--port=8888) which is connected to the outside.

Data Persistence

In case you want to keep data you worked on/created while using the container, it may be convenient to mount a local directory into the home directory of the docker container by adding -v <local_path>:/home/jovyan/ to the docker run command:

docker run -i -t  -v <local_path>:/home/jovyan/ -p 8888:8888 <image_name> /bin/bash -c "source /opt/conda/bin/activate; jupyter notebook --notebook-dir=/home/jovyan/ --ip='*' --port=8888"

docker-stacks's People

Contributors

Stargazers

Watchers

Forkers

ruginski

docker-stacks's Issues

[New Container] Electrochemistry

To make our activities in electrochemistry better visible and also to make training environments easierly accessible, we should provide a container for that based on pyiron_electrochemistry.

There are some issues regarding this: The module itself has not been worked on since quite a while. However, I thought that @ahmedabdelkawy and coworkers actually still develop things for electrochemistry, I just don't know where these are hosted.

I think we should discuss on how this module shall be updated and also that more recent developments are available to a larger audience. When we have done that, a container can easily be provided.

[Bug] Environment issue

The conda environment is not correctly initialised. On a mybinder container, which does not use a predefined docker image:
https://github.com/matbinder/fenics-example
When I execute the following line in a jupyter notebook:

!env | grep CONDA

I get:

CONDA_SHLVL=1
CONDA_EXE=/srv/conda/bin/conda
GSETTINGS_SCHEMA_DIR_CONDA_BACKUP=
CONDA_PREFIX=/srv/conda/envs/notebook
CONDA_DIR=/srv/conda
_CONDA_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_x86_64_conda_cos6_linux_gnu
CONDA_PYTHON_EXE=/srv/conda/bin/python
_CE_CONDA=
CONDA_PROMPT_MODIFIER=(notebook) 
CONDA_BUILD_SYSROOT=/srv/conda/envs/notebook/x86_64-conda-linux-gnu/sysroot
CONDA_DEFAULT_ENV=notebook

When I use the Docker Image we build here and execute the same command, I get:

CONDA_DIR=/opt/conda
CONDA_VERSION=4.9.2

Only when I activate the environment manually I get the correct output:

!source activate; env | grep CONDA

Output:

CONDA_EXE=/opt/conda/bin/conda
_CONDA_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_x86_64_conda_cos6_linux_gnu
CONDA_PREFIX=/opt/conda
CONDA_PROMPT_MODIFIER=(base) 
_CE_CONDA=
CONDA_SHLVL=1
CONDA_DIR=/opt/conda
CONDA_PYTHON_EXE=/opt/conda/bin/python
CONDA_DEFAULT_ENV=base
CONDA_VERSION=4.9.2
CONDA_BUILD_SYSROOT=/opt/conda/x86_64-conda-linux-gnu/sysroot

So something is wrong with the initialisation of the environment. This should also fix the initialisation of the compilers on the conda level which is the issue in #54 which is currently addressed by installing additional compilers on the operation system level.

Add example notebooks for pyiron_base

@muh-hassani Can you add the notebooks from https://github.com/pyiron/pyiron_base/tree/master/notebooks to be included in pyiron_base ? And then they should also be included in the tests.

[Feature] Generate the environment file without installing

At the moment we install the conda environment in the Docker container to primarily get the environment file with all the resolved dependencies. This can also be achieve with a simple dry-run command:

def get_detailed_environment(environment_input_file, environment_output_file):
    output_start = subprocess.check_output(
        ["conda", "env", "create", "-n", "testenv", "-f", environment_input_file, "--dry-run", "--json"], 
        shell=False, 
        universal_newlines=True
    )
    output_start_dict = json.loads(output_start)
    output_dict = output_start_dict.copy()

    if output_dict["name"] is None:
        del output_dict["name"]
    output_dict["dependencies"] = list(sorted([
        dep.split("::")[-1].replace("==", "=") 
        for dep in output_dict["dependencies"]
    ]))
    with open(environment_output_file, "w") as f:
        f.writelines(yaml.dump(output_dict))

    output_extended = subprocess.check_output(
        ["conda", "env", "create", "-n", "testenv", "-f", environment_output_file, "--dry-run", "--json"], 
        shell=False, 
        universal_newlines=True
    )
    output_extended_dict = json.loads(output_extended)
    return output_extended_dict == output_start_dict

So maybe it is possible to first create the conda environment files and then install them in the docker containers.

[New Container] Ontology

By now we have several developments for ontology related workflows:

So it would be great to provide a Docker container to make these developments available for a larger audience.

installation of damask requires mpi=1.0=mpich

The installation of damask requires mpich and subsquently mpi=1.0=mpich, but since lammps=2021.05.27=*openmpi* is installed, it can not switch to mpich.

Hierarchy of containers

I noticed that most of the containers are derived from: ARG ROOT_CONTAINER=pyiron/atomistic:latest rather than the hierarchical approach we describe in the readme.

[Bug] CMTI image running out of space

The issue appeared in:

Maybe it makes sense to build the CMTI image separately.

Create Github releases for every new push to main

An example how to do this automatically is available here:
https://github.com/pyiron/pyiron-installer/blob/master/.github/workflows/deploy.yml#L18
The advantage is that it then is much easier to identify the status of the Github repository which went into a specific version of the Docker container.

@mbruns91 Can you take a look at this?

Incompatibility of the lammps > 2021.07.31 with continuum packages

The lammps version >2021.07.31 have incompatibility with fenics package. That is why I have pinned it in the continuum/environment.yml. Each time, I update the environments, I try to see if the conflicts are resolved, but so far it is not.

installation of sphinx=3.0.3 failed

The Sphinx-dft installation failed. For now, I changed the version to 2.7.0. The error is as below:

package sphinxdft-3.0.3-h62cddd8_0 requires mkl >=2021.2.0,<2022.0a0, but none of the providers can be installed

jupyterhub can not spawn images without `jovyan` user

I wanted to update the docker images on our different servers. I haven't had done that for a while, since there was no mailing list to inform users about maintenance downtime.
Anyway, now that I tried to use our recent images, I noticed that the jupyterhub fails to spawn the docker containers. It took me a while, to understand that the issue is with our recent images. I found out that when the images have jovyan as the NB_USER it can do the spawning, but it fails when we remove the NB_USER in the base image.
I also tried to unset NB_USER and assign its value to pyiron, but it also fails. I believe there has been a change in the start.sh script, which starts the jupyter containers.

Travis structure

Currently we have different travis stages - which equal separate containers - to build the individual containers. The issue with this approach is that we have to upload a given image to have it available in the next stage. So the recommended way would be to build all images in the same instance, then we can use the local cache.

need for C compiler on pyiron/fenics

Although CI/CD test shows that fenics_tutorial.ipynb runs successfully on the docker image, when I pulled pyiron/fenics and run the notebook, after running the 3rd cell, I receive the following error:

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
/opt/conda/lib/python3.8/site-packages/dolfin/jit/jit.py in compile_class(cpp_data, mpi_comm)
    164     try:
--> 165         module, signature = dijitso_jit(cpp_data, module_name, params,
    166                                         generate=cpp_data['jit_generate'],

/opt/conda/lib/python3.8/site-packages/dolfin/jit/jit.py in mpi_jit(*args, **kwargs)
     46         if MPI.size(mpi_comm) == 1:
---> 47             return local_jit(*args, **kwargs)
     48 

/opt/conda/lib/python3.8/site-packages/dolfin/jit/jit.py in dijitso_jit(*args, **kwargs)
    102 def dijitso_jit(*args, **kwargs):
--> 103     return dijitso.jit(*args, **kwargs)
    104 

/opt/conda/lib/python3.8/site-packages/dijitso/jit.py in jit(jitable, name, params, generate, send, receive, wait)
    176             status, output, lib_filename, err_info = \
--> 177                 build_shared_library(signature, header, source, dependencies,
    178                                      params)

/opt/conda/lib/python3.8/site-packages/dijitso/build.py in build_shared_library(signature, header, source, dependencies, params)
    152     # library
--> 153     status, output = get_status_output(cmd)
    154 

/opt/conda/lib/python3.8/site-packages/dijitso/system.py in _get_status_output_subprocess(cmd, input, cwd, env)
     39         cmd = cmd.strip().split()
---> 40     pipe = subprocess.Popen(cmd, shell=False, cwd=cwd, env=env,
     41                             stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

/opt/conda/lib/python3.8/subprocess.py in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, encoding, errors, text)
    853 
--> 854             self._execute_child(args, executable, preexec_fn, close_fds,
    855                                 pass_fds, cwd, env,

/opt/conda/lib/python3.8/subprocess.py in _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, restore_signals, start_new_session)
   1701                         err_msg = os.strerror(errno_num)
-> 1702                     raise child_exception_type(errno_num, err_msg, err_filename)
   1703                 raise child_exception_type(err_msg)

FileNotFoundError: [Errno 2] No such file or directory: 'c++'

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-3-0df548aa61d7> in <module>
      2 job.domain = job.create.domain.unit_mesh.square(8, 8)
      3 
----> 4 u_D = job.Expression('1 + x[0]*x[0] + 2*x[1]*x[1]', degree=2)  # String expressions must have valid C++ syntax
      5 job.BC = job.create.bc.dirichlet(u_D)
      6 

/opt/conda/lib/python3.8/site-packages/pyiron_contrib/continuum/fenics/job/generic.py in Expression(self, *args, **kwargs)
    362     @copy_docstring(FEN.Expression)
    363     def Expression(self, *args, **kwargs):
--> 364         return FEN.Expression(*args, **kwargs)
    365 
    366     @copy_docstring(FEN.Identity)

/opt/conda/lib/python3.8/site-packages/dolfin/function/expression.py in __init__(self, cpp_code, *args, **kwargs)
    398                     raise KeyError("User parameter key must be a string")
    399 
--> 400             self._cpp_object = jit.compile_expression(cpp_code, params)
    401             self._parameters = ExpressionParameters(self._cpp_object, params)
    402 

/opt/conda/lib/python3.8/site-packages/dolfin/function/jit.py in compile_expression(statements, properties)
    156                 'name': 'expression', 'jit_generate': jit_generate}
    157 
--> 158     expression = compile_class(cpp_data, mpi_comm=mpi_comm)
    159     return expression

/opt/conda/lib/python3.8/site-packages/dolfin/jit/jit.py in compile_class(cpp_data, mpi_comm)
    168         submodule = dijitso.extract_factory_function(module, "create_" + module_name)()
    169     except Exception:
--> 170         raise RuntimeError("Unable to compile C++ code with dijitso")
    171 
    172     if name == 'expression':

RuntimeError: Unable to compile C++ code with dijitso

We should only de

We should make sure only the master branch is pushed to docker hub:
https://github.com/pyiron/pyiron/blob/master/binder/docker/deploy.sh

[Infrastructure] Publish Docker Container on quay.io

Conda-forge and bioconda seem to use https://quay.io rather than docker hub, so we should check if it makes sense to host our images on both registries and what the restrictions for the individual registries are.

GPAW notebooks fails

[NbConvertApp] Converting notebook /home/pyiron/energy_volume_curve.ipynb to notebook
1185[NbConvertApp] Executing notebook with kernel: python3
1186Traceback (most recent call last):
1187  File "/opt/conda/bin/jupyter-nbconvert", line 11, in <module>
1188    sys.exit(main())
1189  File "/opt/conda/lib/python3.8/site-packages/jupyter_core/application.py", line 270, in launch_instance
1190    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
1191  File "/opt/conda/lib/python3.8/site-packages/traitlets/config/application.py", line 837, in launch_instance
1192    app.start()
1193  File "/opt/conda/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 350, in start
1194    self.convert_notebooks()
1195  File "/opt/conda/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 524, in convert_notebooks
1196    self.convert_single_notebook(notebook_filename)
1197  File "/opt/conda/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 489, in convert_single_notebook
1198    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
1199  File "/opt/conda/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 418, in export_single_notebook
1200    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
1201  File "/opt/conda/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 181, in from_filename
1202    return self.from_file(f, resources=resources, **kw)
1203  File "/opt/conda/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 199, in from_file
1204    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
1205  File "/opt/conda/lib/python3.8/site-packages/nbconvert/exporters/notebook.py", line 32, in from_notebook_node
1206    nb_copy, resources = super().from_notebook_node(nb, resources, **kw)
1207  File "/opt/conda/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 143, in from_notebook_node
1208    nb_copy, resources = self._preprocess(nb_copy, resources)
1209  File "/opt/conda/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 318, in _preprocess
1210    nbc, resc = preprocessor(nbc, resc)
1211  File "/opt/conda/lib/python3.8/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
1212    return self.preprocess(nb, resources)
1213  File "/opt/conda/lib/python3.8/site-packages/nbconvert/preprocessors/execute.py", line 79, in preprocess
1214    self.execute()
1215  File "/opt/conda/lib/python3.8/site-packages/nbclient/util.py", line 74, in wrapped
1216    return just_run(coro(*args, **kwargs))
1217  File "/opt/conda/lib/python3.8/site-packages/nbclient/util.py", line 53, in just_run
1218    return loop.run_until_complete(coro)
1219  File "/opt/conda/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
1220    return future.result()
1221  File "/opt/conda/lib/python3.8/site-packages/nbclient/client.py", line 535, in async_execute
1222    await self.async_execute_cell(
1223  File "/opt/conda/lib/python3.8/site-packages/nbconvert/preprocessors/execute.py", line 123, in async_execute_cell
1224    cell, resources = self.preprocess_cell(cell, self.resources, cell_index)
1225  File "/opt/conda/lib/python3.8/site-packages/nbconvert/preprocessors/execute.py", line 146, in preprocess_cell
1226    cell = run_sync(NotebookClient.async_execute_cell)(self, cell, index, store_history=self.store_history)
1227  File "/opt/conda/lib/python3.8/site-packages/nbclient/util.py", line 74, in wrapped
1228    return just_run(coro(*args, **kwargs))
1229  File "/opt/conda/lib/python3.8/site-packages/nbclient/util.py", line 53, in just_run
1230    return loop.run_until_complete(coro)
1231  File "/opt/conda/lib/python3.8/site-packages/nest_asyncio.py", line 95, in run_until_complete
1232    return f.result()
1233  File "/opt/conda/lib/python3.8/asyncio/futures.py", line 178, in result
1234    raise self._exception
1235  File "/opt/conda/lib/python3.8/asyncio/tasks.py", line 280, in __step
1236    result = coro.send(None)
1237  File "/opt/conda/lib/python3.8/site-packages/nbclient/client.py", line 827, in async_execute_cell
1238    self._check_raise_for_error(cell, exec_reply)
1239  File "/opt/conda/lib/python3.8/site-packages/nbclient/client.py", line 735, in _check_raise_for_error
1240    raise CellExecutionError.from_cell_and_msg(cell, exec_reply['content'])
1241nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
1242------------------
1243for strain in np.linspace(0.95, 1.05, 7):
1244    strain_str = str(strain).replace('.', '_')
1245    job_vasp_strain = pr.create_job(job_type=pr.job_type.Gpaw, job_name='gpaw_' + strain_str)
1246    job_vasp_strain.set_encut(320.0)
1247    job_vasp_strain.structure = basis.copy()
1248    job_vasp_strain.structure.set_cell(cell=basis.cell * strain ** (1/3), scale_atoms=True)
1249    job_vasp_strain.run()
1250------------------
1251
1252---------------------------------------------------------------------------
1253RuntimeError                              Traceback (most recent call last)
1254<ipython-input-1-938fbdfb8715> in <module>
1255      5     job_vasp_strain.structure = basis.copy()
1256      6     job_vasp_strain.structure.set_cell(cell=basis.cell * strain ** (1/3), scale_atoms=True)
1257----> 7     job_vasp_strain.run()
1258
1259/opt/conda/lib/python3.8/site-packages/pyiron_base/job/generic.py in run(self, delete_existing_job, repair, debug, run_mode, run_again)
1260    682                 status = "created"
1261    683             if status == "initialized":
1262--> 684                 self._run_if_new(debug=debug)
1263    685             elif status == "created":
1264    686                 que_id = self._run_if_created()
1265
1266/opt/conda/lib/python3.8/site-packages/pyiron_base/job/generic.py in _run_if_new(self, debug)
1267   1392         else:
1268   1393             self.save()
1269-> 1394             self.run()
1270   1395 
1271   1396     def _run_if_created(self):
1272
1273/opt/conda/lib/python3.8/site-packages/pyiron_base/job/generic.py in run(self, delete_existing_job, repair, debug, run_mode, run_again)
1274    684                 self._run_if_new(debug=debug)
1275    685             elif status == "created":
1276--> 686                 que_id = self._run_if_created()
1277    687                 if que_id:
1278    688                     self._logger.info(
1279
1280/opt/conda/lib/python3.8/site-packages/pyiron_base/job/generic.py in _run_if_created(self)
1281   1409             self.run_if_manually()
1282   1410         elif self.server.run_mode.modal:
1283-> 1411             self.run_static()
1284   1412         elif self.server.run_mode.non_modal or self.server.run_mode.thread:
1285   1413             self.run_if_non_modal()
1286
1287/opt/conda/lib/python3.8/site-packages/pyiron/gpaw/gpaw.py in run_static(self)
1288    152         pre_run_mode = self.server.run_mode
1289    153         self.server.run_mode.interactive = True
1290--> 154         self.run_if_interactive()
1291    155         self.interactive_close()
1292    156         self.server.run_mode = pre_run_mode
1293
1294/opt/conda/lib/python3.8/site-packages/pyiron/gpaw/gpaw.py in run_if_interactive(self)
1295    177             self.structure.set_calculator(calc)
1296    178         self.status.running = True
1297--> 179         self.structure.calc.calculate(self.structure)
1298    180         self.interactive_collect()
1299    181 
1300
1301/opt/conda/lib/python3.8/site-packages/gpaw/calculator.py in calculate(self, atoms, properties, system_changes)
1302    282                 self.hamiltonian = None
1303    283                 self.scf = None
1304--> 284                 self.initialize(atoms)
1305    285 
1306    286             self.set_positions(atoms)
1307
1308/opt/conda/lib/python3.8/site-packages/gpaw/calculator.py in initialize(self, atoms, reading)
1309    555         realspace = (mode.name != 'pw' and mode.interpolation != 'fft')
1310    556 
1311--> 557         self.create_setups(mode, xc)
1312    558 
1313    559         if not realspace:
1314
1315/opt/conda/lib/python3.8/site-packages/gpaw/calculator.py in create_setups(self, mode, xc)
1316    779 
1317    780         Z_a = self.atoms.get_atomic_numbers()
1318--> 781         self.setups = Setups(Z_a,
1319    782                              self.parameters.setups, self.parameters.basis,
1320    783                              xc, filter, self.world)
1321
1322/opt/conda/lib/python3.8/site-packages/gpaw/setup.py in __init__(self, Z_a, setup_types, basis_sets, xc, filter, world)
1323   1358                 if isinstance(basis, basestring):
1324   1359                     basis = Basis(symbol, basis, world=world)
1325-> 1360                 setup = create_setup(symbol, xc, 2, type,
1326   1361                                      basis, setupdata=setupdata,
1327   1362                                      filter=filter, world=world)
1328
1329/opt/conda/lib/python3.8/site-packages/gpaw/setup.py in create_setup(symbol, xc, lmax, type, basis, setupdata, filter, world)
1330     84                                  'the %s functional.' % xc.get_setup_name())
1331     85         else:
1332---> 86             setupdata = SetupData(symbol, xc.get_setup_name(),
1333     87                                   type, True,
1334     88                                   world=world)
1335
1336/opt/conda/lib/python3.8/site-packages/gpaw/setup_data.py in __init__(self, symbol, xcsetupname, name, readxml, zero_reference, world, generator_version)
1337    120 
1338    121         if readxml:
1339--> 122             self.read_xml(world=world)
1340    123 
1341    124     def __repr__(self):
1342
1343/opt/conda/lib/python3.8/site-packages/gpaw/setup_data.py in read_xml(self, source, world)
1344    138 
1345    139     def read_xml(self, source=None, world=None):
1346--> 140         PAWXMLParser(self).parse(source=source, world=world)
1347    141         nj = len(self.l_j)
1348    142         self.e_kin_jj.shape = (nj, nj)
1349
1350/opt/conda/lib/python3.8/site-packages/gpaw/setup_data.py in parse(self, source, world)
1351    436         setup = self.setup
1352    437         if source is None:
1353--> 438             setup.filename, source = search_for_file(setup.stdfilename, world)
1354    439 
1355    440         setup.fingerprint = hashlib.md5(source).hexdigest()
1356
1357/opt/conda/lib/python3.8/site-packages/gpaw/setup_data.py in search_for_file(name, world)
1358    421 https://wiki.fysik.dtu.dk/gpaw/install.html#install-paw-datasets
1359    422 for details."""
1360--> 423         raise RuntimeError('%s\n%s' % (err, helpful_message))
1361    424 
1362    425     return filename, source
1363
1364RuntimeError: Could not find required PAW dataset file "Fe.PBE".
1365
1366You need to set the GPAW_SETUP_PATH environment variable to point to
1367the directories where PAW dataset and basis files are stored.  See
1368https://wiki.fysik.dtu.dk/gpaw/install.html#install-paw-datasets
1369for details.
1370RuntimeError: Could not find required PAW dataset file "Fe.PBE".
1371
1372You need to set the GPAW_SETUP_PATH environment variable to point to
1373the directories where PAW dataset and basis files are stored.  See
1374https://wiki.fysik.dtu.dk/gpaw/install.html#install-paw-datasets
1375for details.
1376
1377

Mybinder issue

Mybinder removes everything from the home directory:

Step 2/7 : RUN rm ${HOME}/*

DAMASK example

@muh-hassani Can you add the notebook from Vitesh? https://github.com/vitesh13/DAMASK_workflows/blob/master/DAMASK_WORKFLOWS.ipynb and the required example files these are available at https://github.com/vitesh13/DAMASK_workflows/tree/master/Examples In addition the notebook also requires two more python modules https://github.com/vitesh13/DAMASK_workflows/blob/master/Plot_stress_strain.py and https://github.com/vitesh13/DAMASK_workflows/blob/master/simulation_runner.py . I created a cleaned up fork of the repository here https://github.com/matbinder/DAMASK-example so either you can use this or fork it to your account to add the elastic constant example to demonstrate the combined workflow of LAMMPS and DAMASK.

[Optimization] Size of the current Docker Images

The Docker Images we create in particular for pyiron/potentialworkshop, pyiron/continuum and pyiron/pyiron are rather large >5GB:

pyiron/experimental        2023-11-14   8f3834de78a2   26 seconds ago   3.58GB
pyiron/experimental        latest       8f3834de78a2   26 seconds ago   3.58GB
pyiron/potentialworkshop   2023-11-14   929f5c149f76   3 minutes ago    7.51GB
pyiron/potentialworkshop   latest       929f5c149f76   3 minutes ago    7.51GB
pyiron/pyiron              2023-11-14   aa03d4dda95b   6 minutes ago    6.18GB
pyiron/pyiron              latest       aa03d4dda95b   6 minutes ago    6.18GB
pyiron/continuum           2023-11-14   74ca564c0563   12 minutes ago   6.5GB
pyiron/continuum           latest       74ca564c0563   12 minutes ago   6.5GB
pyiron/md                  2023-11-14   0c33e18a8724   16 minutes ago   3.96GB
pyiron/md                  latest       0c33e18a8724   16 minutes ago   3.96GB
pyiron/base                2023-11-14   e9f88a9cf943   18 minutes ago   1.91GB
pyiron/base                latest       e9f88a9cf943   18 minutes ago   1.91GB

In the case of pyiron/potentialworkshop and pyiron/pyiron this is most likely related to including the DFT simulation codes and in case of pyiron/continuum it is the continuum codes. I am not sure if we can improve the images in a reasonable time frame, still I thought it is interesting to share the size of the current images from time to time.

pyiron continuum image

As we have proceeded with the modularity of pyiron, I think it would be useful to create a pyiron/continuum image, instead of having both damask and finics as separate images, though we can wait till the fenics_job branch is merged to pyiron_contrib. If you, @jan-janssen, agree, I can work on it next week, after pyiron_contrib merge. There are several codacy issues still to be fix in fenics_job branch, I will fix them first.

DAMASK=*_14 and FEniCS incompatibility

Since the latest builts of DAMASK are based on openmpi, while FEniCS has no openmpi built, it is not possible to have them both in the same conda environment. For the moment, DAMASK=3.0.0=*_9 is used, since pyiron_continuum is not also compatible yet with latest release of DAMASK.

[Awareness] continuum: `damask` and `fenics` have incompatible dependencies

The issue is also discussed here. pyiron/pyiron_continuum#262 (review) .

As pointed out by @jan-janssen , we have to make a decision how to handle this here. I'd vote as well for the 2-container solution, one still called "continuum", containing damask and another one containing fenics, which we could simply call "fenics".

I opened this issue to keep track of this. I can take care of it after my easter-vacation.

[Bug] Notebook Tests not executed

Run docker run --rm pyiron/experimental /bin/bash -c 'source /opt/conda/bin/activate; i=0; for f in $(ls "${HOME}"/notebooks_*/*.ipynb); do jupyter nbconvert --ExecutePreprocessor.timeout=9999999 --to notebook --execute $f || i=$((i+1)); done; if [ $i -gt 0 ]; then exit 1; fi;'
ls: cannot access '/home/jovyan/notebooks_*/*.ipynb': No such file or directory

Confirm that NGLview is working for both Jupyterlab and Jupyter notebooks

Currently I have the feeling NGLview is not installed for Jupyterlab

Which Docker Images do we need?

As already mentioned in #23 (comment) while I previously suggested to have more images I guess it should be fine to reduce the number of images:

pyiron/base - which includes - pyiron_base
pyiron/lammps - which is based on pyiron/base and additionally includes pyiron, nglview and lammps.
pyiron/pyiron - which is based on pyiron/lammps and additionally includes sphinx and gpaw.

In addition we should include Dockerfiles for all images used on the internal server, especially the TEM and DAMASK examples.

Add unit tests for pyiron_base

@muh-hassani Can you add the notebooks from https://github.com/pyiron/pyiron_base/tree/master/notebooks to be included in pyiron_base ? And then they should also be included in the tests.

[New Container] Workflow

@liamhuber As we already prepare a series of examples for the up-coming ADIS workshop, it would be great to also create a Docker Container for the new workflow environment so people can test it as part of the Material Digital Cloud.

pyiron atomistic

Is it possible to derive pyiron_atomistic from pyiron_base rather than jupyter:notebook-base ?

Specifying versions for the dependencies

Based on the new implementation, all the dependencies are specified by the version. I have a concern and a question about this.
Question: The versions specified are the latest versions. So why shouldn't the docker images be based on the latest versions?
Concern: In case of any update to the dependecies, we need a bot to update the versions and start a pull-request. But then we have to check whether pyiron implementation is compatible with the latest version of the dependencies or not as well.

[automation] Dependant and weekly releases

With the recent pull requests #147 and #150 this repository should now be automatically updated. Dependabot opens pull requests, when new python packages are available and new software stacks are released on a weekly bases.

ToDos:

I just fixed the weekly release part today, so we should validate that it is automatically released next week.
Dependabot currently only updates the requirements.txt file in the core of the repository to inform us about updates. But the conda environment files need to be updated manually. @niklassiemer is it possible to extend the Dependabot action to update the conda environment files as well? You did it for other repositories in the pyiron organisation so I think it should be possible here as well, the only challenge is that there are a range of different environment files with different requirements.

[Clean up] Remove Custodian dependency in pyiron/md

In #159 I added custodian as additional dependency to resolve the issue of mp-api and emmet-core both not including custodian in their dependencies. Still once this issue is resolved then we should remove custodian from the dependencies again.

[Bug] scikit-learn is broken

The scikit-learn build scikit-learn=1.3.2=py311hc009520_0 which is linked in the current images is no longer available on conda-forge. This was related to a bug I identified that the package included the source code of 1.4.dev0 rather than 1.3.2:
conda-forge/scikit-learn-feedstock#237
As a consequence the conda environments published as part of our docker releases can no longer be created. Unless the line scikit-learn=1.3.2=py311hc009520_0 is replaced with scikit-learn=1.3.2=py311hc009520_1 then everything works fine again.

Still the underlying issue is that conda-forge always has the freedom to mark packages as broken, resulting in our environments being no longer reproducible. To address this challenge, it would be great to test the images from time to time and summarise the results on a Github Page. As broken packages no longer need to be tested and older images do not need to be tested that frequently I think it should be possible to automate the process.

The steps I would take to address this:

Query the Github API to get all releases of this repository and the links to the attached environment files.
Create a weekly Github Action, which queries the Github API for the environments and builds a table of which environments are available and where the corresponding docker container can be found. This table could be published as HTML table and maybe as CSV or JSON, so we can use the existing table for further processing.
Finally, develop a script which tests if environments can be created inside a virtual machine and mark the environments as broken if this is no longer possible.

Consistent naming

I guess it is better to use the same names for the folders in this repository like we use them for the docker images. So the folder pyiron_atomistic should be renamed atomistic and so on. @muh-hassani Can you do this?

[Infrastructure] Compatibility with the MaterialDigital Cloud

When we started updating the Docker Containers, there was an issue with the old pyiron docker images not being compatible to the latest version of Jupyterhub used by the MaterialDigital Cloud. Is this issue resolved with the release of the new images? Or is there something we can do from the pyiron side to improve the compatibility?

[MPIE] The pyiron installation on the cluster is outdated

I created a pull request to include the recent updates:
#329

This includes the changes in:

Inconsistent use of .dockerignore

Some subfolders contain .dockerignore files but I do not think they are necessary with our current configuration as the REAME file is located in the main directory.

Add tests?

As we copy the notebooks is it possible to execute those during the testing of the containers? By using different stages for building the image and testing the images we could even run all the tests in parallel - the pyiron CI script uses such stages to test on multiple operation systems and then build the documentation only when the tests were successful https://github.com/pyiron/pyiron/blob/master/.travis.yml#L3

[New Container] Replace the `potentialworkshop` image by a `potentialfit` image

The potentialworkshop image is based on the potentials workshop from 2021. Shall we maybe replace it by an image providing developments from pyiron_potentialfit?

[Feature] Automated weekly releases

At the moment a merged pull request no longer triggers a new release, only when a new tag is created a new release is triggered. So the suggestion would be to create a Github action which creates a new tag every week to update the containers. While this might not change the specified dependencies, it can still change the dependencies which we do not explicitly specify.

Rename the repository to docker-stacks

I guess we should follow the same naming scheme jupyter is using https://github.com/jupyter/docker-stacks

Name of the docker containers

Currently all the Docker container names end with -image is this necessary?
https://hub.docker.com/u/muhhassani

I would prefer something like:
pyiron/pyiron
or even:
pyiron/atomistic, pyiron/base and so on

jupyterlab and nglview

As you know nglview lab extension is not compatible with jupytelab3, and by default I have chosen lab to be the environment for the workshop cluster. The first solution would be to suppress the version of jupyterlab to 2.2.9 at pyiron/base or pyiron/md.
The second way would be to change the environment to notebook rather than lab.
@jan-janssen, what do you think?

itkwidgets requires python <3.9.0

Since itkwidgets supports in limited to python=3.10, for pyiron/continuum image, I have downgraded python to python=3.8.
I thought it would be best that I document it here. Then we can later on remove the downgrade of python continuum/environment.yaml.