The instructions do not say to create a new directory called earth-analytics-binder, but in order to cd to it, I created it and copied environment.yml and Dockerfile into this new folder.

When running the provided command (docker build -t earthlab/earth-analytics .), the build seems to fail when trying to download repodata.json.bz2 file from https://repo.anaconda.com/pkgs/main/linux-64/. I checked that the file exists, but it appears to have been updated today and may have an error.

Per this pr
earthlab/earthpy#6

waiting to figure out whether it should be installed via pip vs conda forge which is preferred.

Ideally, students shouldn't have to worry about choosing among environments within a Docker container. It would be best to be able to install the environment into the "base" or "root" conda environment, and have that be the only environment available within the container.

• ---

• name : Bug report
• about: Create a report to help us improve
• title: Issue with earth analytics python environment
• labels : ''
• assiganees: ''

• ---

Describe the bug
Unable to run any code that uses the matplotlib.pyplot package

To Reproduce
Steps to reproduce the behavior:

1. import matplotlib.pyplot as plt
2. run code
3. See error

ImportError: dlopen(/Users/[username-removed]/miniconda3/envs/earth-analytics-python/lib/python3.7/site-packages/PIL/_imaging.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libwebp.7.dylib

Referenced from: /Users/[username-removed]/miniconda3/envs/earth-analytics-python/lib/libtiff.5.dylib
Reason: Incompatible library version: libtiff.5.dylib requires version 9.0.0 or later, but libwebp.7.dylib provides version 8.0.0

**What Operating System Are you Running? **

• Mac

Additional context
I have upgraded / updated both packages - libtiff and libwebp without luck

@nkorinek

This will be used in github actions to more efficiently test notebooks using the earth-analytics-python environment.

Windows and linux releases could also be added eventually so that users can take advantage of quicker install times, but we'll start with Mac OS.

The constructors will not be tracked because they are too large. A construct.yaml will be added, the .gitignore updated to ignore the constructors, and the README updated.

The URL generated by the current command (docker run -it -p 8888:8888 earthlab/earth-analytics-python-env) substitutes the container ID for localhost.

Per jupyter/docker-stacks#642, command needs to be updated to include --hostname [hostname] (e.g. docker run --hostname localhost -it -p 8888:8888 earthlab/earth-analytics-python-env).

It would be nice if we could verify that this conda environment builds on Windows, Mac, and Linux. We already have Linux covered via Docker, but adding CI for the other platforms will save us some trouble and effort later on.

• Test on Windows via AppVeyor
• Test on Mac OS X via Travis
• Test on Linux via Travis

good morning @mbjoseph i'm so much more productive now that i have turned OFF filtering of my github message in my email.

Anyway does this repo have auto rebuild capacity or can it? it would be super duper cool to set that up to support additions to our conda environment? I'm working on a scope of work now and would like to identify what things we can "easily" build into our system vs things we need to contract out. it sounded like you may have a clever way already to attach travis to this so our dockerhub is updated as i update the environment. Can you kindly help me understand if this is possible?!

Thank you!! :)

Need to switch the primary branch of this repo from master to main. I still don't have permissions to do this, but should be easy to do.

@nkorinek

We are now 4 minor versions behind. We have been advised that 3.11 will break some crucial libraries, but that upgrading from 3.8 to 3.10 will help stay current

Hi there!

I got a problem in the pixel values when calculating hillshade with earthpy it rarely yields the same values than making the same operation with gdal or acrpy.
What is it due to?

Let's update this environment to

• use the jupyter jupyter/minimal-notebook
• update our envt to install packages needed including things like scikit, etc
• Leah and karen work together to add packages as needed to the envt file and docker image
• Make sure that all of the packages we need are there and such... and then karen can take over

next version - change the maintainer to "Earthlab Education [email protected]"

Currently when i make an update, i'm just adding new packages without considering versions. This is NOT a good idea as sometimes things update in a way that breaks other packages.

After this course is over, experiment with pinning each package version so that we have a set of known packages that work together at x version.

@mbjoseph -- do you have any thoughts on how i could setup testing (or could you help me??) so that we test that all packages import with a new version of this envt? I am not even sure how to start but i'd sure love to get a message prior to the students encountering errors as a part of the docker build or even BEFORE the docker build.

An alternative to having dockerhub build the image is to build it on travis. Not sure if this will be much faster but it could be?

It wouldn't solve the problem that you need to know to wait for the image to have been built and available on dockerhub before using it in hub-ops.

Thinking about this our ultimate envt will be launched from jupyter hub (i think) to support classes so it may be OK that this build currently breaks binder (still testing it).

The EA envt is really only powerful when attached to some data to work with unless there's a sneaky way to read stuff into memory. probably not ideal for our class.

I believe that a dockerfile pre-empts a yaml envt so that could be why the build is breaking.

Hello all,

I am new here 👋! Trying to get up to speed with things.

Is there a reason to have a Dockerfile that you created by hand? Depending on what it is needed for it might be easier to switch to http://repo2docker.readthedocs.io/ which takes care of creating containers for you (without you having to write a Dockerfile).

Right now it looks like that as a student all I need is the environment.yml and conda to get setup. In my experience student's are good at getting going with conda, but if we need them to use docker things get slower/more questions/mistakes.

If the reason to have a Dockerfile is so we can use this for JupyterHub and/or Binder I'd suggest ditching it because both JHub and mybinder.org use repo2docker, so we can generate the container image from the environment.yml.

hey @mbjoseph currently it seems like the EA dockerimage here

https://hub.docker.com/r/earthlab/earth-analytics/

only says earth-analytics. it's important this says earth-analytics-python as we have an R course too and i could see making several environments in the future. I dug around this morning and wasn't sure how to make that change however. can you help me understand how do to that?

Need to switch this repo to github actions so it no longer depends on travis CI.

It needs to include that git needs to be installed before following the instructions to install the conda environment. Otherwise, the install will fail.

currently we have removed nb_conda and nb_conda_kernels from our conda envt.
nb conda adds a nice interface to notebooks that allows students to explore environments on their computer, update packages etc.

https://github.com/Anaconda-Platform/nb_conda

While this is nice for teaching, i'm not clear how it interfaces in a jupyterhub environment because in that envt, presumably students don't have permissions to CHANGE their envt - correct? so i do think that exposing students to the packages and such is powerful. but can they change them in a jupyterhub implementation where nb_conda is available? i don't want that to happen.

i'm curious if @choldgraf bu chance has any thoughts on using nb_conda in a jupyterhub envt in the cloud. if we remove it from the envt then students will have to install it manually.

hey @betatim there is something fundamental that i'm struggling w still with repo2docker.
i just think it's not working properly on my computer. i'll let you work but if at any point if you have an idea as to why i've not been able to get this to run, please let me know. I'm getting a different error than i was prior

cu-biot-seec2-172:0-python lewa8222$ git clone https://github.com/earthlab/earth-analytics-python-env.git
Cloning into 'earth-analytics-python-env'...
remote: Counting objects: 54, done.
remote: Compressing objects: 100% (45/45), done.
remote: Total 54 (delta 28), reused 19 (delta 9), pack-reused 0
Unpacking objects: 100% (54/54), done.
cu-biot-seec2-172:0-python lewa8222$ cd earth-analytics-python-env/
cu-biot-seec2-172:earth-analytics-python-env lewa8222$ ls
Dockerfile	README.md	environment.yml
cu-biot-seec2-172:earth-analytics-python-env lewa8222$ jupyter-repo2docker https://github.com/earthlab/earth-analytics-python-env.git
Cloning into '/var/folders/43/4q82487d5xsfpxdx6nl_c1wmhckx08/T/repo2dockerr8_am1em'...
remote: Counting objects: 54, done.
remote: Compressing objects: 100% (45/45), done.
remote: Total 54 (delta 28), reused 19 (delta 9), pack-reused 0
Unpacking objects: 100% (54/54), done.
Using DockerBuildPack builder
Step 1/6 : FROM jupyter/scipy-notebook
Get https://registry-1.docker.io/v2/jupyter/scipy-notebook/manifests/latest: unauthorized: incorrect username or passwordcu-biot-seec2-172:earth-analytics-python-env lewa8222$ ls
Dockerfile	README.md	environment.yml
cu-biot-seec2-172:earth-analytics-python-env lewa8222$ jupyter-repo2docker https://github.com/jakevdp/PythonDataScienceHandbook
Cloning into '/var/folders/43/4q82487d5xsfpxdx6nl_c1wmhckx08/T/repo2dockernwswssvh'...
remote: Counting objects: 1623, done.
remote: Total 1623 (delta 0), reused 0 (delta 0), pack-reused 1623
Receiving objects: 100% (1623/1623), 96.08 MiB | 20.32 MiB/s, done.
Resolving deltas: 100% (1139/1139), done.
Using PythonBuildPack builder
Step 1/31 : FROM buildpack-deps:artful
Get https://registry-1.docker.io/v2/library/buildpack-deps/manifests/artful: unauthorized: incorrect username or passwordcu-biot-seec2-172:earth-analytics-python-env lewa8222$ 

We would like to have an automated build process on Docker Hub that rebuilds the Docker image any time the master branch is updated.

Describe the issue
I can't install the Earth Analytics Python Conda Environment. When I try to run conda env create -f environment.yml code in the Terminal, I get the following output:

Solving environment: failed

ResolvePackageNotFound:

• nbgrader

Desktop:

• OS: Windows 7, 32
• Browser Chrome
• Version 87.0.4280.66
• Conda version 4.8.3
• Python version 3.8.3.final.0

Look into using nbclean to hide specific parts of the lesson code. @lwasser

There are a few issues currently happening with our builds.

• tweepy and geocoder are conflicting with python v. 3.x as they both require 3.6 on conda forget but eh updated version of both (installed via pip) seems to be fine with the most currently v of python. I've since moved them to pip install via #46

The second issue is more curious:

when installing the envt i get this error:

urllib3-1.24.3       | 151 KB    | ########## | 100% 
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... Uninstalling jupyter-js-widgets jupyter-js-widgets/extension
Removing: /opt/conda/share/jupyter/nbextensions/jupyter-js-widgets

Enabling nb_conda_kernels...
Status: enabled

Enabling notebook extension jupyter-js-widgets/extension...
      - Validating: OK

++ /opt/conda/bin/python -c 'import logging; from jupyter_contrib_core.notebook_compat.nbextensions import install_nbextension_python; install_nbextension_python('\''jupyter_highlight_selected_word'\'', sys_prefix=True, logger=logging.getLogger())'

++ /opt/conda/bin/python -c 'import logging; from jupyter_contrib_core.notebook_compat.nbextensions import install_nbextension_python; install_nbextension_python('\''latex_envs'\'', sys_prefix=True, logger=logging.getLogger())'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/lib/python3.7/site-packages/notebook/nbextensions.py", line 211, in install_nbextension_python
    m, nbexts = _get_nbextension_metadata(module)
  File "/opt/conda/lib/python3.7/site-packages/notebook/nbextensions.py", line 1122, in _get_nbextension_metadata
    m = import_item(module)
  File "/opt/conda/lib/python3.7/site-packages/traitlets/utils/importstring.py", line 42, in import_item
    return __import__(parts[0])
  File "/opt/conda/lib/python3.7/site-packages/latex_envs/__init__.py", line 3, in <module>
    from . import latex_envs
  File "/opt/conda/lib/python3.7/site-packages/latex_envs/latex_envs.py", line 20, in <module>
    from nbconvert.exporters.exporter import Exporter
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/__init__.py", line 7, in <module>
    from . import postprocessors
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/postprocessors/__init__.py", line 5, in <module>
    from .serve import ServePostProcessor
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/postprocessors/serve.py", line 19, in <module>
    class ProxyHandler(web.RequestHandler):
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/postprocessors/serve.py", line 21, in ProxyHandler
    @web.asynchronous
AttributeError: module 'tornado.web' has no attribute 'asynchronous'

failed
ERROR conda.core.link:_execute(543): An error occurred while installing package 'conda-forge::jupyter_latex_envs-1.4.4-py37_1000'.
LinkError: post-link script failed for package conda-forge::jupyter_latex_envs-1.4.4-py37_1000
running your command again with `-v` will provide additional information
location of failed script: /opt/conda/bin/.jupyter_latex_envs-post-link.sh
==> script messages <==
++ /opt/conda/bin/python -c 'import logging; from jupyter_contrib_core.notebook_compat.nbextensions import install_nbextension_python; install_nbextension_python('\''latex_envs'\'', sys_prefix=True, logger=logging.getLogger())'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/lib/python3.7/site-packages/notebook/nbextensions.py", line 211, in install_nbextension_python
    m, nbexts = _get_nbextension_metadata(module)
  File "/opt/conda/lib/python3.7/site-packages/notebook/nbextensions.py", line 1122, in _get_nbextension_metadata
    m = import_item(module)
  File "/opt/conda/lib/python3.7/site-packages/traitlets/utils/importstring.py", line 42, in import_item
    return __import__(parts[0])
  File "/opt/conda/lib/python3.7/site-packages/latex_envs/__init__.py", line 3, in <module>
    from . import latex_envs
  File "/opt/conda/lib/python3.7/site-packages/latex_envs/latex_envs.py", line 20, in <module>
    from nbconvert.exporters.exporter import Exporter
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/__init__.py", line 7, in <module>
    from . import postprocessors
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/postprocessors/__init__.py", line 5, in <module>
    from .serve import ServePostProcessor
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/postprocessors/serve.py", line 19, in <module>
    class ProxyHandler(web.RequestHandler):
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/postprocessors/serve.py", line 21, in ProxyHandler
    @web.asynchronous
AttributeError: module 'tornado.web' has no attribute 'asynchronous'


Attempting to roll back.

Rolling back transaction: ...working... done

LinkError: post-link script failed for package conda-forge::jupyter_latex_envs-1.4.4-py37_1000
running your command again with `-v` will provide additional information
location of failed script: /opt/conda/bin/.jupyter_latex_envs-post-link.sh
==> script messages <==
++ /opt/conda/bin/python -c 'import logging; from jupyter_contrib_core.notebook_compat.nbextensions import install_nbextension_python; install_nbextension_python('\''latex_envs'\'', sys_prefix=True, logger=logging.getLogger())'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/lib/python3.7/site-packages/notebook/nbextensions.py", line 211, in install_nbextension_python
    m, nbexts = _get_nbextension_metadata(module)
  File "/opt/conda/lib/python3.7/site-packages/notebook/nbextensions.py", line 1122, in _get_nbextension_metadata
    m = import_item(module)
  File "/opt/conda/lib/python3.7/site-packages/traitlets/utils/importstring.py", line 42, in import_item
    return __import__(parts[0])
  File "/opt/conda/lib/python3.7/site-packages/latex_envs/__init__.py", line 3, in <module>
    from . import latex_envs
  File "/opt/conda/lib/python3.7/site-packages/latex_envs/latex_envs.py", line 20, in <module>
    from nbconvert.exporters.exporter import Exporter
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/__init__.py", line 7, in <module>
    from . import postprocessors
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/postprocessors/__init__.py", line 5, in <module>
    from .serve import ServePostProcessor
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/postprocessors/serve.py", line 19, in <module>
    class ProxyHandler(web.RequestHandler):
  File "/opt/conda/lib/python3.7/site-packages/nbconvert/postprocessors/serve.py", line 21, in ProxyHandler
    @web.asynchronous
AttributeError: module 'tornado.web' has no attribute 'asynchronous'

this error seems to be related to using the jupyter/scipy docker container as the jupyter/minimal-notebook seems to work ok. but i'm not sure how to trouble shoot the above.

Investigate options for making the built docker image smaller.

@nkorinek these are the packages I don't think we use:

• pyqt
• tqdm
• kiwisolver
• plotly (we could discuss this...I know some people like it, but I can just never understand how they are supposed to work business- and license-wise so I don't love teaching it to students)
• pysal (maybe we should be using this? I think most of what it does we handle through rioxarray and scikit-learn)
• geopy
• descartes
• contextily
• mapboxgl
• geocoder
• tweepy
• netcdf4 (I think rioxarray covers this? but maybe it's needed as a back end)
• nltk
• textblob

As far as the jupyter environment, I am not sure if we need:

• autopep8 (really I'm just not sure it works with the new jupyter release?)
• ipython
• jupyter_contrib_nbextensions (again, I don't think this works with the new jupyter extension system)
• nbclean
• nbresuse

I've also removed matplotcheck. But we maybe should put that back in, because I have some testing utilities that I'm copying and pasting around that would be nice to automate, and that package might be a good spot to keep that stuff.

In addition:

• rasterstats could probably be replaced by zonal_statistics from xarray-spatial and built-in xarray interpolation tools
• regionmask is not super active. But it works last I checked, and I don't know of anything else that does the same thing.

For some lessons we want to facilitate remote reads of datasets (e.g., reading GeoJSON from a URL). To do this we need to pin GeoPandas to something greater than or equal to 0.4.0.

to force an envt rebuild on docker hub i'm pinning nbgrader to version 0.5.6

#56

however, we will want to unpin this in the future once we have the newest version just to avoid future dep issues.

Environment users would like to be able to use the earth-analytics-python-env on Google Colab without having to wait for more than a minute or two each time.

This will be accomplished using conda constructor. Constructor generates a shell script that can be used to quickly duplicate a conda environment (it saves the environment solution). Using the condacolab package, users will be able to install the environment from a github download url.

It would be good to verify that the packages installed in this environment and used in lessons can be imported. Doing this in the CI build would require:

1. Fetching the lessons
2. Using modulefinder to generate a set of imported modules
3. Iterating over that set to verify that modules can be imported

If any module cannot be imported, the build is broken.

Currently we suggest source activate my-env to activate the environment, but Windows users should just do activate my-env.

Use new functions added to earthpy through earthpy.plot to make notebooks easier. For example, in the intro to lidar raster landing page, there is a code block that looks like this:

# create histogram
fig, ax = plt.subplots(figsize=(10, 6))
data = diff_chm_crop.ravel()[~np.isnan(diff_chm_crop.ravel())]
ax.hist(data, bins=20, range=(-10, 10), color='purple', edgecolor='k')

# set titles & display properties
ax.set_facecolor('w')
ax.spines['left'].set_color('k')
ax.spines['bottom'].set_color('k')
ax.set(xlabel='Elevation (m)', ylabel='Frequency',
       title='Homework plot 1: \n Histogram of Pre/Post Flood CHM Difference\n post flood minus pre flood')
plt.show()

I am fairly certain this could be replaced with

import earthpy.plot as ep
ep.hist(diff_chm_crop, bins=20, colors=["purple"], title=''Homework plot 1: \n Histogram of Pre/Post Flood CHM Difference\n post flood minus pre flood")

It may be useful not to utilize the functions so the students get used to wrestling with matplotlib fig and ax, but it would certainly be more effecient. Let me know thoughts! @lwasser

Looks like there have been some conda updates that cause issues with appveyor building

look into the fix following the issue above.

example commit that fixed things?? conda/conda@44557fc

Describe the issue
earth analytics python environment setup for Mac M1 does not automate currently. I manually installed the environment instead. Here are some changes that were required from the automated instructions:

1. miniconda did not install correctly. I switched to miniforge, and intstalled it using the command brew install miniforge
2. Conda packages didn't install smoothly from the environment.yml file with the command conda env create --debug -f environment.yml. Instead, I individually installed each package. The majority of packages required the pip installer instead of the conda installer. Here are my notes on which packages installed with installer.

# dependencies:
# python=3.10.13 # previously specified python 3.8
# pip # already installed

# Core scientific python
numpy # conda
pyqt # conda
tqdm # conda
kiwisolver # conda

# Plotting
matplotlib # conda
plotly # conda
seaborn # conda

# Spatial packages
pysal # conda
pyproj>=3.4 # conda, package was previously specified as 3.0
rasterstats # pip3
geopy # pip3
cartopy # conda
descartes # conda
contextily # DID NOT INSTALL, COULD NOT SOLVE
earthpy # pip3
folium # conda
geojson # conda
mapboxgl # pip3
hydrofunctions # pip3
geocoder # pip3
tweepy # pip3
xarray # pip3
rioxarray>=0.3.0 # pip3
scipy # pip3
netcdf4 # pip3
nc-time-axis # pip3
regionmask # pip3

# Natural language processing
nltk # pip3
textblob # pip3

# Jupyter Environment
papermill # pip3
autopep8 # pip3
jupyterlab # pip3
notebook # pip3
ipython # pip3
jupyter_contrib_nbextensions # pip3
nbclean # pip3
nbresuse # pip3

# Autograding
matplotcheck>=0.1.3 # pip3
nbgrader # pip3

# Added by jkinsler
geopandas # pip3
git+https://github.com/earthlab/earthpy@apppears # newest appears library
chardet # pip3
hvplot # pip3

Desktop (please complete the following information):

• OS: iOS 13.5.1 for Mac M1
• Browser chrome

• conda install -c conda-forge plotly # interactive plot goodness
• pip install hydrofunctions # api calls to get data from NWIS. pretty sweet little pakage.