Earth-2 Model Intercomparison Project (MIP) is a python framework that enables climate researchers and scientists to inter-compare AI models for weather and climate.
Home Page: https://nvidia.github.io/earth2mip/
License: Apache License 2.0
New Feature
Critical (currently preventing usage)
-this includes updating the packages to new checkpoint system
No response
New Feature
Critical (currently preventing usage)
Add ability for data-sources to cache the data locally on the machine and reuse it, instead of always downloading.
No response
na
No response
https://nvidia.github.io/earth2mip/api.html should show some autodocs
No response
source -main
No response
if I try to run the ensemble notebook with pangu as the weather model I get
File "/usr/local/lib/python3.10/dist-packages/earth2mip/inference_ensemble.py", line 135, in run_ensembles
iterator = model(time, x, normalize=False)
TypeError: PanguInference.call() got an unexpected keyword argument 'normalize'
interactive session with this container
`gitlab-master.nvidia.com/earth-2/fcn-mip:latest`New Feature
Critical (currently preventing usage)
Currently, GFS only supports loading from the analysis (initial condition), but it's simple to also work with forecasts. We can add a forecast step configuration.
No response
Critical (currently preventing usage)
New
No response
New Feature
Critical (currently preventing usage)
For exploring ensemble forecast and for exploring the physical validity of these models (and other research goals) a method that can inject a perturbation with units of variables / dt (here dt is the model tilmestep) between model steps is needed.
I would like to collect suggestions of how to do that , different than PR #42 before starting to code
see changes in #42
New Feature
Critical (currently preventing usage)
Add support for diagnostic models. More information coming
No response
source
Source
DLWP model registry does not work with the one generated by the notebook. Needs env variables to get set up.
No response
New Feature
Medium
Add apptainer support for running examples for those that cant run docker.
No response
source
No response
Right now notebook one demo a CLI command. This can cause some issues when relying on a ipython kernel in some Jupyter lab environments.
No response
main
No response
I get this error sometimes
INFO:/root/earth2mip/earth2mip/model_registry.py:Downloading Pangu 6hr / 24hr model checkpoints, this may take a bit
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/root/earth2mip/earth2mip/inference_medium_range.py", line 301, in <module>
main()
File "/root/earth2mip/earth2mip/inference_medium_range.py", line 279, in main
model = _cli_utils.model_from_args(args, dist.device)
File "/root/earth2mip/earth2mip/_cli_utils.py", line 41, in model_from_args
return networks.get_model(args.model, device=device, metadata=metadata)
File "/root/earth2mip/earth2mip/networks/__init__.py", line 337, in get_model
package = registry.get_model(model)
File "/root/earth2mip/earth2mip/model_registry.py", line 222, in get_model
return self.get_builtin_model(name)
File "/root/earth2mip/earth2mip/model_registry.py", line 237, in get_builtin_model
return PanguPackage(self.get_path(name), seperator=self.SEPERATOR)
File "/root/earth2mip/earth2mip/model_registry.py", line 186, in PanguPackage
urllib.request.urlretrieve(
File "/usr/lib/python3.10/urllib/request.py", line 280, in urlretrieve
raise ContentTooShortError(
urllib.error.ContentTooShortError: <urlopen error retrieval incomplete: got only 38451195 out of 1181711187 bytes>
No response
main
No response
source to normalized values: earth2mip/earth2mip/networks/__init__.py
Line 242 in 4d4c98e
source should be applied on physical units, for the same reasons that we decided to do this with the initial perturbation. Presumably the source will also need to know about channel names in order to perturb the correct variable.
No response
Current
No response
https://github.com/NVIDIA/earth2mip/blob/main/earth2mip/inference_ensemble.py#L334
Not sure why this is.
No response
No response
No response
main
No response
Presently GFS does not support Pangu channel set. Should probably just add it with the channel set for now and update to more general schema later
No response
This is needed to ensure the examples actually work.
Critical (currently preventing usage)
Update
Improve config documentation to show options and also some more samples than what the notebooks provides
Current
No response
GFS data source is presently not running when you pip install. The JSON files are not included:
File "<dir>/conda-envs/earth2mip/lib/python3.10/pathlib.py", line 1119, in open
return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: `<dir>/conda-envs/earth2mip/lib/python
3.10/site-packages/earth2mip/datasets/gfs/data73var.json'
No response
New Feature
Critical (currently preventing usage)
Need to get both climate net and the precip model to the NGC model registry and get it hooked up
No response
For legacy reasons, inference_ensemble requires methods/attributes not present in the TimeLoop abstraction, and is tightly coupled to earth2mip.networks.Inference. The main source of coupling is related to normalization.
To get around this, the perturbation function should happen in physical units not in normalized values. Here are the steps that need to happen:
To do this:
earth2mip/earth2mip/inference_ensemble.py
Line 231 in 5999d7e
earth2mip/earth2mip/inference_ensemble.py
Line 261 in 5999d7e
Is that clear?
New Feature
Medium
Enable static type checking in repo.
No response
main
No response
Scipy import is not used in DLWP script. Left over from converting the pangu script
No response
main
No response
we should add coverage for these use case. mypy should be able to catch the first (#52). the second requires a test to assert the shape is as expected at run time.
No response
New Feature
Medium
Add a download function inside models that can get called on load for known model packages that are not served on s3.
This should help enable better CLI use of built in models.
No response
to allow custom time looping in an extensible manner we need a new abstraction. #42 for example requires touching both model code and inference_ensemble---these routines are too tightly coupled. Exposing an iterator interface as TimeLoop does is not enough for use-cases like diagnostic models or time dependent forcing which works for various models.
To control the timestep we should define a TimeStepper abstraction. Does this name seem okay?
Here is a proposed interface:
State = Tuple[Tensor, datetime]
Outputs = Tensor
class TimeStepper:
"""Callable time stepper
Updates the state and returns some outputs.
The state contains time and a tensor of data representing the domain.
The outputs are diagnostics outputs not required for time-stepping, but useful.
Attributes:
state_metadata: information about the state object, dimensions of the tensor, channels, grid, etc
output_metadata: information about the outputs, dimensions etc.
Examples:
stepper : TimeStepper = ...
state = get_initial_condition(time)
channel_to_save = "tcwv"
for i in range(n):
state, output = stepper(state)
idx = stepper.output_metadata.index(channel_to_save)
tcwv = state.data[idx]
plt.pcolormesh(tcwv)
plt.savefig(f"{i}.png")
"""
def __call__(self, state: State) -> Tuple[State, Outputs]:
thoughts @NickGeneva?
New Feature
Medium
Add AFNO model from Modulus
No response
Critical (currently preventing usage)
New
Literally first sentence "is a python"
main
Pip
I get this bug
In [3]: >>> import datetime
...: >>> from earth2mip.networks import get_model
...: >>> from earth2mip.initial_conditions import cds
...: >>> from earth2mip.inference_ensemble import run_basic_inference
...: >>> time_loop = get_model("e2mip://fcn", device="cuda:0")
...: >>> data_source = cds.DataSource(time_loop.in_channel_names)
...: >>> ds = run_basic_inference(time_loop, n=10, data_source=data_source, time=datetime.datetime(2018, 1, 1))
...: >>> ds.chunk()
File ~/.local/lib/python3.10/site-packages/modulus/models/afno/afno.py:372, in PatchEmbed.forward(self, x)
369 def forward(self, x: Tensor) -> Tensor:
370 B, C, H, W = x.shape
371 assert (
--> 372 H == self.img_size[0] and W == self.img_size[1]
373 ), f"Input image size ({H}*{W}) doesn't match model ({self.img_size[0]}*{self.img_size[1]})."
374 x = self.proj(x).flatten(2).transpose(1, 2)
375 return x
AssertionError: Input image size (721*1440) doesn't match model (720*1440).
this is probably a bug in run_basic_inference.
No response
Current
Source
https://github.com/NVIDIA/earth2mip/blob/main/examples/workflows/pangu_24.py#L24
Update load import
No response
No response
No response
Current
Source
Instead of making Pangu have the same defunct interface as earth2mip.networks.Inference, run_inference should be modified to work with objects obeying the earth2mip.time_loop.TimeLoop interface.I
No response
No response
No response
Medium
New
No response
New Feature
Critical (currently preventing usage)
When MODEL_REGISTRY is not set, e2mip will default to downloading models to "/", usually resulting in a permission error.
We can probably chose a default in the users home directory. .e.g .cache/e2mip/model-registry.
No response
Current
No response
https://docs.pydantic.dev/2.3/migration/#basesettings-has-moved-to-pydantic-settings
No response
main
No response
No response
Current
Source
Glob load of netcdf files is not ordered.
If the load occurs out of order this can lead to issues such that the 0th loaded netCDF does not actually correspond to the 0th netCDF file saved.
No response
No response
No response
Fix or delete the "netcdf" option.
Originally posted by @nbren12 in #68 (comment)
Critical (currently preventing usage)
New
Improve singularity docs for running examples with bind location, pip install command, remove unnecessary flags, etc.
main
No response
Known bug at the moment due to graph cast addition / upstreamed fixes needed from Modulus. Documenting here for users with temporary fix below when installing from source.
pip install .
pip install -r requirements.txt
No response
main
No response
https://github.com/NVIDIA/modulus/blame/ef54c7a4c3b241c48f48695362c2688d150c6ce5/modulus/models/afno/afno.py#L414 changed the kwargs of AFNO, so the fcn package now fails. with this error
__________________________________________________________________________________________________________________________________________________________________________________________________________________ test_run_basic_inference ____________________________________________________________________________________________________________________________________________________________________________________________________________________
@pytest.mark.slow
def test_run_basic_inference():
> time_loop = get_model("e2mip://fcn", device="cuda:0")
tests/test_end_to_end.py:134:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
earth2mip/networks/__init__.py:335: in get_model
return _load_package_builtin(package, device, name=url.netloc)
earth2mip/networks/__init__.py:282: in _load_package_builtin
return inference_loader(package, device=device)
earth2mip/networks/fcn.py:64: in load
core_model = modulus.Module.from_checkpoint(package.get("fcn.mdlus"))
/usr/local/lib/python3.10/dist-packages/modulus/models/module.py:327: in from_checkpoint
model = cls.instantiate(args)
/usr/local/lib/python3.10/dist-packages/modulus/models/module.py:150: in instantiate
return _cls(**arg_dict["__args__"])
/usr/local/lib/python3.10/dist-packages/modulus/models/module.py:59: in __new__
bound_args = sig.bind_partial(
/usr/lib/python3.10/inspect.py:3193: in bind_partial
return self._bind(args, kwargs, partial=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <Signature (self, inp_shape: List[int], in_channels: int, out_channels: int, patch_size: List[int] = [16, 16], embed_d... float = 0.0, num_blocks: int = 16, sparsity_threshold: float = 0.01, hard_thresholding_fraction: float = 1.0) -> None>, args = (None,), kwargs = {'img_size': [720, 1440]}
def _bind(self, args, kwargs, *, partial=False):
"""Private method. Don't use directly."""
arguments = {}
parameters = iter(self.parameters.values())
parameters_ex = ()
arg_vals = iter(args)
while True:
# Let's iterate through the positional arguments and corresponding
# parameters
try:
arg_val = next(arg_vals)
except StopIteration:
# No more positional arguments
try:
param = next(parameters)
except StopIteration:
# No more parameters. That's it. Just need to check that
# we have no `kwargs` after this while loop
break
else:
if param.kind == _VAR_POSITIONAL:
# That's OK, just empty *args. Let's start parsing
# kwargs
break
elif param.name in kwargs:
if param.kind == _POSITIONAL_ONLY:
msg = '{arg!r} parameter is positional only, ' \
'but was passed as a keyword'
msg = msg.format(arg=param.name)
raise TypeError(msg) from None
parameters_ex = (param,)
break
elif (param.kind == _VAR_KEYWORD or
param.default is not _empty):
# That's fine too - we have a default value for this
# parameter. So, lets start parsing `kwargs`, starting
# with the current parameter
parameters_ex = (param,)
break
else:
# No default, not VAR_KEYWORD, not VAR_POSITIONAL,
# not in `kwargs`
if partial:
parameters_ex = (param,)
break
else:
msg = 'missing a required argument: {arg!r}'
msg = msg.format(arg=param.name)
raise TypeError(msg) from None
else:
# We have a positional argument to process
try:
param = next(parameters)
except StopIteration:
raise TypeError('too many positional arguments') from None
else:
if param.kind in (_VAR_KEYWORD, _KEYWORD_ONLY):
# Looks like we have no parameter for this positional
# argument
raise TypeError(
'too many positional arguments') from None
if param.kind == _VAR_POSITIONAL:
# We have an '*args'-like argument, let's fill it with
# all positional arguments we have left and move on to
# the next phase
values = [arg_val]
values.extend(arg_vals)
arguments[param.name] = tuple(values)
break
if param.name in kwargs and param.kind != _POSITIONAL_ONLY:
raise TypeError(
'multiple values for argument {arg!r}'.format(
arg=param.name)) from None
arguments[param.name] = arg_val
# Now, we iterate through the remaining parameters to process
# keyword arguments
kwargs_param = None
for param in itertools.chain(parameters_ex, parameters):
if param.kind == _VAR_KEYWORD:
# Memorize that we have a '**kwargs'-like parameter
kwargs_param = param
continue
if param.kind == _VAR_POSITIONAL:
# Named arguments don't refer to '*args'-like parameters.
# We only arrive here if the positional arguments ended
# before reaching the last parameter before *args.
continue
param_name = param.name
try:
arg_val = kwargs.pop(param_name)
except KeyError:
# We have no value for this parameter. It's fine though,
# if it has a default value, or it is an '*args'-like
# parameter, left alone by the processing of positional
# arguments.
if (not partial and param.kind != _VAR_POSITIONAL and
param.default is _empty):
raise TypeError('missing a required argument: {arg!r}'. \
format(arg=param_name)) from None
else:
if param.kind == _POSITIONAL_ONLY:
# This should never happen in case of a properly built
# Signature object (but let's have this check here
# to ensure correct behaviour just in case)
raise TypeError('{arg!r} parameter is positional only, '
'but was passed as a keyword'. \
format(arg=param.name))
arguments[param_name] = arg_val
if kwargs:
if kwargs_param is not None:
# Process our '**kwargs'-like parameter
arguments[kwargs_param.name] = kwargs
else:
> raise TypeError(
'got an unexpected keyword argument {arg!r}'.format(
arg=next(iter(kwargs))))
E TypeError: got an unexpected keyword argument 'img_size'
nvidia-modulus @ https://github.com/nbren12/modulus/archive/refs/heads/tisr.tar.gz#sha256=84f7044c4cb030d30aa1b4a55ca25af90fe7bdb45252c3e7738896274dcea609Critical (currently preventing usage)
New
Update PR templates to include check for linking to issue
Update issue template:
main
No response
The model data is downloading by all ranks of a parallel job in parallel. This will result in corrupt files if the files haven't been download into the model registry yet.
No response
New Feature
Critical (currently preventing usage)
This prescribed nudging is applied between steps, with units of var/day, where var could be dimensional or non dimensional depending on the model. A notebook for preforming a perturbation response experiment in earth2mip is added.
No response
Current
Source
Add missing deps needed.
No response
No response
No response
New Feature
Medium
The File system should support the use of NGC model registry natively and download files with urls such as ngc://model_url. The existing NGC packages should then be cleaned up to use this feature.
This includes then cleaning up the built in packages.
Some additional discussion is found here: #55
No response
source
No response
def Nvidia_cmap():
colors = [(0.0, "white"), (1.0, "darkgreen")] # (position, color)
cmap = mcolors.LinearSegmentedColormap.from_list("custom_cmap", colors)
return cmap
Change to
def Nvidia_cmap():
colors = ['#456115', '#bfda8e'] # (position, color)
cmap = mcolors.LinearSegmentedColormap.from_list("custom_cmap", colors)
return cmap
No response
New Feature
Critical (currently preventing usage)
Currently the code applied an initial condition perturbation to all variables in a model. To allow a full exploration of the model it is useful to allow to perturb a subset of the variables, that if config specified for reproducibility. This is need for us and some of our collaborators to be able to migrate to using this code.
No response
New Feature
Critical (currently preventing usage)
Add this version of noise back into Earth-2 MIP
No response
main
Source
Presently Earth-2 MIP fails for CPU runs because of some hard coded cuda() commands which should be swapped for .to(device) which should allow for CPU inference.
No response
New Feature
Critical (currently preventing usage)
To account for model uncertainties in ensemble forecasting we need to enable perturbing the model. The simplest way to achieve this is injecting noise between steps. It is important that the type of noise injection and amplitude will be determined from the config file.
No response
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.