Desktop Application for Model Execution (DAME) is a command line interface for executing environmental models
Home Page: https://dame-cli.readthedocs.io/en/latest/
License: Apache License 2.0
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
This issue is to update the documentation with the end of may release.
Executing dame browse opens up a browser, which does not work if you are in a server. That command should show a list of the model configs and setups.
I want to test all the modelconfigurationsetups automatically.
If a modelconfigurations needs information about the inputs or parameters, ignore it.
This functionality does not work anymore
At the moment, when you do dame run <setup id>, you don't know the size of the files and Docker image that you are going to download. It would be great to show:
And if above a certain size (500MB), ask for user confirmation. Example of desired behavior:
dame run cycles-0.10.2-alpha-collection-oromia-single-point
The information of the setup is complete
This setup requires 500MB (Docker image) + 30 Mb (input files). Proceed? Y/n: Y
Executing the setup
To run this model configuration, a daily-weather file (.tar.gz file) is required.
Please enter a url for it:
Please enter a url for it: sad
To run this model configuration, a monthly-weather file (.tar.gz file) is required.
Please enter a url for it: afdasfThis issue needs to be discussed, @sirspock.
The context is that if a model needs to be calibrated, DAME should provide the means to do so through DAKOTA. It is unclear to me at the moment how this workflow for this will be, because DAME was designed as a testing tool. Maybe this is to start testing the calibration of a model and show how to set it up.
Describe the bug
If you don't have singularity installed, the CLI will show that the setup is executing and then hang. It should fail.
From Yolanda's feedback, we should change the following messages:
The information of the setup is complete
Execution line
cd executions/hand_v2_raster_706c74de-835b-11ea-9d3d-f8f21e3c1558/hand_v2_mint_component/src
/usr/bin/singularity exec docker://mintproject/hand:v2.1.0 ./run -i1 Travis-DEM-10m-HUC120902050408buf.tif -o1 distance-down.tif -o3 shape.shp -o4 geojson.json -o2 distance-down-raster.tif -p1 500
Do you want to run the setup? [Y/n]:Y
Should be changed to:
The information needed to run the model is complete, and I can execute the model as follows:
Invocation command:
cd executions/hand_v2_raster_706c74de-835b-11ea-9d3d-f8f21e3c1558/hand_v2_mint_component/src
/usr/bin/singularity exec docker://mintproject/hand:v2.1.0 ./run -i1 Travis-DEM-10m-HUC120902050408buf.tif -o1 distance-down.tif -o3 shape.shp -o4 geojson.json -o2 distance-down-raster.tif -p1 500
Do you want to proceed and submit it for execution? [Y/n]:Y
If for some reason you want to mess up and test with the parameter values in DAME, the parameter positions should be shown in the invocation as well. For example, right now DAME shows:
Parameters
- weed_fraction: 0.05
- use_forcing: FALSE
- end_planting_day: 149
- end_year: 2017
(other parameters omitted for brevity)What it should show:
Parameters
- weed_fraction (p1): 0.05
- use_forcing (p2): FALSE
- end_planting_day (p3): 149
- end_year (p4): 2017
(other parameters omitted for brevity)Where p1..p4 are the parameters that will be shown in the invocation command. E.g.,:
-i2 crops.crop -o1 cycles_soilProfile.dat -o9 cycles_outputs.txt -o3 cycles_crop.dat -o5 cycles_season.dat -o2 cycles_som.dat -o8 cycles_water.dat -o4 cycles_nitrogen.dat -o7 cycles_weatherOutput.dat -o6 cycles_summary.dat -p4 100 -p2 2017 -p6 0 -p3 Maize -p5 149 -p8 FALSE -p7 0.05 -p1 2000```
Is your feature request related to a problem? Please describe.
I proposed the following approach
$ mint data upload outputs.txt
Please, enter the information related to the resource: *outputs.txt*
Enter the license (default: apache):
Enter the credit:
Enter the citation:
Enter the funding:
Enter the data quality:
$ ls directory/
file1 file2 file3
$ mint data upload directory/
Please, enter the information related to the resource: *file1*
Enter the license (default: apache):
Enter the credit:
Enter the citation:
Enter the funding:
Enter the data quality:
Adding the file2
Do you want to reuse the info from the previous file? [y/N]
(user enter y)
Adding the file3
Do you want to reuse the info from the previous file? [y/N]
(user enter N )
Enter the license (default: apache):
Enter the credit:
Enter the citation:
Enter the funding:
Enter the data quality:
(.venv) node1 ~ mint setup run hand_v2_travis
Exception when calling ModelConfigurationApi->custom_modelconfigurations_id_get: (404)
Reason: NOT FOUND
HTTP response headers: HTTPHeaderDict({'Server': 'nginx/1.16.1', 'Date': 'Fri, 28 Feb 2020 17:27:04 GMT', 'Content-Type': 'application/json', 'Content-Length': '12', 'Connection': 'keep-alive', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Methods': 'GET, POST, OPTIONS, PUT', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range', 'Access-Control-Expose-Headers': 'Content-Length,Content-Range'})
HTTP response body: "Not found"
Traceback (most recent call last):
File "/data/users/runner/.venv/bin/mint", line 8, in <module>
sys.exit(cli())
File "/data/users/runner/.venv/lib/python3.6/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/data/users/runner/.venv/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/data/users/runner/.venv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/data/users/runner/.venv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/data/users/runner/.venv/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/data/users/runner/.venv/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/data/users/runner/.venv/lib/python3.6/site-packages/mint/__main__.py", line 105, in run
edit_inputs_setup(setup)
File "/data/users/runner/.venv/lib/python3.6/site-packages/mint/cli_methods.py", line 16, in edit_inputs_setup
for _input in model_configuration.has_input:
AttributeError: 'NoneType' object has no attribute 'has_input'
Do a review of the documentation and usage instructions in Windows.
Is your feature request related to a problem? Please describe.
Support run the containers using Docker.
Describe the solution you'd like
The user can choose Docker to run the containers
As we have seen in #50 Singularity has issues when converting Docker images in macOS. An alternative would be to build the singularity images in unix so Singularity does not have to build them. This requires:
Some models don't have setups with examples, such as the latest version of TopoFlow.
DAME documentation should guide users on how to find data for TopoFlow, e.g., using the data catalog or any other pointer.
By March 30th we need to generate documentation on how to install and test models from your laptop using mint cli. We can use HAND as a test case and TopoFlow as an advanced setup.
The tutorial should explain how to install everything with Singularity and configure it so we can run it locally.
Return a link to mint-ui
Describe the bug
dame run 7d9ac3ba-c575-467e-9183-e85711b0f8d8
Error:
Traceback (most recent call last):
File "/data/users/runner/.venv/lib/python3.6/site-packages/dame/cli_methods.py", line 136, in execute_setups
status, file_dir = execute_setup(setup_files)
File "/data/users/runner/.venv/lib/python3.6/site-packages/dame/executor.py", line 108, in execute_setup
raise e
File "/data/users/runner/.venv/lib/python3.6/site-packages/dame/executor.py", line 106, in execute_setup
setup_cmd_line, cwd_path = build_command_line(setup_dict, execution_dir_path)
File "/data/users/runner/.venv/lib/python3.6/site-packages/dame/executor.py", line 83, in build_command_line
l = build_input(inputs, src_path)
File "/data/users/runner/.venv/lib/python3.6/site-packages/dame/executor.py", line 31, in build_input
position = _input["position"][0]
KeyError: 'position'
This setup corresponds to https://rdfexplorer.mint.isi.edu/page/okn/i/mint/7d9ac3ba-c575-467e-9183-e85711b0f8d8
This setup was created automatically with the portal. It is equivalent to others like https://rdfexplorer.mint.isi.edu/page/okn/i/mint/a044daca-eb62-4ef9-b806-508e0cee27fa
All input/outputs have position (I checked)
Before, we had the ability of doing dame setup list, but now we can't anymore.
In my case, I cannot do dame browse because I execute the cli through a virtualized Ubuntu console (no browser available).
I think doing dame list should return a list of all configurations and setups available. Filtering by label as we did before was also great. I hope to see this in a future release.
Is your feature request related to a problem? Please describe.
The subcommand must download the outputs
mint execution download
Describe the solution you'd like
Download the csv for each thread
curl https://ingestion.mint.isi.edu/data/PvjIvVMa4ELIQQcDYgIM/summary.json \
| jq -r '.thread.results'
Convert to json and get the link
curl https://ingestion.mint.isi.edu/data/PvjIvVMa4ELIQQcDYgIM/summary.json \
| jq -r '.thread.results[].inputs_outputs_csv'
The directory structure must be:
{model_name}_{region}/{thread_id}/inputs/
{model_name}_{region}/{thread_id}/outputs/
{model_name}_{region}/{thread_id}/summary.json
{model_name}_{region}/{thread_id}/metadata.json
{model_name}_{region}/{thread_id}/{csvname}@sirspock, this will add new dependencies in the model catalog and will need to be discussed in detail. This new functionality implies the execution of data transformation and postprocessing steps for a model.
Assuming this information is available in the knowledge base, DAME will be able to invoke the transformation pipeline (dockerized) to produce inputs to be used by the model. Initially this will be done in 2 steps: the user activates the transformation giving the original input; and then executes the model with DAME.
When DAME sets up an execution, indicates where to find the log.
I would like that, if requested with a flag such as --v (verbose) it printed basic information about the sources and parameters being used; the link to the component location, the docker image being used and the invocation command used for singularity.
Example:
> dame run <setup> --v
Inputs used:
precipitation_rates: "http://example.com/precip.csv"
thresholds:100
component_location: http://github/my/component.zip
software_image:mintproject/hand:latest
singularity execution command: docker run -it -v"vol":out ...
Instead of URLs for missing configuration inputs. Useful when testing models locally.
Describe the bug
Reported by @varunratnakar :
Running with the example configuration in Usage notes ( modflow )
- It asks "Please, enter the url of the previous input:". It wasn't very clear what this previous input is.
Action
Clarify the description of the requested inputs
Is your feature request related to a problem? Please describe.
The user can:
We would like to search for relevant datasets for those model configurations that do not have any files available (e.g., TopoFlow). This can be done by asking the data catalog to list datasets that are available through their API. @dnfeldman will refine the client in https://github.com/mintproject/data-catalog-client to make sure it is updated and we can use it for querying.
Right now DAME works against [email protected] graph. This should be configurable (some setups in Texas may not exist in MINT)
Version: DAME: v4.1.1
input: dame browse --help
Result:
WARNING: You are using wcm version 4.1.0, however version 4.1.1 is available.
You should consider upgrading via the 'pip install --upgrade wcm' command.
Why does wcm appears?
Also, dame version --help shows:
Usage: dame version [OPTIONS]
Show wcm version
When it should be DAME version
I installed dame-cli from scratch:
pip install -I dame-cli
and when doing
dame run cycles-0.10.2-alpha-collection
I got the following result:
The information of the setup is complete
Executing the setup
2020-04-16 20:16:34,008 root ERROR 'NoneType' object is not subscriptable
Traceback (most recent call last):
File "/data/users/runner/.venv/lib/python3.6/site-packages/dame/cli_methods.py", line 136, in execute_setups
status, file_dir = execute_setup(setup_files)
File "/data/users/runner/.venv/lib/python3.6/site-packages/dame/executor.py", line 108, in execute_setup
raise e
File "/data/users/runner/.venv/lib/python3.6/site-packages/dame/executor.py", line 106, in execute_setup
setup_cmd_line, cwd_path = build_command_line(setup_dict, execution_dir_path)
File "/data/users/runner/.venv/lib/python3.6/site-packages/dame/executor.py", line 83, in build_command_line
l = build_input(inputs, src_path)
File "/data/users/runner/.venv/lib/python3.6/site-packages/dame/executor.py", line 29, in build_input
url = _input["has_fixed_resource"][0]["value"][0]
TypeError: 'NoneType' object is not subscriptable
The configuration is NOT complete, it should ask for the input (hence the error makes sense). Is there a problem in the API?
Rajiv installed DAME on his mac and typed:
(venv) ✓ mayani@Mayani-MacBook-Pro /tmp/x 🐍 global:vscode dame run cycles-0.10.2-alpha-collection-oromia-single-point
The information of the setup is complete
Executing the setup
2020-04-16 20:16:49,979 root INFO Execution cycles-0.10.2-alpha-collection-oromia-single-point running, check the logs on executions/cycles-0.10.2-alpha-collection-oromia-single-point_defcb382-8059-11ea-ac25-acde48001122/output.log
/.singularity.d/actions/exec: line 9: /Users/mayani/run: No such file or directory
[ 1.308773] reboot: Power down
[cycles-0.10.2-alpha-collection-oromia-single-point] The execution has been successful
[cycles-0.10.2-alpha-collection-oromia-single-point] Results available at: executions/cycles-0.10.2-alpha-collection-oromia-single-point_defcb382-8059-11ea-ac25-acde48001122/cycles-0.10.2-alpha-collection/src
(venv) ✓ mayani@Mayani-MacBook-Pro /tmp/x 🐍 global:vscode ls executions/cycles-0.10.2-alpha-collection-oromia-single-point_defcb382-8059-11ea-ac25-acde48001122/cycles-0.10.2-alpha-collection/src
Arsi_Amigna_7.884865046N_40.19527054E.zip crops.crop io.sh template-weed.operation template.operation
__pegasus-job.properties cycles-wrapper.py run template.ctrl
I have not been able to reproduce the error on a linux server, and Rajiv neither. It may be mac-specific.
Right now we take all default values from the model config/setup. However, users may want to test potential setups that have not been declared yet. We should allow a new way of running dame that allows selecting and overwriting parameters.
This adds more complexity to the original design.
test a simple command in:
language: python
python:
Right now we can select files only if they have not been declared. We should support overwriting one of the default files as well.
Describe the bug
I have an error when running dame run e60cf1d0-11f8-454d-a48b-686612fdaf90.
The error I see is
Traceback (most recent call last):
File "C:\Users\dgarijo\Documents\GitHub\dame_cli\env\Scripts\dame-script.py", line 11, in <module>
load_entry_point('dame-cli', 'console_scripts', 'dame')()
return self.main(*args, **kwargs)
File "c:\users\dgarijo\documents\github\dame_cli\env\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "c:\users\dgarijo\documents\github\dame_cli\env\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\users\dgarijo\documents\github\dame_cli\env\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "c:\users\dgarijo\documents\github\dame_cli\env\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "c:\users\dgarijo\documents\github\dame_cli\src\dame\__main__.py", line 127, in run
run_method_setup(resource, interactive, data)
File "c:\users\dgarijo\documents\github\dame_cli\src\dame\cli_methods.py", line 181, in run_method_setup
component_src_dir, execution_dir, setup_cmd_line, setup_name, image = prepare_execution(file_path)
File "c:\users\dgarijo\documents\github\dame_cli\src\dame\executor.py", line 201, in prepare_execution
raise e
File "c:\users\dgarijo\documents\github\dame_cli\src\dame\executor.py", line 199, in prepare_execution
image, setup_cmd_line, src_path = build_command_line(setup_dict, execution_dir_path)
File "c:\users\dgarijo\documents\github\dame_cli\src\dame\executor.py", line 168, in build_command_line
l = build_input(inputs, src_path)
File "c:\users\dgarijo\documents\github\dame_cli\src\dame\executor.py", line 111, in build_input
file_name = get_file(destination_dir, url, _format)
File "c:\users\dgarijo\documents\github\dame_cli\src\dame\executor.py", line 79, in get_file
file_path, file_name = download_data_file(url, destination_dir, _format)
File "c:\users\dgarijo\documents\github\dame_cli\src\dame\utils.py", line 93, in download_data_file
filepath = filepath.with_suffix(validate_suffix(format))
File "C:\Users\dgarijo\AppData\Local\Programs\Python\Python37\lib\pathlib.py", line 853, in with_suffix
raise ValueError("Invalid suffix %r" % (suffix,))
ValueError: Invalid suffix '.https://portals-api.tacc.utexas.edu/postits/v2/065f978cc95f4bd266310556949cc16a'
I thought the setup had a problem in the URL, but I can see it fine in the UI. I wonder what could be wrong.
Other setups for the same model work fine.
To Reproduce
dame run e60cf1d0-11f8-454d-a48b-686612fdaf90
Version 4.0.0 has a slightly different workflow to be more transparent.
Documentation should be updated accordingly.
Describe the bug
~/.mint ❯ dame run hand_v2_raster
Information about the model configuration
Inputs
- input-dem: No information
Parameters
- threshold: 500
Docker Image
- Name: mintproject/hand:v2.1.0 - https://hub.docker.com/r/mintproject/hand
Component Location
- Link: https://github.com/mintproject/HAND-TauDEM/raw/v2.1.4/hand_v2_mint_component.zip
To run this model configuration, a input-dem file (.tif file) is required.
Please enter a url for it: https://github.com/dhardestylewis/HAND-TauDEM/raw/master/regions/Texas/Travis-10m/Travis-DEM-10m-HUC120902050408buf.tif
The information needed to run the model is complete, and I can execute the model as follows:
Invocation command
cd executions/hand_v2_raster_643e8902-8a33-11ea-b216-faffc21691f4/hand_v2_mint_component/src
/usr/local/bin/singularity exec docker://mintproject/hand:v2.1.0 ./run -i1 Travis-DEM-10m-HUC120902050408buf.tif -o1 distance-down.tif -o3 shape.shp -o4 geojson.json -o2 distance-down-raster.tif -p1 500
Do you want to proceed and submit it for execution? [Y/n]:
2020-04-29 09:06:41,245 root INFO Execution hand_v2_raster running, check the logs on executions/hand_v2_raster_643e8902-8a33-11ea-b216-faffc21691f4/output.log
[hand_v2_raster] The execution has failed
~/.mint ❯ cat executions/hand_v2_raster_643e8902-8a33-11ea-b216-faffc21691f4/output.log took 7m 45s
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob sha256:b8f262c62ec67f02536f49654de586c022043652bbb6bbf76a8dab1542627a8d
25.84 MiB / 25.84 MiB [====================================================] 1s
Copying blob sha256:0a43c0154f168ca6fe36f31e366dc85ba2f95da95a9bf69399d927d513d501f9
76.37 MiB / 76.37 MiB [====================================================] 3s
Copying blob sha256:906d7b5da8fb08f9d6c98d3e7df1d621d03d249c26524b64b09641b1f70fc27e
48.80 MiB / 48.80 MiB [====================================================] 1s
Copying blob sha256:44d594344f814f6d0c9a4b3e29401d95e69e922cb82d64243c6903e7fbf29ef8
358.43 MiB / 358.43 MiB [=================================================] 14s
Copying blob sha256:4f3be302f998049e0b99b2c35ad2f1a83b5a7cdab312eedfe45756e7a5cc0ee3
2.47 MiB / 2.47 MiB [======================================================] 0s
Copying blob sha256:a94fae2a8e469d623e81b3411bf49ddc10a44b6bf7ca9365765c776aeaeb310b
251 B / 251 B [============================================================] 0s
Copying blob sha256:317585371b91274d79682e1376b49f762cefc80b30dd5a03d0ca7e58f9841799
1.10 GiB / 1.10 GiB [===================================================] 2m25s
Copying blob sha256:4614779006c97a96c0eb2060de91010107721db2d1fad2d7e0e4fb79d095f5a8
2.47 MiB / 2.47 MiB [======================================================] 0s
Copying config sha256:a30ab4e97d6e18f11eedb37f95f23fbfe1a9c4be5df0f635544307492f5ed92e
3.77 KiB / 3.77 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
FATAL: Unable to handle docker://mintproject/hand:v2.1.0 uri: unable to build: packer failed to pack: while unpacking tmpfs: unpack: error extracting layer: link /var/folders/pf/4j2w8m9x5_19r08n39_t9nc40000gn/T/sbuild-180433821/fs/opt/conda/pkgs/ncurses-6.1-hf484d3e_1002/share/terminfo/N/NCR260VT300WPP /var/folders/pf/4j2w8m9x5_19r08n39_t9nc40000gn/T/sbuild-180433821/fs/opt/conda/share/terminfo/N/NCR260VT300WPP: too many levels of symbolic linksTo Reproduce
Run:
$ dame run hand_v2_rasterAdditional context
In Linux, we don't have problem.
(.venv) node1 ~ dame run hand_v2_raster
Information about the model configuration
Inputs
- input-dem: No information
Parameters
- threshold: 500
Docker Image
- Name: mintproject/hand:v2.1.0 - https://hub.docker.com/r/mintproject/hand
Component Location
- Link: https://github.com/mintproject/HAND-TauDEM/raw/v2.1.4/hand_v2_mint_component.zip
To run this model configuration, a input-dem file (.tif file) is required.
Please enter a url for it: https://github.com/dhardestylewis/HAND-TauDEM/raw/master/regions/Texas/Travis-10m/Travis-DEM-10m-HUC120902050408buf.tif
The information needed to run the model is complete, and I can execute the model as follows:
Invocation command
cd executions/hand_v2_raster_7f784c30-8a33-11ea-b3df-f8f21e3c1558/hand_v2_mint_component/src
/usr/bin/singularity exec docker://mintproject/hand:v2.1.0 ./run -i1 Travis-DEM-10m-HUC120902050408buf.tif -o1 distance-down.tif -o3 shape.shp -o4 geojson.json -o2 distance-down-raster.tif -p1 500
Do you want to proceed and submit it for execution? [Y/n]:
2020-04-29 09:07:21,875 root INFO Execution hand_v2_raster running, check the logs on executions/hand_v2_raster_7f784c30-8a33-11ea-b3df-f8f21e3c1558/output.log
[hand_v2_raster] The execution has been successful
[hand_v2_raster] Results available at: executions/hand_v2_raster_7f784c30-8a33-11ea-b3df-f8f21e3c1558/hand_v2_mint_component/src
If I had another model catalog API (e.g., in TACC), I should be able to configure DAME to query it instead of the default one.
We have updated documentation for DAME 4.1.3, but we should update it for the release that cover all items of this milestone:
The new name should be DAME (Desktop Application for Model Execution).
This change should be done:
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.