Comments (5)
(related to Zendesk ticket #1742)
from valohai-utils.
valohai.prepare
does not set the command. The file name to execute is not an explicit part of the yaml step configuration, either: you can write whatever list of commands you want as the yaml command
field. We do not and should never modify the existing command definition while executing yaml step
.
To address this issue, we should instead change the default command to use a parameter to choose what file it will execute, and yaml step
will set the filename it is parsing as a new implicit (Python wise, explicit yaml wise) parameter.
Existing steps will behave as before: the special filename parameter only gets created or updated if it already exists on the original. This is important for backwards compatibility.
from valohai-utils.
PRs in path dependency order; each step needs to be done prior to the next:
- valohai/valohai-yaml#115
- #119
- TODO: Upgrade Roi with new valohai-yaml and valohai-utils
- TODO: Upgrade CLI with new valohai-yaml and valohai-utils
Problem: Roi needs to support the new yaml and utils prior to CLI getting them in order to interpolate commands for executions, but CLI does not pin yaml and utils dependencies so users installing CLI will get too new yaml and utils that Roi doesn't support yet.
Solutions:
- Either Valohai-CLI needs to be released with a pinning to use valohai-utils and valohai-yaml of a specific version, and release an updated version once Roi has been released
- Or: Release yaml and utils as beta/private versions that CLI doesn't upgrade to, upgrade and release Roi with new yaml and utils, release yaml and utils to the general public
- Or: Make the new command-and-source-path behaviour optional on the CLI end, so it won't be used for new steps for users who don't need it (Needs CLI PR to introduce command flags to support this)
from valohai-utils.
testing with everything upgraded on local... (cli, roi)
from valohai-utils.
Also, while additional fields don't break older versions of yaml/utils/cli, using old versions will drop the source-path parameter and the step will break for executions due to referring to the special option that no longer exists.
However, this is easily noticed or reverted as all changes should be in git. This does involve some customer awareness though.
Nevermind, old versions will refuse to read a valohai.yaml with extra fields newly added to the schema.
We should probably add a valohai.yaml
version field to the schema, allowing older versions of the CLI to notice that they're out of their depth modifying the file and provide a helpful error message.
from valohai-utils.
Related Issues (20)
- Command-line overriding is not working if the parameter name has a '-'
- Local downloads not working on Windows HOT 1
- Add `distributed.json` parsing
- Utils can't find local cached input file by filename HOT 4
- Can't define the environment in valohai.prepare()
- Cannot repare multiple styled parameter
- Access data stores connected to project locally with valohai-utils
- Users other than root cannot write to /valohai/inputs HOT 2
- Improved pipeline creation with valohai-utils
- Allow creating parameter choices with valohai-utils
- Default docker image could be using the local python version and not hard coded value
- Metadata printing: Detect if `sys.stdout` has been redirected
- Add API for getting an output directory
- Add API for writing metadata sidecars
- API for accessing execution config
- Update permissions for compressed output files
- Consider prioiritizing Python-defined parameters
- Allow typing for parameter and input dictionaries
- Don't override the whole step when updating valohai.yaml with vh step command
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from valohai-utils.