Comments (7)
might also be nice to support an env var pointing at a config -- e.g. export LUIGI_CFG=/path/to/my/special.cfg
from luigi.
I would additionally vote for having it configurable from a command line parameter to any luigi scripts.
We found that a client.cfg
file in the workflow folder is really convenient for adding runtime configurations (such as whether to run a task locally or on the HPC system, and if the latter, with which parameters such as time limit, accounted project etc).
We often start separate runs managed from the same machine, with different configurations for each. Thus it would be really useful to be able to point out the config file with a command line parameter.
Right now we just rely on symlinking to different variants of the config, before starting a luigi run, but that can get a bit messy and error prone.
from luigi.
Well, I guess @jcrobak's suggestion with an env var would do for us too, although a commandline param maybe feels a bit more explicit and unambiguous.
from luigi.
I like a param rather than an env var, too (although I suspect it's a bit more work to support... a lot of the config stuff currently happens on import).
from luigi.
I've been doing using the env var approach as well for setting up the Luigi config.
I played around using OptParse (which has been deprecated btw) at my script level and passing the same OptParse object into Luigi. It works however I don't like how the Luigi usage message gets replaced by the top level script's usage message.
Does anyone have insight on how to properly have your own args specially to dynamically set the config file?
from luigi.
There is already an env var for this:
Line 11 in e496d16
from luigi.
I'm closing this issue.
- It has been inactive for +4 months
- It's not about luigi core, so not as many users are affected about this.
- The change seems quite big, it's unlikely to be sporadically picked up.
- The owner haven't responded or disappeared.
- I don't understand what this is issue is about.
- There exists a reasonable workaround for this.
Every open issue adds some clutter, and we try to make the issues fewer and make it easier for new collaborators to find. Currently we try to close any issue that meets the first checkbox + one other.
Feel free to reopen this issue at any point if you have the intent to continue to work this. :)
from luigi.
Related Issues (20)
- `h5py` compatibility with `S3Target` ? HOT 2
- Facing test failures for test/servertest.py on PPC64LE
- Only one worker process being created despite of passing workers as 3 and having simultaneous tasks to perform
- Faulty DONE status assignment HOT 2
- sqla.SQLAlchemyTarget.exist() won't work with SqlAlchemy 2.x version
- TypeError: <Target> object is not iterable when check_complete(dep) is False HOT 3
- Questions about Copilot + Open Source Software Hierarchy
- `JobTask.dump` raises `TypeError` when replacing bytes to string
- Change `subprocess.check_output` option for bytes/string mismatch HOT 1
- Feature request: Include completed task runs in task history
- Bug-Report: Tenacity 8.4.0 HOT 4
- In configuration docs, email_interval_minutes should be email_interval
- In the configuration docs, batch_notifier should be batch_email
- scheduler prune_on_get_work configuration parameter is not documented
- Sequential processing of a specific task with resouces:1 seems to be not working
- Clear-text Logging of Sensitive Information in `luigi/contrib/pai.py`
- Arbitrary file write during tarfile extraction in `luigi/contrib/sge_runner.py`
- Arbitrary file write during tarfile extraction in `luigi/contrib/lsf_runner.py`
- Overly permissive file permissions in `luigi/lock.py`
- Unsafe shell command constructed from library input in `luigi/contrib/lsf.py`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from luigi.