luigi should allow for a user level config, not just a root path and system level about luigi HOT 7 CLOSED

might also be nice to support an env var pointing at a config -- e.g. export LUIGI_CFG=/path/to/my/special.cfg

from luigi.

I would additionally vote for having it configurable from a command line parameter to any luigi scripts.

We found that a client.cfg file in the workflow folder is really convenient for adding runtime configurations (such as whether to run a task locally or on the HPC system, and if the latter, with which parameters such as time limit, accounted project etc).

We often start separate runs managed from the same machine, with different configurations for each. Thus it would be really useful to be able to point out the config file with a command line parameter.

Right now we just rely on symlinking to different variants of the config, before starting a luigi run, but that can get a bit messy and error prone.

from luigi.

Well, I guess @jcrobak's suggestion with an env var would do for us too, although a commandline param maybe feels a bit more explicit and unambiguous.

from luigi.

I like a param rather than an env var, too (although I suspect it's a bit more work to support... a lot of the config stuff currently happens on import).

from luigi.

I've been doing using the env var approach as well for setting up the Luigi config.

I played around using OptParse (which has been deprecated btw) at my script level and passing the same OptParse object into Luigi. It works however I don't like how the Luigi usage message gets replaced by the top level script's usage message.

Does anyone have insight on how to properly have your own args specially to dynamically set the config file?

from luigi.

There is already an env var for this:

from luigi.

I'm closing this issue.

• It has been inactive for +4 months
• It's not about luigi core, so not as many users are affected about this.
• The change seems quite big, it's unlikely to be sporadically picked up.
• The owner haven't responded or disappeared.
• I don't understand what this is issue is about.
• There exists a reasonable workaround for this.

Every open issue adds some clutter, and we try to make the issues fewer and make it easier for new collaborators to find. Currently we try to close any issue that meets the first checkbox + one other.

Feel free to reopen this issue at any point if you have the intent to continue to work this. :)

from luigi.