Environment klio* version

Hey <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url=

klio-cli -> v1.0.5 klio-exec -> 0.2.2 klio-core -> 0.

Full traceback: <div class="snippet-clipboard-content notranslate position-relativ

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Templating args passed to klio-cli are not transmitted to klioexec about klio HOT 10 OPEN

spotify commented on May 12, 2024

Templating args passed to klio-cli are not transmitted to klioexec

from klio.

Comments (10)

pomdtr commented on May 12, 2024

Ok looking at it further it looks like /usr/src/config/materialized_config.yaml correctly resolve the secrets. But why is the preprocessing step running again in klioexec ?

from klio.

pomdtr commented on May 12, 2024

Klioexec is trying to process the build time config in order to compare it with the run time config. However it is not able to process it due to the missing template args.

from klio.

riordan commented on May 12, 2024

Great catch; thank you! We'll dig into this.

Is this a blocking issue for you or is there a workaround in the interim?

from klio.

pomdtr commented on May 12, 2024

We are not sure we want to use the template feature to pass secrets to the pipelines. I currently use a wrapper to use directly klioexec instead, and pass the the secrets through the --env-file docker parameter.

However templating would still be useful to handle our different environment, so this issue is a bit problematic for us.

from klio.

DanSimon commented on May 12, 2024

Hey @pomdtr, can you include the versions of klio-exec, klio, and klio-core you're using in your image? Can you also include whether your project is using setup.py or FnApi for its deployment.

Also to verify, is the KlioConfigTemplatingException happening as soon as klioexec starts up, or is the error coming from a worker after the pipeline has started?

from klio.

pomdtr commented on May 12, 2024

klio-cli -> v1.0.5
klio-exec -> 0.2.2
klio-core -> 0.2.2
klio -> 0.2.4

Our project is using the worker_harness_container_image beam option, we are not defining any setup.py.

The error is triggered as soon as the klioexec command starts up.

from klio.

pomdtr commented on May 12, 2024

Full traceback:

INFO:root:Found worker image: gcr.io/dailymotion-data-backbone-stg/klio-frames-worker:1c6edd0a-dirty
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/klio_core/config/_preprocessing.py", line 159, in _apply_templates
    return template.substitute(**templates)
  File "/usr/local/lib/python3.6/string.py", line 130, in substitute
    return self.pattern.sub(convert, self.template)
  File "/usr/local/lib/python3.6/string.py", line 123, in convert
    return str(mapping[named])
KeyError: 'dailymotion_api_key'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/bin/klioexec", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/klio_core/utils.py", line 238, in wrapper
    func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/klio_exec/cli.py", line 80, in run_pipeline
    if _compare_runtime_to_buildtime_config(klio_config) is False:
  File "/usr/local/lib/python3.6/site-packages/klio_exec/cli.py", line 62, in _compare_runtime_to_buildtime_config
    buildtime_config = config.KlioConfig(_get_config(buildtime_config_path))
  File "/usr/local/lib/python3.6/site-packages/klio_core/config/core.py", line 101, in __init__
    raw_override_list=raw_overrides or [],
  File "/usr/local/lib/python3.6/site-packages/klio_core/config/_preprocessing.py", line 241, in process
    raw_config_data, template_dict
  File "/usr/local/lib/python3.6/site-packages/klio_core/config/_preprocessing.py", line 161, in _apply_templates
    raise exceptions.KlioConfigTemplatingException(e)
klio_core.exceptions.KlioConfigTemplatingException: 'dailymotion_api_key' missing in key template overrides.

from klio.

fallonchen commented on May 12, 2024

@pomdtr The fix for this specific issue should be included in our latest release candidate. Would you mind giving them a try? You can get them by setting your klio-* libraries to version 21.2.0rc1.

Note that we uncovered another issue while investigating this one - basically, all klio subcommands won't work with templated values unless you pass along the templated argument. So when running something like klio message publish, you will probably encounter another KlioConfigTemplatingException. We are working on a fix now, but for now you can work around it by passing in the template argument klio message publish --template templated_value=whatever.

from klio.

pomdtr commented on May 12, 2024

Thanks a lot! I will check it as soon as possible :)

from klio.

alecgarza96 commented on May 12, 2024

Is this still an issue?

from klio.

Templating args passed to klio-cli are not transmitted to klioexec about klio HOT 10 OPEN

Comments (10)

Related Issues (6)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent