spotify / klio Goto Github PK

View Code? Open in Web Editor NEW

829.0 20.0 51.0 75.49 MB

Smarter data pipelines for audio.

Home Page: https://docs.klio.io

License: Apache License 2.0

Python 97.43% Smarty 1.14% Dockerfile 0.37% Shell 0.32% CSS 0.34% HTML 0.40%

audio-processing data-pipeline signal-processing media-processing

klio's Introduction

`klio`: Smarter data pipelines for audio

Latest Versions of Klio Packages:

What is Klio?

Klio is an ecosystem that allows you to process audio files – or any binary files – easily and at scale.

Klio jobs are opinionated data pipelines in Python (streaming or batch) built upon Apache Beam and tuned for audio and binary file processing.

Klio was built by Spotify to run our large-scale audio intelligence systems and is used by teams of engineers and audio researchers to help develop and deploy next generation audio algorithms.

Get Started

Check out our Quick Start guide on how to start using Klio.

Resources

Have a look at our complete documentation.
Check out our contributors' guide.
Read up on some of our FAQs.
Review our Code of Conduct - this is how we roll.

klio's People

Contributors

Stargazers

Watchers

klio's Issues

Cannot install as instructed in quickstart guide

Is this a BUG REPORT or FEATURE REQUEST?:
type: bug

Environment

klio version(s): N/A
Operating System(s): Ubuntu 20.04.1 LTS x86_64
Python version(s): 3.8.5

What happened

Running pipx install klio-cli on a new Python install on Ubuntu 20.04 after running sudo apt install pipx python3-venv leads to an error about package incompatibilities.

carl@workstation:~/klio_quickstart$ pipx install klio-cli

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

google-api-python-client 1.12.5 requires google-api-core<2dev,>=1.21.0, but you'll have google-api-core 1.20.1 which is incompatible.
klio-core 0.2.0 requires google-api-python-client<1.12,>=1.10.0, but you'll have google-api-python-client 1.12.5 which is incompatible.
Could not find package klio-cli. Is the name correct?

What you expected to happen

I expected to be able to run klio --version just after.

How to reproduce it (as minimally and precisely as possible)

pipx install klio-clio, I assume. Perhaps it's specific to my Ubuntu install's Python somehow? All I've really done AFAIK is to run sudo apt install pipx python3-venv on a clean slate.

Provide access to the --env-file docker parameter through the klio job run command

Describe the feature you'd like

I would like to be able to pass env variables to my application using the --env-file flag of the docker-run command

Is your feature request related to a problem?

We are currently struggling to pass secrets to our klio pipeline, and I'm not sure the template feature is the right way to achieve this.

@decorators.timeout breaks @mock.patch

Environment

klio* version: 21.2.0rc1
Operating System: macOS Big Sur 11.2.3
Python version: 3.8.7
Docker version: 3.2.2

Description of the bug

Adding @decorators.timeout(seconds=30) to the process method of ProcessEncode class breaks @mock.patch decorator.
In my case, I have @mock.patch("google.cloud.storage.Blob") and it stops working after I add @decorators.timeout to process.

What you expected to happen

I expect Blob mock to be injected and its methods to be called and assertable in the tests.

How to reproduce (as minimally and precisely as possible)

Add @decorators.timeout to the process method that calls Blob's mocked method and observe it not being called anymore.

Templating args passed to klio-cli are not transmitted to klioexec

Environment

klio* version(s):
- klio-cli: v1.0.5

Description of the bug

When running the klio job run command with template args:

klio job run --config-file config/streaming/klio-job-stage.yaml \
  --template dailymotion_api_key=$DAILYMOTION_API_KEY \
  --template dailymotion_api_secret=$DAILYMOTION_API_SECRET \
  --direct-runner --force-build

The template args are not passed to klioexec entrypoint:

'command': ['run',
             '--image-tag',
             '1c6edd0a-dirty',
             '--direct-runner',
             '--config-file',
             '/usr/src/config/materialized_config.yaml']

Causing the following exception

klio_core.exceptions.KlioConfigTemplatingException: 'dailymotion_api_key' missing in key template overrides.

What you expected to happen

Template args should be passed to the klioexec command

How to reproduce (as minimally and precisely as possible)

Run any command involving the template args.

installation guide in API docs is a 404

Hey klio team!

The installation guide linked from your API docs goes to a 404. Wanted to let you know!

klio_core.exceptions.KlioConfigTemplatingException thrown for `klio` subcommands when templated values are used

Environment

klio* version(s): latest release candidates (klio-*==21.2.0rc1) and latest from master (klio-exec==0.2.2, klio-cli==1.0.5, klio-core==0.2.2)
Operating System(s): OSX
Python version(s) (python -V): 3.6.9
Docker version (if relevant) (docker --version):

Description of the bug

When templated values are used in klio-job.yaml, klio subcommands also need the templated value to be passed in (e.g. you need to run klio message publish --template templated_value=MYVALUE) . Otherwise, a KlioConfigTemplatingException is thrown.

What you expected to happen

I expect the non-run subcommands to run correctly without the templated values being explicitly passed in.

How to reproduce (as minimally and precisely as possible)

Create a klio job: klio job create --use-fnapi --use-defaults --job-type streaming
Add a templated value to your klio-job.yaml:

<!--snip-->
job_config:
  api_key: ${templated_key}
<!--snip-->

Run any klio subcommand that isn't klio job run: klio message publish, klio job create, klio job verify, etc

You should then see a stacktrace like this:

(klio-current) ➜  klio-template-bug-fnapi git:(master) ✗ klio job verify --create-resources                             
Traceback (most recent call last):
  File "/Users/fallon/.pyenv/versions/3.6.9/envs/klio-current/lib/python3.6/site-packages/klio_core/config/_preprocessing.py", line 159, in _apply_templates
    return template.substitute(**templates)
  File "/Users/fallon/.pyenv/versions/3.6.9/lib/python3.6/string.py", line 130, in substitute
    return self.pattern.sub(convert, self.template)
  File "/Users/fallon/.pyenv/versions/3.6.9/lib/python3.6/string.py", line 123, in convert
    return str(mapping[named])
KeyError: 'templated_value'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/fallon/.pyenv/versions/klio-current/bin/klio", line 11, in <module>
    sys.exit(main())
  File "/Users/fallon/.pyenv/versions/3.6.9/envs/klio-current/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/fallon/.pyenv/versions/3.6.9/envs/klio-current/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/fallon/.pyenv/versions/3.6.9/envs/klio-current/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/fallon/.pyenv/versions/3.6.9/envs/klio-current/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/fallon/.pyenv/versions/3.6.9/envs/klio-current/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/fallon/.pyenv/versions/3.6.9/envs/klio-current/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/fallon/.pyenv/versions/3.6.9/envs/klio-current/lib/python3.6/site-packages/klio_core/utils.py", line 228, in wrapper
    raw_overrides=raw_overrides,
  File "/Users/fallon/.pyenv/versions/3.6.9/envs/klio-current/lib/python3.6/site-packages/klio_core/config/core.py", line 101, in __init__
    raw_override_list=raw_overrides or [],
  File "/Users/fallon/.pyenv/versions/3.6.9/envs/klio-current/lib/python3.6/site-packages/klio_core/config/_preprocessing.py", line 241, in process
    raw_config_data, template_dict
  File "/Users/fallon/.pyenv/versions/3.6.9/envs/klio-current/lib/python3.6/site-packages/klio_core/config/_preprocessing.py", line 161, in _apply_templates
    raise exceptions.KlioConfigTemplatingException(e)
klio_core.exceptions.KlioConfigTemplatingException: 'templated_value' missing in key template overrides.

spotify / klio Goto Github PK

klio's Introduction

klio: Smarter data pipelines for audio

What is Klio?

Get Started

Resources

klio's People

Contributors

Stargazers

Watchers

Forkers

klio's Issues

Environment

What happened

What you expected to happen

How to reproduce it (as minimally and precisely as possible)

Describe the feature you'd like

Is your feature request related to a problem?

Environment

Description of the bug

What you expected to happen

How to reproduce (as minimally and precisely as possible)

Environment

Description of the bug

What you expected to happen

How to reproduce (as minimally and precisely as possible)

Environment

Description of the bug

What you expected to happen

How to reproduce (as minimally and precisely as possible)

Anthing else we need to know?

Recommend Projects

Recommend Topics

Recommend Org

`klio`: Smarter data pipelines for audio