Comments (13)
Yes here are the traceback
Encountered exception during execution:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 1215, in orchestrate_task_run
result = await task.fn(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/prefect_dbt/cli/commands.py", line 136, in trigger_dbt_cli_command
profile = dbt_cli_profile.get_profile()
File "/usr/local/lib/python3.9/site-packages/prefect_dbt/cli/credentials.py", line 111, in get_profile
"outputs": {self.target: self.target_configs.get_configs()},
File "/usr/local/lib/python3.9/site-packages/prefect_dbt/cli/configs/bigquery.py", line 114, in get_configs
configs_json[key] = getattr(google_credentials, key)
AttributeError: 'Credentials' object has no attribute 'refresh_token'
We could use something like this
https://programtalk.com/python-more-examples/google.auth.compute_engine.credentials.Credentials/
Will try to make a PR
from prefect-dbt.
Hi @lucienfregosibodyguard, thanks for reporting this. Do you have a traceback?
Also, would you be interested in contributing a fix; might be here:
https://github.com/PrefectHQ/prefect-gcp/blob/main/prefect_gcp/credentials.py#L127-L144
from prefect-dbt.
@ahuang11 I tried to use the code snippet above but it doesn't work
from prefect-dbt.
@ahuang11 do you have time to help on this ? Still stuck on my side :/
from prefect-dbt.
@lucienfregosibodyguard can you try pulling this branch to test? #98
from prefect-dbt.
Hi @ahuang11 I looked at the PR but it seems that token
is not an attribute of the class
https://google-auth.readthedocs.io/en/master/reference/google.auth.compute_engine.credentials.html
maybe I miss something but I can't see how it can works 😕
from prefect-dbt.
There might be multiple credentials: I'm looking at Google auth default. https://google-auth.readthedocs.io/en/master/reference/google.auth.html
[docs]def default(scopes=None, request=None, quota_project_id=None, default_scopes=None):
"""Gets the default credentials for the current environment.
`Application Default Credentials`_ provides an easy way to obtain
credentials to call Google APIs for server-to-server or local applications.
This function acquires credentials from the environment in the following
order:
1. If the environment variable ``GOOGLE_APPLICATION_CREDENTIALS`` is set
to the path of a valid service account JSON private key file, then it is
loaded and returned. The project ID returned is the project ID defined
in the service account file if available (some older files do not
contain project ID information).
If the environment variable is set to the path of a valid external
account JSON configuration file (workload identity federation), then the
configuration file is used to determine and retrieve the external
credentials from the current environment (AWS, Azure, etc).
These will then be exchanged for Google access tokens via the Google STS
endpoint.
The project ID returned in this case is the one corresponding to the
underlying workload identity pool resource if determinable.
2. If the `Google Cloud SDK`_ is installed and has application default
credentials set they are loaded and returned.
To enable application default credentials with the Cloud SDK run::
gcloud auth application-default login
If the Cloud SDK has an active project, the project ID is returned. The
active project can be set using::
gcloud config set project
3. If the application is running in the `App Engine standard environment`_
(first generation) then the credentials and project ID from the
`App Identity Service`_ are used.
4. If the application is running in `Compute Engine`_ or `Cloud Run`_ or
the `App Engine flexible environment`_ or the `App Engine standard
environment`_ (second generation) then the credentials and project ID
are obtained from the `Metadata Service`_.
5. If no credentials are found,
:class:`~google.auth.exceptions.DefaultCredentialsError` will be raised.
.. _Application Default Credentials: https://developers.google.com\
/identity/protocols/application-default-credentials
.. _Google Cloud SDK: https://cloud.google.com/sdk
.. _App Engine standard environment: https://cloud.google.com/appengine
.. _App Identity Service: https://cloud.google.com/appengine/docs/python\
/appidentity/
.. _Compute Engine: https://cloud.google.com/compute
.. _App Engine flexible environment: https://cloud.google.com\
/appengine/flexible
.. _Metadata Service: https://cloud.google.com/compute/docs\
/storing-retrieving-metadata
.. _Cloud Run: https://cloud.google.com/run
from prefect-dbt.
"google.oauth2.service_account.Credentials" seems to have the token attr.
from prefect-dbt.
Yes you're right it has a token attribute indeed !
Few remarks :
- in order to get the token we need to refresh the credentials like this
request = google.auth.transport.requests.Request()
google_credentials.refresh(request)
configs_json["token"] = google_credentials.token
-
We need to add a method object
configs_json["method"] = "service-account"
-
And then I got this profile file :
config: {}
dbt_models:
outputs:
<target>:
method: service-account
project: <project>
schema: <dataset>
threads: 4
token: <token>
type: bigquery
target: <target>
And I got the error Database Error expected str, bytes or os.PathLike object, not NoneType
Then I add dbname: <project>
but I got a new error Got duplicate keys: (project) all map to "database"
Hope that helps
from prefect-dbt.
Oh following this
We need method: oauth-secrets
and it works !!!
from prefect-dbt.
I did a PR
#100
from prefect-dbt.
This should now be fixed in v0.2.6. Please reopen if that's not the case!
from prefect-dbt.
Hi @ahuang11 sadly it's still doesn't work, got a new error
Encountered exception during execution:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1339, in orchestrate_task_run
result = await task.fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/prefect_dbt/cli/commands.py", line 139, in trigger_dbt_cli_command
yaml.dump(profile, f, default_flow_style=False)
File "/usr/local/lib/python3.10/site-packages/yaml/__init__.py", line 253, in dump
return dump_all([data], stream, Dumper=Dumper, **kwds)
File "/usr/local/lib/python3.10/site-packages/yaml/__init__.py", line 241, in dump_all
dumper.represent(data)
File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 27, in represent
node = self.represent_data(data)
File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 48, in represent_data
node = self.yaml_representers[data_types[0]](self, data)
File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 207, in represent_dict
return self.represent_mapping('tag:yaml.org,2002:map', data)
File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 118, in represent_mapping
node_value = self.represent_data(item_value)
File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 48, in represent_data
node = self.yaml_representers[data_types[0]](self, data)
File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 207, in represent_dict
return self.represent_mapping('tag:yaml.org,2002:map', data)
File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 118, in represent_mapping
node_value = self.represent_data(item_value)
File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 48, in represent_data
node = self.yaml_representers[data_types[0]](self, data)
File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 207, in represent_dict
return self.represent_mapping('tag:yaml.org,2002:map', data)
File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 118, in represent_mapping
node_value = self.represent_data(item_value)
File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 52, in represent_data
node = self.yaml_multi_representers[data_type](self, data)
File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 317, in represent_object
reduce = data.__reduce_ex__(2)
TypeError: cannot pickle 'coroutine' object
I guess it's related to the sync definition but don't know how to fix it
from prefect-dbt.
Related Issues (20)
- Prefect dbt 0.2.6 coroutine error HOT 26
- Saving `DbtCliProfile` with `prefect_sqlalchemy.DatabaseCredentials` raises `TypeError: Type is not JSON serializable: URL` HOT 2
- Using private key in SnowflakeTargetConfigs
- GlobalConfigs' "log_format" should be of type String instead of Bool. HOT 3
- TargetConfigs' credentials gets dropped upon loading a DbtCliProfile
- Support SqlAlchemyConnector for PostgresTargetConfigs
- Issues with logging and dbt 1.4 (protobuf) HOT 8
- Credentials missing when using SnowflakeTargetConfigs in CLIProfile HOT 4
- Postgres port set as string in profiles.yml HOT 1
- Typo on dbt prefect documentation side
- add `trigger_dbt_cli_command` example to docs
- DBT improvements: dbt_build_task
- Potential path parsing issue with Windows HOT 3
- Integrations test for forked repos HOT 1
- block register command for creds block is incorrect HOT 1
- Execute a dbt CLI command HOT 2
- Block registration states 4 new block types created, but UI shows 6 new block types HOT 1
- Change poll log level
- Show polling status updates selectively by comparing previous status vs current status
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from prefect-dbt.