Giter VIP home page Giter VIP logo

Comments (13)

lucienfregosibodyguard avatar lucienfregosibodyguard commented on May 29, 2024 1

Yes here are the traceback

Encountered exception during execution:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 1215, in orchestrate_task_run
    result = await task.fn(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/prefect_dbt/cli/commands.py", line 136, in trigger_dbt_cli_command
    profile = dbt_cli_profile.get_profile()
  File "/usr/local/lib/python3.9/site-packages/prefect_dbt/cli/credentials.py", line 111, in get_profile
    "outputs": {self.target: self.target_configs.get_configs()},
  File "/usr/local/lib/python3.9/site-packages/prefect_dbt/cli/configs/bigquery.py", line 114, in get_configs
    configs_json[key] = getattr(google_credentials, key)
AttributeError: 'Credentials' object has no attribute 'refresh_token'

We could use something like this
https://programtalk.com/python-more-examples/google.auth.compute_engine.credentials.Credentials/

Will try to make a PR

from prefect-dbt.

ahuang11 avatar ahuang11 commented on May 29, 2024

Hi @lucienfregosibodyguard, thanks for reporting this. Do you have a traceback?

Also, would you be interested in contributing a fix; might be here:
https://github.com/PrefectHQ/prefect-gcp/blob/main/prefect_gcp/credentials.py#L127-L144

from prefect-dbt.

lucienfregosibodyguard avatar lucienfregosibodyguard commented on May 29, 2024

@ahuang11 I tried to use the code snippet above but it doesn't work

from prefect-dbt.

lucienfregosibodyguard avatar lucienfregosibodyguard commented on May 29, 2024

@ahuang11 do you have time to help on this ? Still stuck on my side :/

from prefect-dbt.

ahuang11 avatar ahuang11 commented on May 29, 2024

@lucienfregosibodyguard can you try pulling this branch to test? #98

from prefect-dbt.

lucienfregosibodyguard avatar lucienfregosibodyguard commented on May 29, 2024

Hi @ahuang11 I looked at the PR but it seems that token is not an attribute of the class
https://google-auth.readthedocs.io/en/master/reference/google.auth.compute_engine.credentials.html

maybe I miss something but I can't see how it can works 😕

from prefect-dbt.

ahuang11 avatar ahuang11 commented on May 29, 2024

There might be multiple credentials: I'm looking at Google auth default. https://google-auth.readthedocs.io/en/master/reference/google.auth.html

 [docs]def default(scopes=None, request=None, quota_project_id=None, default_scopes=None):
    """Gets the default credentials for the current environment.

    `Application Default Credentials`_ provides an easy way to obtain
    credentials to call Google APIs for server-to-server or local applications.
    This function acquires credentials from the environment in the following
    order:

    1. If the environment variable ``GOOGLE_APPLICATION_CREDENTIALS`` is set
       to the path of a valid service account JSON private key file, then it is
       loaded and returned. The project ID returned is the project ID defined
       in the service account file if available (some older files do not
       contain project ID information).

       If the environment variable is set to the path of a valid external
       account JSON configuration file (workload identity federation), then the
       configuration file is used to determine and retrieve the external
       credentials from the current environment (AWS, Azure, etc).
       These will then be exchanged for Google access tokens via the Google STS
       endpoint.
       The project ID returned in this case is the one corresponding to the
       underlying workload identity pool resource if determinable.
    2. If the `Google Cloud SDK`_ is installed and has application default
       credentials set they are loaded and returned.

       To enable application default credentials with the Cloud SDK run::

            gcloud auth application-default login

       If the Cloud SDK has an active project, the project ID is returned. The
       active project can be set using::

            gcloud config set project

    3. If the application is running in the `App Engine standard environment`_
       (first generation) then the credentials and project ID from the
       `App Identity Service`_ are used.
    4. If the application is running in `Compute Engine`_ or `Cloud Run`_ or
       the `App Engine flexible environment`_ or the `App Engine standard
       environment`_ (second generation) then the credentials and project ID
       are obtained from the `Metadata Service`_.
    5. If no credentials are found,
       :class:`~google.auth.exceptions.DefaultCredentialsError` will be raised.

    .. _Application Default Credentials: https://developers.google.com\
            /identity/protocols/application-default-credentials
    .. _Google Cloud SDK: https://cloud.google.com/sdk
    .. _App Engine standard environment: https://cloud.google.com/appengine
    .. _App Identity Service: https://cloud.google.com/appengine/docs/python\
            /appidentity/
    .. _Compute Engine: https://cloud.google.com/compute
    .. _App Engine flexible environment: https://cloud.google.com\
            /appengine/flexible
    .. _Metadata Service: https://cloud.google.com/compute/docs\
            /storing-retrieving-metadata
    .. _Cloud Run: https://cloud.google.com/run

from prefect-dbt.

ahuang11 avatar ahuang11 commented on May 29, 2024

"google.oauth2.service_account.Credentials" seems to have the token attr.
image

from prefect-dbt.

lucienfregosibodyguard avatar lucienfregosibodyguard commented on May 29, 2024

Yes you're right it has a token attribute indeed !

Few remarks :

  • in order to get the token we need to refresh the credentials like this
                request = google.auth.transport.requests.Request()
                google_credentials.refresh(request)
                configs_json["token"] = google_credentials.token
  • We need to add a method object
    configs_json["method"] = "service-account"

  • And then I got this profile file :

config: {}
dbt_models:
  outputs:
    <target>:
      method: service-account
      project: <project>
      schema: <dataset>
      threads: 4
      token: <token>
      type: bigquery
  target: <target>

And I got the error Database Error expected str, bytes or os.PathLike object, not NoneType

Then I add dbname: <project> but I got a new error Got duplicate keys: (project) all map to "database"

Hope that helps

from prefect-dbt.

lucienfregosibodyguard avatar lucienfregosibodyguard commented on May 29, 2024

Oh following this

We need method: oauth-secrets and it works !!!

from prefect-dbt.

lucienfregosibodyguard avatar lucienfregosibodyguard commented on May 29, 2024

I did a PR
#100

from prefect-dbt.

ahuang11 avatar ahuang11 commented on May 29, 2024

This should now be fixed in v0.2.6. Please reopen if that's not the case!

from prefect-dbt.

lucienfregosibodyguard avatar lucienfregosibodyguard commented on May 29, 2024

Hi @ahuang11 sadly it's still doesn't work, got a new error

Encountered exception during execution:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1339, in orchestrate_task_run
    result = await task.fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect_dbt/cli/commands.py", line 139, in trigger_dbt_cli_command
    yaml.dump(profile, f, default_flow_style=False)
  File "/usr/local/lib/python3.10/site-packages/yaml/__init__.py", line 253, in dump
    return dump_all([data], stream, Dumper=Dumper, **kwds)
  File "/usr/local/lib/python3.10/site-packages/yaml/__init__.py", line 241, in dump_all
    dumper.represent(data)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 27, in represent
    node = self.represent_data(data)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 317, in represent_object
    reduce = data.__reduce_ex__(2)
TypeError: cannot pickle 'coroutine' object

I guess it's related to the sync definition but don't know how to fix it

from prefect-dbt.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.