Prefect dbt 0.2.6 coroutine error about prefect-dbt HOT 26 CLOSED

Do you have this wrapped in a flow? Also are you working in notebook?

Here's something you can try in notebook / script.

from prefect import flow
from prefect_gcp import GcpCredentials
from prefect_dbt.cli import DbtCliProfile
from prefect_dbt.cli.configs import BigQueryTargetConfigs


@flow
def test_flow():
    credentials = GcpCredentials()

    target_configs = BigQueryTargetConfigs(
        credentials=credentials,
        schema="schema",
        project="project",
    )
    dbt_cli_profile = DbtCliProfile(
        name="jaffle_shop",
        target="dev",
        target_configs=target_configs,
    )
    profile = dbt_cli_profile.get_profile()
    return profile

test_flow()

from prefect-dbt.

No the code is directly run within a kubernetes pod.

Locally it seems to work

from prefect-dbt.

Can you try to see if this branch potentially fixes the issue? I made the base class method also sync compatible, but unsure if it'll help.

https://github.com/PrefectHQ/prefect-dbt/pull/105/files

from prefect-dbt.

I have the same issue sadly :/

Finished in state Failed(message="Flow run encountered an exception. TypeError: cannot pickle 'coroutine' object\n", type=FAILED, result=TypeError("cannot pickle 'coroutine' object"))

from prefect-dbt.

I suspect it's due to PrefectHQ/prefect-gcp#80 not yet released. Could you try pulling prefect-gcp main?

e.g. in requirements.txt prefect-gcp @ git+https://github.com/PrefectHQ/prefect-gcp.git@main

from prefect-dbt.

Same issue with the prefect-gcp version 0.1.8+2.g28f72ba

from prefect-dbt.

To clarify: "No the code is directly run within a kubernetes pod."

Is your code wrapped in a flow?

from prefect-dbt.

Sure all the code is wrapped in a flow and triggered with prefect

from prefect-dbt.

Thanks for confirming and I appreciate your help testing! I made another commit on https://github.com/PrefectHQ/prefect-dbt/pull/105/files

Can you help me test that again? If this doesn't work, I think we might have to make it sync again.

from prefect-dbt.

@ahuang11 I'm so sorry It still doesn't work :( with same error

from prefect-dbt.

Thanks for testing. Can you provide a minimal reproducible example that we can try on our end?

from prefect-dbt.

hey @ahuang11
Actually I got this error even with the previous authentication method.

The 0.2.6 is broken for us because of the co routine error

It can be hard to reproduce the error because it happens only in a Kubernetes context (it works well locally)

1. Package a docker image based on prefect 2.7.0-python3.10 with this version
   prefect-dbt = "0.2.6"
   dbt-bigquery = "1.3.0"
   prefect-gcp = "0.1.6"

2. Add code to the docker image
   the code can be taken from the official prefect gcp example the error raises with any prefect_dbt.cli.commands import trigger_dbt_cli_command

3. Create a storage & kubernetes block based on the previous docker image which will be used by the flow

4. Start a flow in the prefect UI

If it's too hard for you to replicate this, we can plan a call or even let you use my mac remotely

from prefect-dbt.

Thanks for that; can you try #111 to see if this fixes the issue? I dropped the async.

from prefect-dbt.

@ahuang11 yes now it works perfectly with the last fix

from prefect-dbt.

It it possible to merge and release a 0.2.8 package ? :)

from prefect-dbt.

Yes I'll try to get the PR ready soon!

from prefect-dbt.

@lucienfregosibodyguard by any chance, are you loading a DbtCliProfile like shown in #114?

from prefect-dbt.

Here is my DbtCliProfile Definition

    def _get_credentials(self) -> GcpCredentials:
        return GcpCredentials()

    def _get_target_configs(self) -> BigQueryTargetConfigs:
        return BigQueryTargetConfigs(
            schema=self.dataset_name, project=self.gcp_project, credentials=self._get_credentials()
        )

    def _get_dbt_cli_profile(self) -> DbtCliProfile:
        return DbtCliProfile(
            name=self.dbt_folder_name, target=self.environment, target_configs=self._get_target_configs()
        )

let me know if it's not the up to date version to load DbtCliProfile

from prefect-dbt.

Okay so you don't use .save or .load?

from prefect-dbt.

No I can try with save and load

from prefect-dbt.

It seems better actually i don't have the coroutine error, it starts the DBT command and I got this

�[0m13:50:46  Encountered an error while reading profiles:
�[0m13:50:46    ERROR: Runtime Error
  Credentials in profile "dbt_models", target "" invalid: 'method' is a required property

Which is a "normal" error that should be solved in your PR

from prefect-dbt.

Hi @ahuang11 could you release a new tag to easily test this and update my code ?

from prefect-dbt.

Actually this isn't fixed yet. Will try to get a new release soon...

from prefect-dbt.

@ahuang11 could we make a new tag with the last fix ?

from prefect-dbt.

Done!

from prefect-dbt.

Thaaaaaaaks it works like a charm awesome

from prefect-dbt.