[ADAP-1075] [Bug] Python models are hard to change after a first run about dbt-bigquery HOT 3 CLOSED

Hi @dbeatty10 , thank you for the prompt response and attention to the problem.
We tried to run the model again on the new Dataproc cluster : https://cloud.google.com/dataproc-serverless/docs/release-notes. It has been issued on the 18th of December. And now its working fine, we are able to seamlessly add and remove columns from the python model without manual table deletion.
As we have not changed anything else apart from new Dataproc version, we assumed that could be an issue (which hopefully means its not dbt cloud bug and issue will not reappear!)

I have also updated the thread in dbt community as people were experiencing exactly same problem:
https://discourse.getdbt.com/t/how-to-config-to-allow-new-fields-when-running-pyspark-model-in-gcp-dataproc-serverless/9455/2

Hope it will work for them as well!

from dbt-bigquery.

Thanks for reaching out @Lana-hl !

On first blush, this sounds vaguely similar to #1047

Could you provide the relevant log output with the exact error message you got?

To allow us to reproduce this issue on our end, could you provide the example code for a foo.py? And also any relevant dbt commands to run?

e.g, we're looking for something similar to this:

models/foo.py

import ...

def model(dbt, session):

    my_sql_model_df = dbt.ref("my_sql_model")

    final_df = ...  

    return final_df

Commands:

dbt build -s foo

Any other steps that would allow us to reproduce what you are seeing.

from dbt-bigquery.

Glad the new Dataproc version is working for you @katerina-kogan !
And thanks adding your experience to that Discourse post.

I'm going to close this as resolved outside of dbt-biquery -- just let us know if this pops up again and we can either open a new issue or re-open this one.

from dbt-bigquery.