Missing dbt_project.yml file about dbt-checkpoint HOT 11 CLOSED

Yeah, I know, I went through the same process. Running the command from the dbt folder did not work (works for normal commands but not when they are ran as hooks),then I tried adding the project-dir but realised it wasn't available yet and then gave up and moved the dbt project ...

from dbt-checkpoint.

I am also having the same issue, however, I can't change the project structure since we have many other things at place. What do you suggest in this case?

from dbt-checkpoint.

@PeteCorbettWS Thank you so much for your input. It worked with --manifest and --catalog arguments.

from dbt-checkpoint.

Have you seen that you can set the dbt dir in a .dbt_checkpoint configuration file? We added this bec we recommend the dbt project be in a subdirectory, not in the repo root.

from dbt-checkpoint.

I was getting the same error for the dbt-docs-generate hook but I just had to change my dbt project folder structure to the correct one to make it work :)

from dbt-checkpoint.

@olga-tree what do you mean by correct dbt project folder structure? Isn't having .pre-commit-config.yaml in dbt root folder enough?

from dbt-checkpoint.

It is but I realised I was getting this error because my dbt project folder was inside another folder and then this was a git repo. So I can run dbt commands from the terminal if I am in the right folder but the pre-commit was still searching for my dbt_project.yml at the root of that repo.

from dbt-checkpoint.

Ah okay, so you moved your dbt project to the root folder then. I wish there was a way to run this but keeping the same structure! dbt docs generate doesnt allow project-dir command but maybe it will be implemented at some point. The alternative is to do a cd inside the dbt folder and run the command from there.

from dbt-checkpoint.

@sagar-raythatha There are a number discussions around this in the pre-commit repo e.g. here

Using this approach I have been able to retain dbt project as a sub folder of my repo.

You just need to remember to tell pre-commit-dbt where to find the manifest.json and catalog.json files for the hooks that require them via the --manifest and --catalog arguments e.g. check-model-has-all-columns

Still testing out this approach but appears to be doing the job so far

from dbt-checkpoint.

What about when you need to run dbt-docs-generate?

from dbt-checkpoint.

@DakotaMichael I haven't looked at this for quite some time so there may be a better way to handle dbt-docs-generate but I have written a custom shell script to perform pre-commit setup actions inc. dbt-docs-generate.

I cant share the code here (technically owned by my organisation) but essentially it:

• identifies the script execution location and cds into dbt project root directory if not executing from there
• sets the expected location of manifest.json file (this is referenced later on)
• checks for DBT_PROFILES_DIR env var and presence of profiles.yml and errors out if either do not exist
• identify the target environment (from profiles.yml). We use this to run pre-commit differently between 'local' runs and CI-CD (github workflow) runs
• if executing as a 'local' pre-commit run
  • check for the existence of a manifest.json file. Warn and exit if file does not exist (we expect developers to have executed a 'dbt run/build' before committing code)
  • run dbt docs generate using state selection to only run for modified models
• if executing as a 'ci-cd' pre-commit run
  • run dbt docs generate --select $CHANGED_FILES where $CHANGED_FILES are the models that have changed between the repos main branch and incoming feature/test branch

.pre-commit-config.yml configured to execute the script as the first hook

  hooks:
  #setup
  - id: dbt-docs-generate
    entry: bash -c  "./.pre-commit-artifacts-check.sh"
    name: Generate dbt artifacts files required for pre-commit tests
    verbose: true
    log_file: .pre-commit/pre-commit.log
    fail_fast: true

hope that helps...

from dbt-checkpoint.