For users of dbt Cloud, generating a diagram could be much easier if they didn’t have to download manifest/catalog files but instead query dbt Cloud for them

Describe the solution you'd like
Providing an API token and an environment ID, do not require the use of dbt core to generate manifest files, instead use graphQL to get required information from metadata

https://docs.getdbt.com/docs/dbt-cloud-apis/discovery-querying

Describe alternatives you've considered
Could use the regular dbt Cloud rest API to get the latest artefact files vs discovery API but requires passing in of job IDs which is lest convenient

Describe the bug
I get the following error invoking the dbterd CLI:

Traceback (most recent call last):
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/bin/dbterd", line 5, in <module>
    from dbterd.main import main
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/main.py", line 1, in <module>
    from dbterd.cli import main as cli
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/cli/main.py", line 6, in <module>
    from dbterd.adapters.base import Executor
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/adapters/base.py", line 3, in <module>
    from types import NotImplementedType
ImportError: cannot import name 'NotImplementedType' from 'types' (/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/types.py)

To Reproduce
Steps to reproduce the behavior:

python3.9 -m venv .venv
source .venv/bin.activate
pip install dbterd==1.12.0
dbterd --help

Expected behavior
To be able to use the CLI.

Desktop

• OS: Ubuntu 22.10
• Version: 1.12.0

Additional context
I also tried 1.11.0

Traceback (most recent call last):
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/bin/dbterd", line 5, in <module>
    from dbterd.main import main
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/main.py", line 1, in <module>
    from dbterd.cli import main as cli
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/cli/main.py", line 6, in <module>
    from dbterd.adapters.base import Executor
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/adapters/base.py", line 9, in <module>
    from dbterd.adapters.dbt_cloud.administrative import DbtCloudArtifact
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/adapters/dbt_cloud/administrative.py", line 4, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'

After installing requests manually

Traceback (most recent call last):
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/bin/dbterd", line 5, in <module>
    from dbterd.main import main
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/main.py", line 1, in <module>
    from dbterd.cli import main as cli
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/cli/main.py", line 6, in <module>
    from dbterd.adapters.base import Executor
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/adapters/base.py", line 9, in <module>
    from dbterd.adapters.dbt_cloud.administrative import DbtCloudArtifact
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/adapters/dbt_cloud/administrative.py", line 6, in <module>
    from dbterd.helpers import file
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/helpers/file.py", line 5, in <module>
    from dbt_artifacts_parser import parser
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbt_artifacts_parser/parser.py", line 20, in <module>
    from dbt_artifacts_parser.parsers.manifest.manifest_v1 import ManifestV1
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbt_artifacts_parser/parsers/manifest/manifest_v1.py", line 17, in <module>
    class ManifestMetadata(BaseParserModel):
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/pydantic/_internal/_model_construction.py", line 178, in __new__
    set_model_fields(cls, bases, config_wrapper, types_namespace)
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/pydantic/_internal/_model_construction.py", line 452, in set_model_fields
    fields, class_vars = collect_model_fields(cls, bases, config_wrapper, types_namespace, typevars_map=typevars_map)
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/pydantic/_internal/_fields.py", line 122, in collect_model_fields
    type_hints = get_cls_type_hints_lenient(cls, types_namespace)
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/pydantic/_internal/_typing_extra.py", line 212, in get_cls_type_hints_lenient
    hints[name] = eval_type_lenient(value, globalns, localns)
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/pydantic/_internal/_typing_extra.py", line 224, in eval_type_lenient
    return eval_type_backport(value, globalns, localns)
  File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/pydantic/_internal/_typing_extra.py", line 240, in eval_type_backport
    return typing._eval_type(  # type: ignore
  File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/typing.py", line 292, in _eval_type
    return t._evaluate(globalns, localns, recursive_guard)
  File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/typing.py", line 554, in _evaluate
    eval(self.__forward_code__, globalns, localns),
  File "<string>", line 1, in <module>
TypeError: constr() got an unexpected keyword argument 'regex'

Is your feature request related to a problem? Please describe.
Not a problem, more of an enhancement.

Describe the solution you'd like
Have it modify manifest.json to include the mermaid code into the docs.
It should probably be added as a description for all tables in the model.

Is your feature request related to a problem? Please describe.

• Make top pin to <1.0.0
• Add WARN to upgrade the package for new artifact version which not supported yet
• Add -cv catalog version option

Is your feature request related to a problem? Please describe.
If dbt project was using dbt_constraints, they're using the test named foreign_key.

Describe the solution you'd like
We should support that with the configurable input of the test name. Default test name containing relationship wording.

Describe alternatives you've considered
N/A

Additional context
Sample usage of the dbt-contrainst package:

  - name: DIM_ORDER_LINES
    columns:
      # Single column inline constraints
      - name: OL_PK
        tests:
          - dbt_constraints.primary_key
      - name: OL_UK
        tests:
          - dbt_constraints.unique_key
      - name: OL_CUSTKEY
        tests:
          - dbt_constraints.foreign_key:
              pk_table_name: ref('DIM_CUSTOMERS')
              pk_column_name: C_CUSTKEY

Is your feature request related to a problem? Please describe.
New output as D2

Describe the solution you'd like
D2 tools is here

Describe alternatives you've considered
Existing outputs

Additional context

• Check blogpost
• Sample code:

car: {
  shape: sql_table

  id: int {constraint: primary_key}
  last_updated: timestamp with time zone

  make: string
  model: string
  year: int
}

factory: {
  shape: sql_table

  id: int {constraint: primary_key}

  country: string
}

factory_car_support: {
  shape: sql_table

  id: int {constraint: primary_key}

  factory: int {constraint: foreign_key}
  car: int {constraint: foreign_key}
}

factory_car_support.factory -> factory.id
factory_car_support.car -> car.id

Describe the bug
When running dbterd run -s exposure:exposure_name I get a diagram of models related to that exposure. Prior to this I've ran dbt deps and dbt compile locally.

When I configure the dbt Cloud connection and run dbterd run-metadata -s exposure:exposure_name it returns nothing. I've tried multiple different syntaxes on the exposure name but can't get it working.

To Reproduce
Steps to reproduce the behavior:

1. Configure dbt Cloud environment
2. Run dbterd run-metadata -s exposure:exposure_name

Expected behavior
Returns a dbml model based on dbt Cloud Discovery API which contains models related to the exposure defined

Desktop (please complete the following information):

• OS: MacOS

Is your feature request related to a problem? Please describe.
DBT introduced model contracts in 1.5.

The constraints field has a way of setting up foreign keys as part of the contract. I don't see a way to use this feature in the dbterd documentation. My models are too big for the relationships tests to run.

Describe the solution you'd like
Supporting inferring relationships from DBT model contract native constraints.

If already possible, specifying so in the documentation would be great.

Describe alternatives you've considered
I can use the Snowflake dbt_constraints package, but I'd rather use the native DBT constraints feature if that's available. Also dbt_constraints functionality of creating Snowflake constraints is compatible with native model contracts so that's another reason I'd write the native code instead.

I could also use dbt_utils.relationships_where but I'm not sure how (maybe it's possible with the algorithm selection command line syntax?)

I can help writing the code (really like this project!), but want to make sure there's not something I'm missing first. Thanks!

Is your feature request related to a problem? Please describe.
We would like to run dbterd on GitHub Actions and add markdown files with mermaid as artifacts to the GitHub repository. However, each time we run dbterd, there is a subtle difference in the output markdown file. The rendered diagram is the same, but the order of the output tables and relationships is different, so essentially the same content is being judged as differing.

Describe the solution you'd like
Ensure ordering of tables and relationships. Perhaps apply a sorted function to the tables and relations in the return value of the parse function.

Describe alternatives you've considered
N/A

Additional context
This is probably due to the fact that the contents of manifest.json output by dbt are not guaranteed to be in order.

All selection rules are lowercase-d before comparison, it should preserve the input value instead.

Fix of #73 has ignored the capitalized node_name in the Exact method.

This is to come up with a wider enhancement to get it aligned to other comparison methods as well, and with case sensitive supported

Is your feature request related to a problem? Please describe.

For our usecase, we want to output 2 different ERD diagrams for 2 different sets of data models. These 2 usecases are however located in the same repository. Therefore, we want to setup a flow in which we execute 2 dbterd commands to generate 2 different DBML outputs. However, the name of the DBML output file is always output.dbml. (see: https://github.com/datnguye/dbterd/blob/a03af25c9efe9fe4f32208e4b1a2acd93a51df7a/dbterd/adapters/targets/dbml/dbml_test_relationship.py#L18C14-L18C20). This means we have to create two different directories in which to save the output files to. Also, the name 'output' is kind of generic. Finally, you can't create two seperate output files in the same directory because of this.

Describe the solution you'd like
Make the output file name dynamic and add an extra CLI argument to set a custom output file name.

Describe alternatives you've considered

• Use directory per usecase.

New option --dbt-auto-artifacts will work together with --dbt option in order to trigger the dbt artifact files generated for a dbt project.

The command will look like:

BEFORE:

cd /path/to/dbt/project
dbt docs generate
dbterd run -s +my_model --dbt 

AFTER:

cd /path/to/dbt/project
dbterd run -s +my_model --dbt --dbt-auto-artifacts

Describe the bug
Error: Could not open file 'manifest.json': File manifest.json is corrupted, please rebuild

To Reproduce
dbterd run -mv 10

Expected behavior
Generation of target ERD script

Desktop (please complete the following information):
OS: Ubuntu 22.04 (WSL2)

Additional context
Think this is similar type of issue as
#38

Is your feature request related to a problem? Please describe.
Sometimes the naming convention of the model alone is not enough to use a single -s or -ns to get desired selection.

Describe the solution you'd like
Allow multiple sections using -s and -ns flags, or regex pattern

Describe alternatives you've considered
Generate all models then manually edit.

Is your feature request related to a problem? Please describe.
Add new output as GraphViz

Additional context
Sample code:

digraph g {
  fontname="Helvetica,Arial,sans-serif"
  node [fontname="Helvetica,Arial,sans-serif"]
  edge [fontname="Helvetica,Arial,sans-serif"]
  graph [fontsize=30 labelloc="t" label="" splines=true overlap=false rankdir = "LR"];
  ratio = auto;
  "state0" [ style = "filled, bold" penwidth = 1 fillcolor = "white" fontname = "Courier New" shape = "Mrecord" label =<
    <table border="0" cellborder="0" cellpadding="3" bgcolor="white">
        <tr><td bgcolor="black" align="center" colspan="2"><font color="white">State 0</font></td></tr>
        <tr><td align="left" port="r0">(type) column1 </td></tr>
        <tr><td align="left" port="r1">(type) column2 </td></tr>
        <tr><td align="left" port="r2">(type) column3 </td></tr>
    </table>> ];
  "state1" [ style = "filled, bold" penwidth = 1 fillcolor = "white" fontname = "Courier New" shape = "Mrecord" label =<
    <table border="0" cellborder="0" cellpadding="3" bgcolor="white">
        <tr><td bgcolor="black" align="center" colspan="2"><font color="white">State 1</font></td></tr>
        <tr><td align="left" port="r0">(type) column1 </td></tr>
        <tr><td align="left" port="r1">(type) column2 </td></tr>
        <tr><td align="left" port="r2">(type) column3 </td></tr>
    </table>> ];
  "state5" [ style = "filled" penwidth = 1 fillcolor = "white" fontname = "Courier New" shape = "Mrecord" label =<
    <table border="0" cellborder="0" cellpadding="3" bgcolor="white">
        <tr><td bgcolor="black" align="center" colspan="2"><font color="white">State 5</font></td></tr>
        <tr><td align="left" port="r0">(type) column</td></tr>
    </table>> ];
  "state0" -> "state5" [ penwidth = 1 fontsize = 12 fontcolor = "black" label = "column1 = column" ];
  "state0" -> "state5" [ penwidth = 1 fontsize = 12 fontcolor = "black" label = "column1 = column" ];
  "state1" -> "state5" [ penwidth = 1 fontsize = 12 fontcolor = "black" label = "column1 = column" ];
}

Demo Online

Describe the bug
Hey 💯
I've been testing the Discovery API solution, but it failed because of my ephemeral models.

Graph example:

               "node": {
                "uniqueId": "model.my_project.my_ephemeral_model",
                "name": "my_ephemeral_model",
                "description": "",
                "database": "my_db",
                "schema": "my_schema",
                "alias": "my_ephemeral_model",
                "catalog": null
              }

In base.py def get_table_from_metadata if expects to get columns from the catalog:

    for column in model_metadata.get("node", {}).get("catalog", {}).get("columns", []):
        table.columns.append(
            Column(
                name=column.get("name", "").lower(),
                data_type=column.get("type", "").lower(),
                description=column.get("description", ""),
            )
        )

To Reproduce
Steps to reproduce the behavior:

1. Create a dbt ephemeral model
2. Rerun your dbt Cloud job having docs generated
3. run dbterd run-metadata

Expected behavior
It should handle ephemeral models in any way. I just put the loop into a if-clause, but I'm sure you'll find a good solution being aware of the full application structure.

Best,
Marvin

Hi,

Previously it worked to use dbterd run --dbt -s tag:<tag> but now i get the following message:

$ dbterd run --dbt -s tag:hub
2024-05-02 11:07:00,093 - dbterd - INFO - Run with dbterd==1.13.2 (main.py:54)
2024-05-02 11:07:00,101 - dbterd - ERROR - Unsupported Selection found: tag (base.py:100)
Usage: dbterd run [OPTIONS]
Try 'dbterd run -h' for help.

Error: Unsupported Selection found: tag

When testing the generated output from dbterd it works by adding the tag:<tag> into the string:

failed:

dbterd - DEBUG - Invoking: dbt --quiet --log-level none ls --resource-type model --select <model> --project-dir

working:

dbt --quiet --log-level none ls --resource-type model --select tag:<tag> --project-dir

To Reproduce
Steps to reproduce the behavior:

1. tag a model
2. run bterd run --dbt -s tag:<tag>

Expected behavior
What all objects within specific tag will be be included in dbterd

Describe the solution you'd like
Expose Python APIs officially

• API to get a whole ERD

  from dbterd.api import DbtErd
  
  erd = DbtErd().get_erd()
  print("erd (dbml):", erd)
  
  erd = DbtErd(target="mermaid").get_erd()
  print("erd (mermaid):", erd)

• API to get model's ERD: this model +1 level connected nodes

  from dbterd.api import DbtErd
  
  dim_prize_erd = DbtErd(target="mermaid").get_model_erd(
      node_unique_id="model.dbt_resto.dim_prize"
  )
  print("erd of dim_date (mermaid):", dim_prize_erd)

  Here is the output:

  erDiagram
    "MODEL.DBT_RESTO.DIM_PRIZE" {
      varchar prize_key
      nvarchar prize_name
      int prize_order
    }
    "MODEL.DBT_RESTO.FACT_RESULT" {
      varchar fact_result_key
      varchar box_key
      varchar prize_key
      date date_key
      int no_of_won
      float prize_value
      float prize_paid
      int is_prize_taken
    }
    "MODEL.DBT_RESTO.FACT_RESULT" }|--|| "MODEL.DBT_RESTO.DIM_PRIZE": prize_key
  

API docs will be generated by pdoc 👉 beta site
PS: Mainly to support dbt docs generate's result with the ERD enrichment

Give me a 👍 if you think it should be available asap! Thanks

Is your feature request related to a problem? Please describe.
The table of interest in the ER diagram we want to draw has a very large number of columns. The number of columns can be dozens, making it very difficult to see the relationships between tables in an ER diagram.

Describe the solution you'd like
Add an option to omit columns in the ER diagram to allow output of a diagram without columns. For example, mermaid can only draw relationships between tables.

• https://mermaid.js.org/syntax/entityRelationshipDiagram.html

Describe alternatives you've considered
N/A

Additional context
Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe.
It seems not to be possible to select models by using one or multiple tag.

Describe the solution you'd like
Adding the functionality to run dbterd over tags specified in models metadata.

models:
  - name: test
    config:
      materialized: incremental
      tags: test, test2
    columns:
      - name: test_key
        tests:
          - relationships_test:
              to: ref('test2')
              field: test_key

dbterd run -ad target -s tag:test
or a list of tags
dbterd run -ad target -s tag:test,test2

Is your feature request related to a problem? Please describe.
Recently added or modified functions are typed by mypy, but many of the older functions are not typed by mypy (although it helps that docstrings are listed!).

We would like to have more functions and arguments typed by mypy for code readability and to get more assistance from the IDE.

Describe the solution you'd like
Assigning types by mypy to the arguments and return values of untyped functions!

Describe alternatives you've considered
N/A

Additional context
I will send a pull request over the weekend or whenever I have free time.

Is your feature request related to a problem? Please describe.

Currently when you render a single model w/ dbterd it will only output that model itself without its referenced tables. I couldn't find an option to include referenced models in the output with the model I want to render. It would be nice if there was an option to include all of the referenced models in the final output as well. Refs can be found in the manifest under the node information for a given model.

Describe the solution you'd like
Possibly some sort of flag to include referenced tables in the final output

dbterd run -s "my_model" --include-refs

Describe the bug
Error: Could not open file 'manifest.json': File manifest.json is corrupted, please rebuild

To Reproduce
dbterd run with default settings

Expected behavior
Generation of target ERD script

Desktop (please complete the following information):

• OS: Ubuntu 22.04 (WSL2)

Additional context
Not sure if this is an issue or a feature request. Using dbt 1.5, which bumps the manifest.json version to v9. If dbt 1.5 isn't supported yet, it will be helpful to list supported dbt or manifest.json version.

First of all, great tool! I've been trying to find a tool like this that could generate ERD diagrams on the fly from dbt project files.

Is your feature request related to a problem? Please describe.
I would like to generate an independent graph per exposure in my dbt project. The CLI itself works pretty well but I feel limited by the options the --select commands provides since it's based on pattern matching.

Describe the solution you'd like
I would like to run
dbterd run -s "list:mymodel1,mymodel2" or something similar. Basically giving a comma separated list of models I would like to render in the graph.

Describe alternatives you've considered
I haven't figured any way I could work around this problem since I can't give the CLI tool multiple model names as input when they're in same schema

Is your feature request related to a problem? Please describe.

• Parse semantic_manifest.json and generate Semantic ERD -- see join logic: https://docs.getdbt.com/docs/build/join-logic
• Read dbt Cloud API and generate Semantic ERD

Describe the solution you'd like
TODO

Describe alternatives you've considered
N/A

Additional context
Give thumbs if you think it's super cool to have asap 👍

Is your feature request related to a problem? Please describe.
I am using BigQuery as my DWH. BigQuery sometimes has column types that contain < special characters such as Struct<first_name string, last_name string> or column names that contain . special characters, such as name.first_name in column names. Unfortunately, mermaid has a problem rendering markdown with such special characters.

• mermaid-js/mermaid#1895
• mermaid-js/mermaid#3335

An example would be something like this. Commas are also not allowed in mermaid 😭 .

erDiagram
  "my_package.user" {
    Struct<first_name string, last_name string> name
    string name.first_name
    string name.last_name
  }

Describe the solution you'd like
Replace special characters with, for example

erDiagram
  "my_package.user" {
    Struct[OMITTED] name
    string name__first_name
    string name__last_name
  }

For Struct<first_name string, last_name string> name, there are three problems with <, , and white spaces, and I didn't have a good idea, so I could only come up with Struct[OMITTED] name.

Columns that have such problems can be handled by not displaying them at #77 in the first place. However, since . may appear in relationships, we would like to consider replacing it with __.

Describe alternatives you've considered
Wait for mermaid to be modified to draw even when special characters are included...

Additional context
N/A

Is your feature request related to a problem? Please describe.
Support PlanUML ER as output

Describe the solution you'd like

• Add new output as PlanUML ER here
• Add docs for new output

Describe alternatives you've considered
N/A

Additional context
N/A

Please check: #3
currently supports only resource_type=model, we should optionally add different dbt types.

Similar to this scene:
I want to give an example. All our raw layer data is saved to S3 and corresponding external tables are created in Snowflake. Shouldn't we also add resource_type = source to the ERD?

Describe the bug
Similar to #43, dbt manifest.json of version 11 is unable to be parsed by dbterd:
Error: Could not open file 'manifest.json': File manifest.json is corrupted, please rebuild

To Reproduce
run dbterd run -mv 11

Expected behavior
Generation of target ERD script

Desktop (please complete the following information):

• OS: iOS

Additional context
When I upgraded dbt-artifacts-parser in my virtualenv to version 0.5.0 I was able to parse the manifest correctly.

Is your feature request related to a problem? Please describe.
Currently the relationship type is always assumed as one-to-many

Describe the solution you'd like
Support to detect the relationship type [one-to-many, many-to-one, one-to-one, many-to-many], based on the meta config of the test e.g.

  - name: DIM_ORDER_LINES
      - name: OL_CUSTKEY
        tests:
          - relationship:
              to: ref('DIM_CUSTOMERS')
              field: C_CUSTKEY
              meta:
                relationship_type: one-to-many

DBML Rel types:

<: one-to-many. E.g: users.id < posts.user_id
>: many-to-one. E.g: posts.user_id > users.id
-: one-to-one. E.g: users.id - user_infos.user_id
<>: many-to-many. E.g: authors.id <> books.id

Describe alternatives you've considered
Manual edit after output generated

Additional context
Enrich the ERD

Describe the bug
Say a relationship is defined in schema.yml on model X referencing model Y. However, sometimes in the output.dbml, the relationship is defined as Y => X, rather than X => Y. Furthermore, if the foreign_key column name on X is a different name from the primary_key column name on Y, then those are swapped as well. This also causes "ghost columns" to appear in the involved tables. For example, if the foreign_key column on model X is 'F', and the primary_key on model Y is 'P', then the relationship will sometimes be defined as: X[P] => Y[F]. This creates a ghost column 'F' on table Y, and a ghost column 'P' on table X.

To Reproduce
Steps to reproduce the behavior:
This behavior is random. See "additional context" for discovery

Expected behavior
The relationship in output.dbml reflects the correct column names and direction of the relation.

Desktop (please complete the following information):

• OS: mac
• Version dbterd v1.2.0

Additional context
I dug into the dbterd source code a bit and believe the problem is related to the line of code table_map=manifest.parent_map[x] in the test_relationship.py. It appears that this code expects the first element in the pair of the parent_map to be the foreign model, and the second element in the pair is the "this" model. However, looking in my manifest.json, it looks like DBT makes no such guarantees; as far as its concerned it is an unordered pair, and the order is likely some result of the order of each model in the DAG that it was compiled.

So, I think there needs to be some small logic that determines if the pair at manifest.parent_map[x] needs to be swapped before it is assigned to table_map.

Is your feature request related to a problem? Please describe.
Currently, dbterd cannot draw parent-child relationships such as categories.

For example:

version: 2
sources:
  - name: my_dataset
    database: my_project
    tables:
      - name: Category
        freshness:
        columns:
          - name: parentCategoryId
            tests:
              - relationships:
                  to: source("my_project", "Category")
                  field: categoryId

If you include a test for these recursive relationships, dbterd will fail with the following error

% dbterd run -t mermaid 
2024-02-18 00:04:15,812 - dbterd - INFO - Run with dbterd==1.11.0b1 (main.py:54)
2024-02-18 00:04:15,812 - dbterd - INFO - Using dbt artifact dir at: /Users/yasuhisa.yoshida/my_project/target (base.py:76)
Traceback (most recent call last):
  File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/bin/dbterd", line 8, in <module>
    sys.exit(main())
  File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/main.py", line 6, in main
    cli.dbterd()
  File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/cli/params.py", line 75, in wrapper
    return func(*args, **kwargs)  # pragma: no cover
  File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/cli/params.py", line 201, in wrapper
    return func(*args, **kwargs)  # pragma: no cover
  File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/cli/main.py", line 66, in run
    Executor(ctx).run(**kwargs)
  File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/adapters/base.py", line 32, in run
    self.__run_by_strategy(**kwargs)
  File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/adapters/base.py", line 200, in __run_by_strategy
    result = operation(manifest=manifest, catalog=catalog, **kwargs)
  File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/adapters/targets/mermaid/mermaid_test_relationship.py", line 17, in run
    return ("output.md", parse(manifest, catalog, **kwargs))
  File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/adapters/targets/mermaid/mermaid_test_relationship.py", line 80, in parse
    tables, relationships = test_relationship.parse(
  File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/adapters/algos/test_relationship.py", line 80, in parse
    relationships = base.get_relationships(manifest=manifest, **kwargs)
  File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/adapters/algos/base.py", line 422, in get_relationships
    refs = [
  File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/adapters/algos/base.py", line 425, in <listcomp>
    table_map=get_table_map(test_node=manifest.nodes[x], **kwargs),
  File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/adapters/algos/base.py", line 585, in get_table_map
    if f'("{map[1].split(".")[-1]}")'.lower() in to_model.replace("'", '"').lower():
IndexError: list index out of range

The relevant code is below.

• https://github.com/datnguye/dbterd/blob/main/dbterd/adapters/algos/base.py#L577-L592

In these cases, dbterd will fail because the depends_on.nodes in manifest.json does not contain the two elements.

  "depends_on": {
    "macros": [
      "macro.dbt.test_relationships",
    ],
    "nodes": [
      "source.my_project.my_dataset.Category"
    ]
  },

Describe the solution you'd like
Allow drawing even when recursive relationships are involved. The simplest solution would be to modify the relevant code as follows.

    map = test_node.depends_on.nodes or []
    if len(map) == 1:
        return [map[0], map[0]]
    ....

Describe alternatives you've considered
N/A

Describe the bug
ER diagram is not output when node_name is capitalized.

To Reproduce
Our data source is exported from firebase, so our data source has capitalized table names.

version: 2
sources:
  - name: my_model
    database: my-project
    tables:
      - name: User
      ...

Running dbterd run in this case will only output erDiagram in target/output.md and will fail to generate an ER diagram.

% dbterd run -t mermaid --entity-name-format schema.table \
  --resource-type source \
  --select exact:source.my_model.firestore_export.User

Expected behavior
Generation of target ERD

Desktop (please complete the following information):

• OS: macOS Monterey
• Version: 1.9.1

Additional context
Add any other context about the problem here.

Describe the bug
Hey @datnguye

On the run-metadata command I've been using the --entity-name-format flag.
It applies the format on the tables inside the output.dbml file. However it's not applied on the relationships.
Hence the dbml compiler like dbdocs complains that it cant find the respective tables.

To Reproduce
Steps to reproduce the behavior:

1. dbterd run-metadata -o "./target/" --entity-name-format table
2. dbdocs build "./target/output.dbml"

Expected behavior
Selected entity-name-format on Refs.

Thanks for looking into it!
Best, Marvin

Describe the bug
If using pip to update dbterd, a previous version of dbt-artifacts-parser may satisfy the pinned version but not have the latest manifest object versions defined (e.g. ManifestV11)

To Reproduce
Attempt to pip install --upgrade dbterd with an older version of dbt-artifacts-parser installed (could use v0.4.0 for example). Executing dbterd at this time will raise an error:
ModuleNotFoundError: No module named 'dbt_artifacts_parser.parsers.manifest.manifest_v11

Expected behavior
required versions in the toml file could be more specific and cause pip to auto-update this dependency as needed when upgrading dbterd

Desktop (please complete the following information):

• OS: [macOS]
• Version [1.12.0]

Describe the bug
This bug is coming from dbterd package.

To Reproduce
Steps to reproduce the behavior:

1. Install dbterd
2. dbt docs gererate
3. dbterd run -ad /path/to/target/ -o "/path/to/output

Expected behavior
I expect it to run successfully.

Screenshots
If applicable, add screenshots to help explain your problem.

$ dbterd run -ad path/to/target/ -o path/to/output
2024-01-01 15:23:06,665 - dbterd - INFO - Run with dbterd==1.7.2 (main.py:54)
2024-01-01 15:23:06,666 - dbterd - INFO - Using dbt project dir at: /Users/sunnyshah/CloudFactory/edw-dbt/target (base.py:45)
2024-01-01 15:23:06,666 - dbterd - INFO - Using dbt artifact dir at: /Users/sunnyshah/CloudFactory/edw-dbt/target (base.py:69)
Traceback (most recent call last):
  File "/Users/sunnyshah/.pyenv/versions/edw_dbt/bin/dbterd", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/main.py", line 6, in main
    cli.dbterd()
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/cli/params.py", line 117, in wrapper
    return func(*args, **kwargs)  # pragma: no cover
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/cli/main.py", line 75, in run
    Executor(ctx).run(**kwargs)
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/base.py", line 30, in run
    self.__run_by_strategy(**kwargs)
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/base.py", line 162, in __run_by_strategy
    result = operation(manifest=manifest, catalog=catalog, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/targets/dbml/dbml_test_relationship.py", line 16, in run
    return ("output.dbml", parse(manifest, catalog, **kwargs))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/targets/dbml/dbml_test_relationship.py", line 29, in parse
    tables, relationships = test_relationship.parse(
                            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/algos/test_relationship.py", line 38, in parse
    relationships = get_relationships(manifest=manifest, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/algos/test_relationship.py", line 75, in get_relationships
    refs = [
           ^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/algos/test_relationship.py", line 78, in <listcomp>
    table_map=get_table_map(test_node=manifest.nodes[x], **kwargs),
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/algos/test_relationship.py", line 174, in get_table_map
    if f'("{map[1].split(".")[-1]}")'.lower() in to_model.replace("'", '"').lower():
            ~~~^^^
IndexError: list index out of range

Desktop (please complete the following information):

• OS: Macbook Pro Sonoma 14.2.1 (23C71)
• dbterd version: 1.7.2

Additional context
Add any other context about the problem here.

Is your feature request related to a problem? Please describe.

• Support GraphQL query - built-in query and allow custom query
• New command to connect to dbt Cloud Discovery API

Related to #63

Describe the solution you'd like
The command would be something like dbterd run-metadata [--dbt-cloud-query-file-path /path/to/erd.gql]

Describe alternatives you've considered
Use the existing option with dbt Cloud Admistrative API to download files and lately using dbterd run --dbt-cloud ... command

Hi, want to start off by saying this is a very neat tool, trying to implement this into our workflow at work, but had one quick question.

Is there are particular reason/restricting factor that does not allow for you to pull in the column descriptions, and add them to the .dbml output, so that they show up on the db docs page?

Is your feature request related to a problem? Please describe.
All of my models have a description that describes what it is for and what it is not for. Also it tells limitations.

Describe the solution you'd like
To automatically use the "description" as the "note" in the generated DBML for dbdocs.io

Describe alternatives you've considered
Copying them manually but this is too much work.

Additional context
Example DBML from dbdocs:

Is your feature request related to a problem? Please describe.

Currently the ERD generator will output the ERD with the name of the node within the dbt project, i.e. model.project.name. It would be nice if there was an option to render the ERD with the actual table name itself like database.schema.table or just schema.table instead of the name of the dbt node. Not everyone who views an ERD is involved with dbt development so the node names can throw people off.

Describe the solution you'd like
The ability to be able to output an ERD with the actual tablename instead of the dbt node name.

Describe the bug
Double quotes in a column description in schema.yml produces outbut.dbml that does not parse because the ""s are not \ escaped.

To Reproduce
Steps to reproduce the behavior:

1. Create any column description with quotes, e.g:

- name: my_col
  description: >-
    Here is "my" test column

2. Observe output in output.dbml. Output is:
   "my_col" "character varying(256)" [note: "Here is "my" test column"]

Expected behavior
Output in output.dbml should be:
"my_col" "character varying(256)" [note: "Here is \"my\" test column"]

Desktop (please complete the following information):

• OS: mac
• Version: dbterd v1.2.0

Additional context
I'm not positive that the backslash-escaped quotes is a requirement of the .dbml spec, however adding them allowed my consuming app to successfully load the output.dbml without throwing a parse error.

Describe the bug
When installing the softare using pip install upgrade I get the following error:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
hologram 0.0.15 requires jsonschema<4.0,>=3.0, but you have jsonschema 4.17.3 which is incompatible.

To Reproduce
Steps to reproduce the behavior:

1. Python version 3.10.8 dbt version 1.4.5
2. run pip install dbterd --upgrade

Expected behavior
installation without errors

Is your feature request related to a problem? Please describe.
When running dbterd run -s "model.name", I notice that it includes all models that start with "model.name", even if I just want it to run it for the exact match "model.name". I don't see a way of telling dbterd to run for exact name matches.

Describe the solution you'd like
An option for the run command to run an exact match on the model name, rather than assuming it's a partial name match.

Describe alternatives you've considered
I've considered running dbt ls --exclude to output all models that are not the ones I'm interested in, and then piping that into a dbter run -ns command. Problem is, I have a fairly large dbt project, and subprocess cannot handle more than 32k characters.

Additional context