๐ญ Iโm a ๐ฉธof D(ata) who is currently working on Data field using:
๐ฌ Find me on:
Generate the ERD as a code from dbt artifacts
Home Page: https://dbterd.datnguyen.de/
License: MIT License
Is your feature request related to a problem? Please describe.
Sometimes the naming convention of the model alone is not enough to use a single -s or -ns to get desired selection.
Describe the solution you'd like
Allow multiple sections using -s and -ns flags, or regex pattern
Describe alternatives you've considered
Generate all models then manually edit.
Describe the bug
Error: Could not open file 'manifest.json': File manifest.json is corrupted, please rebuild
To Reproduce
dbterd run
with default settings
Expected behavior
Generation of target ERD script
Desktop (please complete the following information):
Additional context
Not sure if this is an issue or a feature request. Using dbt 1.5, which bumps the manifest.json version to v9. If dbt 1.5 isn't supported yet, it will be helpful to list supported dbt or manifest.json version.
Please check: #3
currently supports only resource_type=model, we should optionally add different dbt types.
Similar to this scene:
I want to give an example. All our raw layer data is saved to S3 and corresponding external tables are created in Snowflake. Shouldn't we also add resource_type = source to the ERD?
First of all, great tool! I've been trying to find a tool like this that could generate ERD diagrams on the fly from dbt project files.
Is your feature request related to a problem? Please describe.
I would like to generate an independent graph per exposure in my dbt project. The CLI itself works pretty well but I feel limited by the options the --select
commands provides since it's based on pattern matching.
Describe the solution you'd like
I would like to run
dbterd run -s "list:mymodel1,mymodel2"
or something similar. Basically giving a comma separated list of models I would like to render in the graph.
Describe alternatives you've considered
I haven't figured any way I could work around this problem since I can't give the CLI tool multiple model names as input when they're in same schema
Is your feature request related to a problem? Please describe.
Currently when you render a single model w/ dbterd
it will only output that model itself without its referenced tables. I couldn't find an option to include referenced models in the output with the model I want to render. It would be nice if there was an option to include all of the referenced models in the final output as well. Refs can be found in the manifest under the node information for a given model.
Describe the solution you'd like
Possibly some sort of flag to include referenced tables in the final output
dbterd run -s "my_model" --include-refs
Is your feature request related to a problem? Please describe.
Currently, dbterd
cannot draw parent-child relationships such as categories.
For example:
version: 2
sources:
- name: my_dataset
database: my_project
tables:
- name: Category
freshness:
columns:
- name: parentCategoryId
tests:
- relationships:
to: source("my_project", "Category")
field: categoryId
If you include a test for these recursive relationships
, dbterd
will fail with the following error
% dbterd run -t mermaid
2024-02-18 00:04:15,812 - dbterd - INFO - Run with dbterd==1.11.0b1 (main.py:54)
2024-02-18 00:04:15,812 - dbterd - INFO - Using dbt artifact dir at: /Users/yasuhisa.yoshida/my_project/target (base.py:76)
Traceback (most recent call last):
File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/bin/dbterd", line 8, in <module>
sys.exit(main())
File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/main.py", line 6, in main
cli.dbterd()
File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/cli/params.py", line 75, in wrapper
return func(*args, **kwargs) # pragma: no cover
File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/cli/params.py", line 201, in wrapper
return func(*args, **kwargs) # pragma: no cover
File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/cli/main.py", line 66, in run
Executor(ctx).run(**kwargs)
File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/adapters/base.py", line 32, in run
self.__run_by_strategy(**kwargs)
File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/adapters/base.py", line 200, in __run_by_strategy
result = operation(manifest=manifest, catalog=catalog, **kwargs)
File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/adapters/targets/mermaid/mermaid_test_relationship.py", line 17, in run
return ("output.md", parse(manifest, catalog, **kwargs))
File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/adapters/targets/mermaid/mermaid_test_relationship.py", line 80, in parse
tables, relationships = test_relationship.parse(
File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/adapters/algos/test_relationship.py", line 80, in parse
relationships = base.get_relationships(manifest=manifest, **kwargs)
File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/adapters/algos/base.py", line 422, in get_relationships
refs = [
File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/adapters/algos/base.py", line 425, in <listcomp>
table_map=get_table_map(test_node=manifest.nodes[x], **kwargs),
File "/Users/yasuhisa.yoshida/.pyenv/versions/3.10.0/lib/python3.10/site-packages/dbterd/adapters/algos/base.py", line 585, in get_table_map
if f'("{map[1].split(".")[-1]}")'.lower() in to_model.replace("'", '"').lower():
IndexError: list index out of range
The relevant code is below.
In these cases, dbterd will fail because the depends_on.nodes
in manifest.json
does not contain the two elements.
"depends_on": {
"macros": [
"macro.dbt.test_relationships",
],
"nodes": [
"source.my_project.my_dataset.Category"
]
},
Describe the solution you'd like
Allow drawing even when recursive relationships are involved. The simplest solution would be to modify the relevant code as follows.
map = test_node.depends_on.nodes or []
if len(map) == 1:
return [map[0], map[0]]
....
Describe alternatives you've considered
N/A
Is your feature request related to a problem? Please describe.
I am using BigQuery as my DWH. BigQuery sometimes has column types that contain <
special characters such as Struct<first_name string, last_name string>
or column names that contain .
special characters, such as name.first_name
in column names. Unfortunately, mermaid has a problem rendering markdown with such special characters.
An example would be something like this. Commas are also not allowed in mermaid ๐ญ .
erDiagram
"my_package.user" {
Struct<first_name string, last_name string> name
string name.first_name
string name.last_name
}
Describe the solution you'd like
Replace special characters with, for example
erDiagram
"my_package.user" {
Struct[OMITTED] name
string name__first_name
string name__last_name
}
For Struct<first_name string, last_name string> name
, there are three problems with <
, ,
and white spaces, and I didn't have a good idea, so I could only come up with Struct[OMITTED] name
.
Columns that have such problems can be handled by not displaying them at #77 in the first place. However, since .
may appear in relationships, we would like to consider replacing it with __
.
Describe alternatives you've considered
Wait for mermaid to be modified to draw even when special characters are included...
Additional context
N/A
Describe the bug
Hey @datnguye
On the run-metadata
command I've been using the --entity-name-format
flag.
It applies the format on the tables inside the output.dbml file. However it's not applied on the relationships.
Hence the dbml compiler like dbdocs complains that it cant find the respective tables.
To Reproduce
Steps to reproduce the behavior:
dbterd run-metadata -o "./target/" --entity-name-format table
dbdocs build "./target/output.dbml"
Expected behavior
Selected entity-name-format on Refs.
Thanks for looking into it!
Best, Marvin
Is your feature request related to a problem? Please describe.
Currently the ERD generator will output the ERD with the name of the node within the dbt project, i.e. model.project.name
. It would be nice if there was an option to render the ERD with the actual table name itself like database.schema.table
or just schema.table
instead of the name of the dbt node. Not everyone who views an ERD is involved with dbt development so the node names can throw people off.
Describe the solution you'd like
The ability to be able to output an ERD with the actual tablename instead of the dbt node name.
Is your feature request related to a problem? Please describe.
-cv
catalog version optionHi,
Previously it worked to use dbterd run --dbt -s tag:<tag>
but now i get the following message:
$ dbterd run --dbt -s tag:hub
2024-05-02 11:07:00,093 - dbterd - INFO - Run with dbterd==1.13.2 (main.py:54)
2024-05-02 11:07:00,101 - dbterd - ERROR - Unsupported Selection found: tag (base.py:100)
Usage: dbterd run [OPTIONS]
Try 'dbterd run -h' for help.
Error: Unsupported Selection found: tag
When testing the generated output from dbterd it works by adding the tag:<tag>
into the string:
failed:
dbterd - DEBUG - Invoking: dbt --quiet --log-level none ls --resource-type model --select <model> --project-dir
working:
dbt --quiet --log-level none ls --resource-type model --select tag:<tag> --project-dir
To Reproduce
Steps to reproduce the behavior:
bterd run --dbt -s tag:<tag>
Expected behavior
What all objects within specific tag will be be included in dbterd
Describe the bug
When running dbterd run -s exposure:exposure_name
I get a diagram of models related to that exposure. Prior to this I've ran dbt deps
and dbt compile
locally.
When I configure the dbt Cloud connection and run dbterd run-metadata -s exposure:exposure_name
it returns nothing. I've tried multiple different syntaxes on the exposure name but can't get it working.
To Reproduce
Steps to reproduce the behavior:
dbterd run-metadata -s exposure:exposure_name
Expected behavior
Returns a dbml model based on dbt Cloud Discovery API which contains models related to the exposure defined
Desktop (please complete the following information):
Is your feature request related to a problem? Please describe.
Related to #63
Describe the solution you'd like
The command would be something like dbterd run-metadata [--dbt-cloud-query-file-path /path/to/erd.gql]
Describe alternatives you've considered
Use the existing option with dbt Cloud Admistrative API to download files and lately using dbterd run --dbt-cloud ...
command
Describe the bug
When installing the softare using pip install upgrade I get the following error:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
hologram 0.0.15 requires jsonschema<4.0,>=3.0, but you have jsonschema 4.17.3 which is incompatible.
To Reproduce
Steps to reproduce the behavior:
pip install dbterd --upgrade
Expected behavior
installation without errors
Is your feature request related to a problem? Please describe.
Not a problem, more of an enhancement.
Describe the solution you'd like
Have it modify manifest.json
to include the mermaid code into the docs.
It should probably be added as a description for all tables in the model.
Describe the bug
Similar to #43, dbt manifest.json of version 11 is unable to be parsed by dbterd:
Error: Could not open file 'manifest.json': File manifest.json is corrupted, please rebuild
To Reproduce
run dbterd run -mv 11
Expected behavior
Generation of target ERD script
Desktop (please complete the following information):
Additional context
When I upgraded dbt-artifacts-parser
in my virtualenv to version 0.5.0
I was able to parse the manifest correctly.
Describe the solution you'd like
Expose Python APIs officially
API to get a whole ERD
from dbterd.api import DbtErd
erd = DbtErd().get_erd()
print("erd (dbml):", erd)
erd = DbtErd(target="mermaid").get_erd()
print("erd (mermaid):", erd)
API to get model's ERD: this model +1 level connected nodes
from dbterd.api import DbtErd
dim_prize_erd = DbtErd(target="mermaid").get_model_erd(
node_unique_id="model.dbt_resto.dim_prize"
)
print("erd of dim_date (mermaid):", dim_prize_erd)
Here is the output:
erDiagram
"MODEL.DBT_RESTO.DIM_PRIZE" {
varchar prize_key
nvarchar prize_name
int prize_order
}
"MODEL.DBT_RESTO.FACT_RESULT" {
varchar fact_result_key
varchar box_key
varchar prize_key
date date_key
int no_of_won
float prize_value
float prize_paid
int is_prize_taken
}
"MODEL.DBT_RESTO.FACT_RESULT" }|--|| "MODEL.DBT_RESTO.DIM_PRIZE": prize_key
API docs will be generated by pdoc
๐ beta site
PS: Mainly to support dbt docs generate
's result with the ERD enrichment
Give me a ๐ if you think it should be available asap! Thanks
Describe the bug
If using pip to update dbterd, a previous version of dbt-artifacts-parser may satisfy the pinned version but not have the latest manifest object versions defined (e.g. ManifestV11)
To Reproduce
Attempt to pip install --upgrade dbterd with an older version of dbt-artifacts-parser installed (could use v0.4.0 for example). Executing dbterd at this time will raise an error:
ModuleNotFoundError: No module named 'dbt_artifacts_parser.parsers.manifest.manifest_v11
Expected behavior
required versions in the toml file could be more specific and cause pip to auto-update this dependency as needed when upgrading dbterd
Desktop (please complete the following information):
Is your feature request related to a problem? Please describe.
If dbt project was using dbt_constraints, they're using the test named foreign_key
.
Describe the solution you'd like
We should support that with the configurable input of the test name. Default test name containing relationship
wording.
Describe alternatives you've considered
N/A
Additional context
Sample usage of the dbt-contrainst package:
- name: DIM_ORDER_LINES
columns:
# Single column inline constraints
- name: OL_PK
tests:
- dbt_constraints.primary_key
- name: OL_UK
tests:
- dbt_constraints.unique_key
- name: OL_CUSTKEY
tests:
- dbt_constraints.foreign_key:
pk_table_name: ref('DIM_CUSTOMERS')
pk_column_name: C_CUSTKEY
Is your feature request related to a problem? Please describe.
Recently added or modified functions are typed by mypy, but many of the older functions are not typed by mypy (although it helps that docstrings are listed!).
We would like to have more functions and arguments typed by mypy for code readability and to get more assistance from the IDE.
Describe the solution you'd like
Assigning types by mypy to the arguments and return values of untyped functions!
Describe alternatives you've considered
N/A
Additional context
I will send a pull request over the weekend or whenever I have free time.
Describe the bug
This bug is coming from dbterd package.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
I expect it to run successfully.
Screenshots
If applicable, add screenshots to help explain your problem.
$ dbterd run -ad path/to/target/ -o path/to/output
2024-01-01 15:23:06,665 - dbterd - INFO - Run with dbterd==1.7.2 (main.py:54)
2024-01-01 15:23:06,666 - dbterd - INFO - Using dbt project dir at: /Users/sunnyshah/CloudFactory/edw-dbt/target (base.py:45)
2024-01-01 15:23:06,666 - dbterd - INFO - Using dbt artifact dir at: /Users/sunnyshah/CloudFactory/edw-dbt/target (base.py:69)
Traceback (most recent call last):
File "/Users/sunnyshah/.pyenv/versions/edw_dbt/bin/dbterd", line 8, in <module>
sys.exit(main())
^^^^^^
File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/main.py", line 6, in main
cli.dbterd()
File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/cli/params.py", line 117, in wrapper
return func(*args, **kwargs) # pragma: no cover
^^^^^^^^^^^^^^^^^^^^^
File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/cli/main.py", line 75, in run
Executor(ctx).run(**kwargs)
File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/base.py", line 30, in run
self.__run_by_strategy(**kwargs)
File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/base.py", line 162, in __run_by_strategy
result = operation(manifest=manifest, catalog=catalog, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/targets/dbml/dbml_test_relationship.py", line 16, in run
return ("output.dbml", parse(manifest, catalog, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/targets/dbml/dbml_test_relationship.py", line 29, in parse
tables, relationships = test_relationship.parse(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/algos/test_relationship.py", line 38, in parse
relationships = get_relationships(manifest=manifest, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/algos/test_relationship.py", line 75, in get_relationships
refs = [
^
File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/algos/test_relationship.py", line 78, in <listcomp>
table_map=get_table_map(test_node=manifest.nodes[x], **kwargs),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sunnyshah/.pyenv/versions/3.11.0/envs/edw_dbt/lib/python3.11/site-packages/dbterd/adapters/algos/test_relationship.py", line 174, in get_table_map
if f'("{map[1].split(".")[-1]}")'.lower() in to_model.replace("'", '"').lower():
~~~^^^
IndexError: list index out of range
Desktop (please complete the following information):
Additional context
Add any other context about the problem here.
Is your feature request related to a problem? Please describe.
DBT introduced model contracts in 1.5.
The constraints field has a way of setting up foreign keys as part of the contract. I don't see a way to use this feature in the dbterd documentation. My models are too big for the relationships tests to run.
Describe the solution you'd like
Supporting inferring relationships from DBT model contract native constraints.
If already possible, specifying so in the documentation would be great.
Describe alternatives you've considered
I can use the Snowflake dbt_constraints package, but I'd rather use the native DBT constraints feature if that's available. Also dbt_constraints functionality of creating Snowflake constraints is compatible with native model contracts so that's another reason I'd write the native code instead.
I could also use dbt_utils.relationships_where
but I'm not sure how (maybe it's possible with the algorithm selection command line syntax?)
I can help writing the code (really like this project!), but want to make sure there's not something I'm missing first. Thanks!
Is your feature request related to a problem? Please describe.
We would like to run dbterd on GitHub Actions and add markdown files with mermaid as artifacts to the GitHub repository. However, each time we run dbterd, there is a subtle difference in the output markdown file. The rendered diagram is the same, but the order of the output tables and relationships is different, so essentially the same content is being judged as differing.
Describe the solution you'd like
Ensure ordering of tables and relationships. Perhaps apply a sorted
function to the tables and relations in the return value of the parse
function.
Describe alternatives you've considered
N/A
Additional context
This is probably due to the fact that the contents of manifest.json output by dbt are not guaranteed to be in order.
Describe the bug
ER diagram is not output when node_name
is capitalized.
To Reproduce
Our data source is exported from firebase, so our data source has capitalized table names.
version: 2
sources:
- name: my_model
database: my-project
tables:
- name: User
...
Running dbterd run
in this case will only output erDiagram
in target/output.md
and will fail to generate an ER diagram.
% dbterd run -t mermaid --entity-name-format schema.table \
--resource-type source \
--select exact:source.my_model.firestore_export.User
Expected behavior
Generation of target ERD
Desktop (please complete the following information):
Additional context
Add any other context about the problem here.
Describe the bug
Error: Could not open file 'manifest.json': File manifest.json is corrupted, please rebuild
To Reproduce
dbterd run -mv 10
Expected behavior
Generation of target ERD script
Desktop (please complete the following information):
OS: Ubuntu 22.04 (WSL2)
Additional context
Think this is similar type of issue as
#38
Describe the bug
Double quotes in a column description in schema.yml produces outbut.dbml that does not parse because the ""s are not \ escaped.
To Reproduce
Steps to reproduce the behavior:
- name: my_col
description: >-
Here is "my" test column
"my_col" "character varying(256)" [note: "Here is "my" test column"]
Expected behavior
Output in output.dbml should be:
"my_col" "character varying(256)" [note: "Here is \"my\" test column"]
Desktop (please complete the following information):
Additional context
I'm not positive that the backslash-escaped quotes is a requirement of the .dbml spec, however adding them allowed my consuming app to successfully load the output.dbml without throwing a parse error.
Is your feature request related to a problem? Please describe.
Currently the relationship type is always assumed as one-to-many
Describe the solution you'd like
Support to detect the relationship type [one-to-many, many-to-one, one-to-one, many-to-many], based on the meta config of the test e.g.
- name: DIM_ORDER_LINES
- name: OL_CUSTKEY
tests:
- relationship:
to: ref('DIM_CUSTOMERS')
field: C_CUSTKEY
meta:
relationship_type: one-to-many
DBML Rel types:
<: one-to-many. E.g: users.id < posts.user_id
>: many-to-one. E.g: posts.user_id > users.id
-: one-to-one. E.g: users.id - user_infos.user_id
<>: many-to-many. E.g: authors.id <> books.id
Describe alternatives you've considered
Manual edit after output generated
Additional context
Enrich the ERD
Describe the bug
I get the following error invoking the dbterd
CLI:
Traceback (most recent call last):
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/bin/dbterd", line 5, in <module>
from dbterd.main import main
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/main.py", line 1, in <module>
from dbterd.cli import main as cli
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/cli/main.py", line 6, in <module>
from dbterd.adapters.base import Executor
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/adapters/base.py", line 3, in <module>
from types import NotImplementedType
ImportError: cannot import name 'NotImplementedType' from 'types' (/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/types.py)
To Reproduce
Steps to reproduce the behavior:
python3.9 -m venv .venv
source .venv/bin.activate
pip install dbterd==1.12.0
dbterd --help
Expected behavior
To be able to use the CLI.
Desktop
1.12.0
Additional context
I also tried 1.11.0
Traceback (most recent call last):
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/bin/dbterd", line 5, in <module>
from dbterd.main import main
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/main.py", line 1, in <module>
from dbterd.cli import main as cli
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/cli/main.py", line 6, in <module>
from dbterd.adapters.base import Executor
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/adapters/base.py", line 9, in <module>
from dbterd.adapters.dbt_cloud.administrative import DbtCloudArtifact
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/adapters/dbt_cloud/administrative.py", line 4, in <module>
import requests
ModuleNotFoundError: No module named 'requests'
After installing requests
manually
Traceback (most recent call last):
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/bin/dbterd", line 5, in <module>
from dbterd.main import main
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/main.py", line 1, in <module>
from dbterd.cli import main as cli
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/cli/main.py", line 6, in <module>
from dbterd.adapters.base import Executor
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/adapters/base.py", line 9, in <module>
from dbterd.adapters.dbt_cloud.administrative import DbtCloudArtifact
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/adapters/dbt_cloud/administrative.py", line 6, in <module>
from dbterd.helpers import file
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbterd/helpers/file.py", line 5, in <module>
from dbt_artifacts_parser import parser
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbt_artifacts_parser/parser.py", line 20, in <module>
from dbt_artifacts_parser.parsers.manifest.manifest_v1 import ManifestV1
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/dbt_artifacts_parser/parsers/manifest/manifest_v1.py", line 17, in <module>
class ManifestMetadata(BaseParserModel):
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/pydantic/_internal/_model_construction.py", line 178, in __new__
set_model_fields(cls, bases, config_wrapper, types_namespace)
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/pydantic/_internal/_model_construction.py", line 452, in set_model_fields
fields, class_vars = collect_model_fields(cls, bases, config_wrapper, types_namespace, typevars_map=typevars_map)
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/pydantic/_internal/_fields.py", line 122, in collect_model_fields
type_hints = get_cls_type_hints_lenient(cls, types_namespace)
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/pydantic/_internal/_typing_extra.py", line 212, in get_cls_type_hints_lenient
hints[name] = eval_type_lenient(value, globalns, localns)
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/pydantic/_internal/_typing_extra.py", line 224, in eval_type_lenient
return eval_type_backport(value, globalns, localns)
File "/home/reuben/.cache/pypoetry/virtualenvs/dbterd-ext-IFoOWQ5l-py3.9/lib/python3.9/site-packages/pydantic/_internal/_typing_extra.py", line 240, in eval_type_backport
return typing._eval_type( # type: ignore
File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/typing.py", line 292, in _eval_type
return t._evaluate(globalns, localns, recursive_guard)
File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/typing.py", line 554, in _evaluate
eval(self.__forward_code__, globalns, localns),
File "<string>", line 1, in <module>
TypeError: constr() got an unexpected keyword argument 'regex'
Is your feature request related to a problem? Please describe.
Support PlanUML ER as output
Describe the solution you'd like
Describe alternatives you've considered
N/A
Additional context
N/A
Is your feature request related to a problem? Please describe.
When running dbterd run -s "model.name"
, I notice that it includes all models that start with "model.name", even if I just want it to run it for the exact match "model.name". I don't see a way of telling dbterd
to run for exact name matches.
Describe the solution you'd like
An option for the run
command to run an exact match on the model name, rather than assuming it's a partial name match.
Describe alternatives you've considered
I've considered running dbt ls --exclude
to output all models that are not the ones I'm interested in, and then piping that into a dbter run -ns
command. Problem is, I have a fairly large dbt project, and subprocess
cannot handle more than 32k characters.
Additional context
For users of dbt Cloud, generating a diagram could be much easier if they didnโt have to download manifest/catalog files but instead query dbt Cloud for them
Describe the solution you'd like
Providing an API token and an environment ID, do not require the use of dbt core to generate manifest files, instead use graphQL to get required information from metadata
https://docs.getdbt.com/docs/dbt-cloud-apis/discovery-querying
Describe alternatives you've considered
Could use the regular dbt Cloud rest API to get the latest artefact files vs discovery API but requires passing in of job IDs which is lest convenient
Is your feature request related to a problem? Please describe.
New output as D2
Describe the solution you'd like
D2 tools is here
Describe alternatives you've considered
Existing outputs
Additional context
car: {
shape: sql_table
id: int {constraint: primary_key}
last_updated: timestamp with time zone
make: string
model: string
year: int
}
factory: {
shape: sql_table
id: int {constraint: primary_key}
country: string
}
factory_car_support: {
shape: sql_table
id: int {constraint: primary_key}
factory: int {constraint: foreign_key}
car: int {constraint: foreign_key}
}
factory_car_support.factory -> factory.id
factory_car_support.car -> car.id
Is your feature request related to a problem? Please describe.
All of my models have a description that describes what it is for and what it is not for. Also it tells limitations.
Describe the solution you'd like
To automatically use the "description" as the "note" in the generated DBML for dbdocs.io
Describe alternatives you've considered
Copying them manually but this is too much work.
Describe the bug
Hey ๐ฏ
I've been testing the Discovery API solution, but it failed because of my ephemeral models.
Graph example:
"node": {
"uniqueId": "model.my_project.my_ephemeral_model",
"name": "my_ephemeral_model",
"description": "",
"database": "my_db",
"schema": "my_schema",
"alias": "my_ephemeral_model",
"catalog": null
}
In base.py def get_table_from_metadata if expects to get columns from the catalog:
for column in model_metadata.get("node", {}).get("catalog", {}).get("columns", []):
table.columns.append(
Column(
name=column.get("name", "").lower(),
data_type=column.get("type", "").lower(),
description=column.get("description", ""),
)
)
To Reproduce
Steps to reproduce the behavior:
Expected behavior
It should handle ephemeral models in any way. I just put the loop into a if-clause, but I'm sure you'll find a good solution being aware of the full application structure.
Best,
Marvin
Is your feature request related to a problem? Please describe.
Describe the solution you'd like
TODO
Describe alternatives you've considered
N/A
Additional context
Give thumbs if you think it's super cool to have asap ๐
Is your feature request related to a problem? Please describe.
It seems not to be possible to select models by using one or multiple tag.
Describe the solution you'd like
Adding the functionality to run dbterd
over tags specified in models metadata.
models:
- name: test
config:
materialized: incremental
tags: test, test2
columns:
- name: test_key
tests:
- relationships_test:
to: ref('test2')
field: test_key
dbterd run -ad target -s tag:test
or a list of tags
dbterd run -ad target -s tag:test,test2
New option --dbt-auto-artifacts
will work together with --dbt
option in order to trigger the dbt artifact files generated for a dbt project.
The command will look like:
BEFORE:
cd /path/to/dbt/project
dbt docs generate
dbterd run -s +my_model --dbt
AFTER:
cd /path/to/dbt/project
dbterd run -s +my_model --dbt --dbt-auto-artifacts
Hi, want to start off by saying this is a very neat tool, trying to implement this into our workflow at work, but had one quick question.
Is there are particular reason/restricting factor that does not allow for you to pull in the column descriptions, and add them to the .dbml output, so that they show up on the db docs page?
Describe the bug
Say a relationship is defined in schema.yml on model X referencing model Y. However, sometimes in the output.dbml, the relationship is defined as Y => X, rather than X => Y. Furthermore, if the foreign_key column name on X is a different name from the primary_key column name on Y, then those are swapped as well. This also causes "ghost columns" to appear in the involved tables. For example, if the foreign_key column on model X is 'F', and the primary_key on model Y is 'P', then the relationship will sometimes be defined as: X[P] => Y[F]. This creates a ghost column 'F' on table Y, and a ghost column 'P' on table X.
To Reproduce
Steps to reproduce the behavior:
This behavior is random. See "additional context" for discovery
Expected behavior
The relationship in output.dbml reflects the correct column names and direction of the relation.
Desktop (please complete the following information):
Additional context
I dug into the dbterd source code a bit and believe the problem is related to the line of code table_map=manifest.parent_map[x]
in the test_relationship.py
. It appears that this code expects the first element in the pair of the parent_map to be the foreign model, and the second element in the pair is the "this" model. However, looking in my manifest.json, it looks like DBT makes no such guarantees; as far as its concerned it is an unordered pair, and the order is likely some result of the order of each model in the DAG that it was compiled.
So, I think there needs to be some small logic that determines if the pair at manifest.parent_map[x]
needs to be swapped before it is assigned to table_map
.
All selection rules are lowercase-d before comparison, it should preserve the input value instead.
Fix of #73 has ignored the capitalized node_name in the Exact method.
This is to come up with a wider enhancement to get it aligned to other comparison methods as well, and with case sensitive supported
Is your feature request related to a problem? Please describe.
The table of interest in the ER diagram we want to draw has a very large number of columns. The number of columns can be dozens, making it very difficult to see the relationships between tables in an ER diagram.
Describe the solution you'd like
Add an option to omit columns in the ER diagram to allow output of a diagram without columns. For example, mermaid can only draw relationships between tables.
Describe alternatives you've considered
N/A
Additional context
Add any other context or screenshots about the feature request here.
Is your feature request related to a problem? Please describe.
For our usecase, we want to output 2 different ERD diagrams for 2 different sets of data models. These 2 usecases are however located in the same repository. Therefore, we want to setup a flow in which we execute 2 dbterd commands to generate 2 different DBML outputs. However, the name of the DBML output file is always output.dbml. (see: https://github.com/datnguye/dbterd/blob/a03af25c9efe9fe4f32208e4b1a2acd93a51df7a/dbterd/adapters/targets/dbml/dbml_test_relationship.py#L18C14-L18C20). This means we have to create two different directories in which to save the output files to. Also, the name 'output' is kind of generic. Finally, you can't create two seperate output files in the same directory because of this.
Describe the solution you'd like
Make the output file name dynamic and add an extra CLI argument to set a custom output file name.
Describe alternatives you've considered
Is your feature request related to a problem? Please describe.
Add new output as GraphViz
Additional context
Sample code:
digraph g {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
graph [fontsize=30 labelloc="t" label="" splines=true overlap=false rankdir = "LR"];
ratio = auto;
"state0" [ style = "filled, bold" penwidth = 1 fillcolor = "white" fontname = "Courier New" shape = "Mrecord" label =<
<table border="0" cellborder="0" cellpadding="3" bgcolor="white">
<tr><td bgcolor="black" align="center" colspan="2"><font color="white">State 0</font></td></tr>
<tr><td align="left" port="r0">(type) column1 </td></tr>
<tr><td align="left" port="r1">(type) column2 </td></tr>
<tr><td align="left" port="r2">(type) column3 </td></tr>
</table>> ];
"state1" [ style = "filled, bold" penwidth = 1 fillcolor = "white" fontname = "Courier New" shape = "Mrecord" label =<
<table border="0" cellborder="0" cellpadding="3" bgcolor="white">
<tr><td bgcolor="black" align="center" colspan="2"><font color="white">State 1</font></td></tr>
<tr><td align="left" port="r0">(type) column1 </td></tr>
<tr><td align="left" port="r1">(type) column2 </td></tr>
<tr><td align="left" port="r2">(type) column3 </td></tr>
</table>> ];
"state5" [ style = "filled" penwidth = 1 fillcolor = "white" fontname = "Courier New" shape = "Mrecord" label =<
<table border="0" cellborder="0" cellpadding="3" bgcolor="white">
<tr><td bgcolor="black" align="center" colspan="2"><font color="white">State 5</font></td></tr>
<tr><td align="left" port="r0">(type) column</td></tr>
</table>> ];
"state0" -> "state5" [ penwidth = 1 fontsize = 12 fontcolor = "black" label = "column1 = column" ];
"state0" -> "state5" [ penwidth = 1 fontsize = 12 fontcolor = "black" label = "column1 = column" ];
"state1" -> "state5" [ penwidth = 1 fontsize = 12 fontcolor = "black" label = "column1 = column" ];
}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.