gouline / dbt-metabase Goto Github PK

It would be useful to automatically hide/show models in Metabase. The metabase API endpoint PUT /api/table/:id accepts a visibility_type value that may be nil, or if non-nil, value must be one of: cruft, hidden, technical. Can we allow the

  meta:
    metabase.visibility_type: hidden

to be defined on models?

While running dbt-metabase config it suggested -1 as a timeout value

But after running dbt-metabase exposures it failed with

File "/home/psegatori/.local/lib/python3.8/site-packages/dbtmetabase/metabase.py", line 121, in sync_and_wait
    raise exceptions.MetabaseUnableToSync(
dbtmetabase.models.exceptions.MetabaseUnableToSync: Timeout provided -1 secs, must be at least 5

Not sure if this is the right place for this but I have built functionality capable of synchronizing dbt tests to a designated collection in Metabase. The flip side of the process uses Metabase API to query each question in the designated collection flagging failures and collecting results to in turn be displayed on Metabase in a unit test dashboard.

Example collection with synchronized unit tests / schema tests / great expectations from dbt Jinja compiled tests.

Example Dashboard from API collected results:

Table links to Metabase questions where I can view the failing records:

While I dont know if this falls into the scope of this project, I wanted to ask from the perspective of a deeper integration what your thoughts were. I have these processes running via Jenkins in a CI/CD env in production with good success. Even without collection synchronization to quickly view results, I havent seen much dbt test query parsing and tabular result generation. Maybe a mention of it from someone else who built an application wrapper.

DBT recently added meta definitions for models and columns.

It'd be nice to be able to use these, rather than tests, to specify metabase column types.

Using meta would remove any impact on test runtime when running DBT, and remove the need for the DBT package specified in this project.

We've just started using dbt-metabase exposures and I've noticed that the output YAML replaces all spaces in the Metabase card names with underscores (e.g. a card with name "Votes by country" becomes "Votes_by_country"). This makes it a bit difficult to read when published in the docs since we wouldn't have expected that title/name to be edited in the process. Is there an option to avoid this?

Hi, i'm trying to use this awesome project with my dbt setup (bigquery) but keep getting this AttributeError: 'NoneType' object has no attribute 'upper'.

I'm running on dbt v0.20.0 and Metabase 0.40.1

Traceback (most recent call last):
  File "/home/edbizarro/.pyenv/versions/analytics-venv-3.7.8/bin/dbt-metabase", line 6, in <module>
    dbtmetabase.main(sys.argv[1:])
  File "/home/edbizarro/.pyenv/versions/3.7.8/envs/analytics-venv-3.7.8/lib/python3.7/site-packages/dbtmetabase/__init__.py", line 248, in main
    include_tags=parsed.include_tags,
  File "/home/edbizarro/.pyenv/versions/3.7.8/envs/analytics-venv-3.7.8/lib/python3.7/site-packages/dbtmetabase/__init__.py", line 114, in export
    mbc.export_models(metabase_database, models, reader.catch_aliases)
  File "/home/edbizarro/.pyenv/versions/3.7.8/envs/analytics-venv-3.7.8/lib/python3.7/site-packages/dbtmetabase/metabase.py", line 156, in export_models
    table_lookup, field_lookup = self.build_metadata_lookups(database_id)
  File "/home/edbizarro/.pyenv/versions/3.7.8/envs/analytics-venv-3.7.8/lib/python3.7/site-packages/dbtmetabase/metabase.py", line 377, in build_metadata_lookups
    table_schema = table.get("schema", "public").upper()
AttributeError: 'NoneType' object has no attribute 'upper'

Command I'm using to test

dbt-metabase export \
    --dbt_manifest_path ./analytics/target/manifest.json \
    --dbt_database analytics \
    --metabase_host ...\
    --metabase_user ... \
    --metabase_password "..." \
    --metabase_database "Analytics" \
    --schema public

I think its very valid for someone to want to parse exposures in CI/CD taking a manifest.json as an input, modifying it, and outputting an updated manifest.json. As of today, we output a yaml which is easily readable, can be committed to source control, etc. but might not be desired by a user who just wishes to see exposures in docs without the additional yml. Lets add a cli flag to make the execution optionally manipulate the artifact instead of generating yml to be consumed by subsequent dbt command.

This paves way for my next PR which involves making the data catalogue explorable through metabase table explorer. Stay tuned for these. Will updates this issue as I go if needed.

An Issue to keep track of what should be updated in the docs before releasing 0.8.0:

• Use of export models and export exposures in the CLI commands

Anything else you are aware?
I can open a PR if needed. I just wanted to do many things in one-go 😄

This logic does not account for self-referencing tables:

Moving the discussion to an issue to avoid polluting the discussion on the PR.

@z3z1ma wrote on #19 that he has a follow-up PR in mind on adding a feature to automatically create dbt exposures based on Metabase questions, making it possible to see on the dbt lineage graph which Metabase questions are using which dbt models. More details on this message as well: #19 (review)

Just today I open-sourced an internal project that does the same, but between dbt and Tableau: https://github.com/voi-oss/dbt-exposures-crawler.

One thing I would like to ask is what's @gouline scope for dbt-metabase. Do you want to keep it narrow and focus on documentation syncing, or do you want to be a comprehensive Metabase <> dbt toolkit and think that @z3z1ma feature would fit on dbt-metabase as well?

@z3z1ma, how much of the existing dbt-metabase codebase you actually use in the exposures feature? Mostly the Metabase API client I guess?

Lets consider our usual flow of

dbt compile -m my_model
navigate to compiled model
copy paste sql
find somewhere to run it to test it
OR
dbt run -m my_model
check in datagrip, dbeaver, dbvisualizer, vscode sql ext, etc.
repeat x?...
which the latter implies potentially multiple CTAS, tables being dropped and rewritten, etc. or the inability to test in production vs compile and pasting somewhere you can run it.

I think we can easily offer an ability to invoke something like:

dbt-metabase explore my_model

And on our end conceptually, we use a subprocess to invoke dbt compile -m my_model
Read the compiled sql from the manifest.json or file.
Construct and pass it into a json object that looks something like {dataset_query: "SELECT * FROM ..."} with your compiled model query inputted
Base64 encode the json dumped string
And construct a URL like this as many Metabase aficionados may be familiar with (its how metabase caches your in progress queries in the editor).

https://my.metabase/question#eyJkYXRhc2V0X3F1ZXJ5Ijp7InR5cGUiO...c2V0dGluZ3MiOnt9fQ==

This will allow, with a simple invocation, using metabase as a front end graphical dbt model explorer without the above workflow being an impedance. I have a similar goal in my project dbt-osmosis which uses a dbt-osmosis workbench invocation with streamlit as a backend for rapidly developing/prototyping and diffing the data output of model changes. I think for Metabase power users, this is a simple and effective alternative too.

We empower users to explore models, all without actually ever asking them for db creds. Interesting right.
Opening this now to gauge feedback prior to implementation.

It appears to me that this logic fails for dbt==1.0.0.

It seems that relationship tests don't contain the related model on the depends_on iterable anymore. It does appear in the refs iterable both pre and post-1.0.0, however only as simple ref string ('my_model_name') instead of a full unique identified (model.[project_name].[model_name]).

Has this been tested yet? Is there a favored approach?

I'd be happy to give this a go on the weekend with a PR, but I don't know the repository well and am wondering if I am simply missing something :)

Seems like there was a commit to add metabase_session_id as a CLI option, but it was added as a parameter only for model and not for exposures. Adding the corresponding lines to exposures resolved the issue for me.

Description

I got the following error : Error: No such option: --schema Did you mean --dbt_schema? when running the README example

dbt-metabase models \
    --dbt_path . \
    --dbt_database business \
    --metabase_host metabase.example.com \
    --metabase_user [email protected] \
    --metabase_password Password123 \
    --metabase_database business \
    --schema public

Suggestion

It's seems that the readme is outdated, schema flag must be replaced by dbt_schema as follow :

dbt-metabase models \
    --dbt_path . \
    --dbt_database business \
    --metabase_host metabase.example.com \
    --metabase_user [email protected] \
    --metabase_password Password123 \
    --metabase_database business \
    --dbt_schema  public

Version

dbt-metabase==0.9.3

In 646a914, you fixed it so that if we haven't specified a metabase.semantic_type.

This is good, but now, how do we explicitly clear a semantic type set by the automagic rules that Metabase applies on schema sync?

Setting metabase.semantic_type: null seems to have no effect. Can we look for the presence of this key, or maybe use a special value?

I'd wager that the usability and prevalence of this marvelous packages would be greatly advanced if there was a basic documentation to go along with the two major models (MetabaseInterface and DbtInterface). For instance, it took me a few hours to realize that my implementation wasn't working because schema is, in fact, not a mandatory field but rather a field limiting the amount of models being parsed and synced.

I think this is a great tool that is easily usable if you know how to, where a bit of docs could go a long way :)

Hello,

I've got a KeyError: 'fk_target_table' when propagating data from dbt table to Metabase.
This error occurs in two specific cases:

1. A column in the the schema.yml has a semantic type equal to type/FK but it has not a tests section within it.
2. A column in the the schema.yml has a semantic type equal to type/FK but its tests section contains a dbt_utils.relationships_where test within it.

Docs say you can use:

    dbt_schema_excludes:
      - development
      - testing

And there's also support for a --dbt_schema_excludes command-line option.

However, you must actually upcase the schema name, unlike the example before, because the code that actually looks at this is:

if schema_excludes and node["schema"].upper() in schema_excludes:

I'm not really clear why the upper() is there. If it's just for the sake of normalising, then we also need to normalise the arguments?

In my personal opinion, if semantic_type is not configured in the dbt schema, the value currently set in Metabase should be retained.

I was surprised when I ran dbt-metabase (in order to upload column descriptions) and found that Metabase's initial guesses at the semantic types (which were mostly correct), plus a few that I'd done manually, had all been reset.

To avoid this, IMHO the semantic type should only be set if metabase.semantic_type is explicitly specified. Changing this behavior would have the downside that if you had previously set metabase.semantic_type and then removed it, the type would not be un-set. But in my opinion that is not surprising behavior.

@gouline do you agree? I won't try and have a go this one if you wouldn't accept it.

I suggest to use /api/database/{id}/metadata instead of /api/database/{id}/fields.

Currently, the code is leveraging the /fields API to retrieve field metadata. However, the response I am getting is like:

{ "id": 39, "name": "Companies House No", "base_type": "type/Text", "special_type": null, "table_name": "Dim Account", "schema": null }

with no stable identifier for the field name (name field is basically the display name in Metabase).

I suggest to use the /metadata API instead, which will return a nested json of tables and fields, where a field will look like:

{ "description": null, "database_type": "STRING", "special_type": "type/Company", "display_name": "Company Number", "base_type": "type/Text", }

(I have summarised the response, which is a lot larger).
I'd be happy to implement this change.

It appears that the code to pull exposures from native queries is unable to recognize model names when they're qualified.

I'd expect the below metabase query to generate an exposure dependent on my_model, but it doesn't.

select * from db.schema.my_model

I think the bug is that the regex used to find from and join clauses doesn't allow for . or " characters, despite the subsequent code appearing to account for those cases.

I'd propose that the regex should instead allow . and " characters.

r"[FfJj][RrOo][OoIi][MmNn]\s+\b([\w.\"]+)\b"

The export_model function supports setting the display name for models from a display_name property on MetabaseModel.

However, this isn't accessible, as the meta: metabase.display_name property is not parsed and populated in the construction of the MetabaseModel in _read_model.

Hello
There seems to be a size limit somewhere
dbt-metabase used to work but as the model grew it suddently stopped working
I get

Traceback (most recent call last):
  File "/usr/local/bin/dbt-metabase", line 5, in <module>
    dbtmetabase.main()
  File "/usr/local/lib/python3.8/dist-packages/dbtmetabase/__init__.py", line 720, in main
    cli(max_content_width=600)  # pylint: disable=unexpected-keyword-arg
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/dist-packages/dbtmetabase/__init__.py", line 113, in invoke
    return super().invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/dbtmetabase/__init__.py", line 262, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/dbtmetabase/__init__.py", line 568, in models
    dbt_models, aliases = dbt.read_models(
  File "/usr/local/lib/python3.8/dist-packages/dbtmetabase/models/interface.py", line 134, in read_models
    return self.parser.read_models(self, include_tags, docs_url)
  File "/usr/local/lib/python3.8/dist-packages/dbtmetabase/parsers/dbt_manifest.py", line 112, in read_models
    self._read_model(
  File "/usr/local/lib/python3.8/dist-packages/dbtmetabase/parsers/dbt_manifest.py", line 213, in _read_model
    depends_on_id = list(
IndexError: list index out of range

To make code exploration and review easier, would it be possible to replicate metabase file and folder hierarchy when generating exposures ? Like reusing the slug

/metabase-exposures/question/42-my-awesome-question.yml
/metabase-exposures/dashboard/84-my-super-dashboard

Or maybe just the ID if to avoid too many file changes when just renaming something ?

/metabase-exposures/cards/1.yml
/metabase-exposures/cards/2.yml
/metabase-exposures/dashboards/1.yml

Currently all exposures are sent to a single file and it's messy

When I run the following command, the session is established but then the process ends without any message:

dbt-metabase models \
    --dbt_manifest_path ./target/manifest.json \
    --dbt_schema dbt_prod \
    --metabase_host $url \
    --metabase_user $metabase_email \
    --metabase_password $metabase_pw \
    --metabase_database $dbname \
    --dbt_database dbt_prod

2021-09-14 09:42:07,987 - INFO - Session established successfully

Am I doing something wrong?

Metabase Metrics (and in particular custom aggregations) are very useful to our end-users. I'm finding myself spending a lot of time in Metabase managing Metrics, the UI is very inconvenient.

I'd like to be able to define a list of Metabase Metrics on DBT models as schema model meta properties. I wonder if this fits this project's scope. The primary challenge is parsing the Metabase expressions which are defined as string (ex: Distinct(case([step] = "hired", [candidate_user_id])) into trees that the API can handle. I think it's easiest to maintain if I compile the JS parser from metabase source code into a small node script that can be committed and called from this project.

This would add a good amount of complexity so it's probably beyond the scope of the project. I thought I would share for the sake of discussion.

I'm using the dbt alias feature in many of my models, which means that my model name doesn't match the respective object in my database and dbt-metabase won't be able to correctly match them when retrieving the schema from Metabase.

This is because dbt-metabase retrieves the model information/name directly from the YAML files written by the dbt users instead of using the dbt compiled artifacts which are now much more stable: https://docs.getdbt.com/reference/artifacts/dbt-artifacts. I also use multiple schemas in my project and the schemas can be defined in multiple places (config block in the model, dbt_project.yml, etc). The compiled manifest.json is the final source of truth of where my model will be materialized (schema, alias, etc).

I propose to expand the CLI and Python interfaces to, besides supporting the existing dbt_path (for backward compatibility), also support a dbt_manifest_path to a manifest.json file (mutually exclusive arguments).

I'm working on a PR for that, but I'm creating this issue here to gather early feedback and see what you think.

PS: Thanks a lot for this project! It will be very useful in my organization.

Not clear what the env var should be called. Readme and first place in the code says MB_PASS:

ENV_VARS = [
    "DBT_DATABASE",
    "DBT_PATH",
    "DBT_MANIFEST_PATH",
    "MB_USER",
    "MB_PASS",

But the actual env var handling code says:

    "DBT_PATH",
    "DBT_MANIFEST_PATH",
    "MB_USER",
    "MB_PASS",

    @click.option(
        "--metabase_password",
        metavar="PASS",
        envvar="MB_PASSWORD",

Is the latter wrong?

Hello,

I've tried to follow the usage and installation instructions but got the following error

File "/home/xxxx/.local/lib/python3.8/site-packages/dbtmetabase/__init__.py", line 99, in process_value
    and ctx.get_parameter_source(self.name)
AttributeError: 'Context' object has no attribute 'get_parameter_source'

I upgraded my local click package and it worked.

It upgraded from Click 7.0 to click 8.1.3

I'm a PHP developer with no experience in Python so my suggestion might be off but could the error be avoided if the project declared that it can only work with a specific version range of click package in the requirements.txt ?

Adding this per suggestion from @z3z1ma from the dbt slack discussion 😄

This feature would be helpful for anyone with a large Metabase deployment - where adding exposures might add a lot of noise to the dbt project.

We could add functionality to assign maturity or tags based on total use, recent use, or some combination of those two. Tagging may be better as it's a bit more granular -- you could have things like tag:broken-exposure or tag:no-views-last-month and use that to find and clean up unused or lesser used exposures.

Hello,

when a user is deactivated, metabase returns 404 for any API requests regarding this user, and this causes an exception during exposures sync:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/dbtmetabase/metabase.py", line 847, in api
    response.raise_for_status()
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://metabase-internal.gatserver.com/api/user/15
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/bin/dbt-metabase", line 5, in <module>
    dbtmetabase.main()
  File "/usr/local/lib/python3.8/site-packages/dbtmetabase/__init__.py", line 720, in main
    cli(max_content_width=600)  # pylint: disable=unexpected-keyword-arg
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/dbtmetabase/__init__.py", line 113, in invoke
    return super().invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/dbtmetabase/__init__.py", line 262, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/dbtmetabase/__init__.py", line 709, in exposures
    metabase.client.extract_exposures(
  File "/usr/local/lib/python3.8/site-packages/dbtmetabase/metabase.py", line 583, in extract_exposures
    creator = self.api("get", f"/api/user/{exposure['creator_id']}")
  File "/usr/local/lib/python3.8/site-packages/dbtmetabase/metabase.py", line 849, in api
    if "password" in kwargs["json"]:

There's missing error handler at:

                   creator = self.api("get", f"/api/user/{exposure['creator_id']}")
                   creator_email = creator["email"]
                   creator_name = creator["common_name"]

I'm considering submitting a PR where 404 errors would be handled and if is happening, both creator email and name would be empty. I wanted to confirm if that would be an acceptable solution in this case?

Hi!

The docs state that this amazing library "Requires Python 3.6 or above", and also setup.py states the same.

However, some recently introduced changes (https://github.com/gouline/dbt-metabase/blob/master/dbtmetabase/models/metabase.py#L28) broke that. Using typing.Literal requires Python 3.8+. Running the library with Python 3.7 for example will raise:

ImportError: cannot import name 'Literal' from 'typing' (/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/typing.py)

Alternatives on what to do:

• Remove this Literal type hint and instead use Enum
• Install an extra dependency that will make this type hint work in 3.6+ (see info box on https://mypy.readthedocs.io/en/stable/literal_types.html)
• Bump the minimum required version to 3.8. Maybe a bit overkill just because of a type hint. dbt itself doesn't run well on 3.9, so requiring a minimum of 3.8 leaves a very tight compatibility range.

Orthogonal to that, we could:

• Explicitly set a Python version (3.6) in the GitHub Actions workflow that runs CI tests using:

- uses: actions/setup-python@v2
  with:
    python-version: '3.6.x' 

• or, Test multiple Python versions with GitHub Actions matrix https://github.com/actions/setup-python#usage
• or, Test multiple Python versions with tox

I'm happy to contribute with a PR once the maintainer and contributors align on how to move forward 😄 .

Support for timeline events would be cool (API).

On Metabase, it's possible to define permissions restrictions to tables for certain groups (Such as in the picture)

In the Open Source plan for Metabase, we can define if this group has either "Unrestricted" or "No self-Service" permission to the table, I am not sure if this changes for the other plans.

Is the automation for the permission system, something planned/wanted for this package?

One suggestion that could make it possible is to add this in the meta section of the table definition, and define which groups have which level of permission:

dbt model names are case insensitive, and so the user should be able to enter the names of included models in lowercase, which is often how models are typed.

Currently, because model_name in dbt_manifest.py is uppercased, model names that are sent to the includes variable as lowercase strings are ignored.

A simple uppercasing of the includes list solves this.

I'll open a PR.

Note: I think a similar issue may exist with dbt_excludes, but I don't need to use that at the moment, and when I tried to include it as a quick fix, it didn't work expected and I don't have time to dig into it, so I'm leaving it for now.

Could you please add an option to connect to metabase API with HTTP ? Currently HTTPS is required.

Hi!
I have am using bigquery and have trouble synchronizing metabase when I try to synchronize with dbt-metabase export.

The console prints a bunch of

...
2021-09-13 17:26:21,324 - WARNING - Model API_DATASET not found
2021-09-13 17:26:21,324 - WARNING - Model API_DATASETITEM not found
2021-09-13 17:26:21,324 - CRITICAL - Sync timeout reached, models still not compatible

None of my models are upper case - could that be the reason why the models are not found?

When running the dbt-metabase exposures command it fails when it hits an item referencing a deactivated user. In particular, this line returns 404, causing the whole script to crash:

Possible solutions:

1. try-catch this block and fill-in the details with a generic anonymous user
2. use instead the /api/user?include_deactivated=true endpoint to list all users once and search by ID. This endpoint has the advantage of also retrieving deactivated user's details, whilst the /api/user/:id one returns 404.

The docs mention models and exposures commands, but the CLI appears to only have the export command as of version 0.7.1.

For example, running dbt-metabase exposures returns the following:

usage: dbt-metabase [-h] --dbt_path PATH --mb_host HOST --mb_user USER --mb_password PASS [--mb_https HTTPS] --database DB --schema SCHEMA [--sync ENABLE] [--sync_timeout SECS] [--includes [MODELS [MODELS ...]]]
                    [--excludes [MODELS [MODELS ...]]]
                    {export}
dbt-metabase: error: argument command: invalid choice: 'exposures' (choose from 'export')

Steps to reproduce:

1. pip install pipenv (creates a new env, ignore if using venv)
2. pipenv install dbt-metabase
3. pipenv run dbt-metabase exposures

Version currently installed:

dbt-metabase==0.7.1
  - PyYAML [required: Any, installed: 6.0]
  - requests [required: Any, installed: 2.26.0]
    - certifi [required: >=2017.4.17, installed: 2021.10.8]
    - charset-normalizer [required: ~=2.0.0, installed: 2.0.7]
    - idna [required: >=2.5,<4, installed: 3.3]
    - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.7]

Metabase 0.40 has breaking changes:
/api/database does not anymore return an array but now returns a dictionary.
/api/fields/{id}does not anymore return an array but now returns a dictionary.

Hey, thanks for sharing you work on this.

I'm having some problems using the library to parse my project. Neither the the manifest parser nor the DbtFolderReader is able to parse it without failing. For my project I'm using dbt 0.20.0, is this version supported by this package? The manifest schema has slightly changed the last couple of versions and I'm wondering if you might be working with another version.

We are using multiple schemas specified in dbt_project.yml. It would be great if dbt-metabase follows the schema settings and does not send everything to the schema from the command line.

models:
  my_project:
    marketing:
      schema: marketing
    product:
      schema: product
    events:
      schema: events

thanks @gouline :)

Would be nice to expose all Metabase's visuals and dashboards in the DBT documentation as exposures

Whether I try dbt-metabase programmatically or with CLI, each time it seems that it cannot find the appropriate database to reference. I could put any string there or even leave it as a blank string and the same issue arises; nothing is being read.

Is this the correct way to load these variables? The DbtInterface seems to be the only portion not funcitoning properly, but I'm unsure how exactly to resolve the issue of it not reading things properly

dbt = DbtInterface(
    path='/home/bradmcglynn/vm_dbt_package/rvezy-dbt-1/models/reports/dimensions',
    manifest_path='/home/bradmcglynn/vm_dbt_package/rvezy-dbt-1/target/manifest.json',
    database='rvezy-1379',
    schema='dimensions.yml',
    schema_excludes=None,
    #includes=dbt_includes,
    excludes=None,
)

See discussion on PR#29.

The current CLI arguments --mb_https and --sync can only be set to false from the command line by passing an empty string. This isn't intuitive.

Suggested changes:

• Remove mb_https and propagate mb_http downstream to make programmatic interface consistent with CLI interface
• Apply a similar change to --sync

The sync timeout should have a default value otherwise the behavior is not as described

Current behavior:

Sync is only attempted if metabase_sync_timeout is defined (default is None)

current docstring:

metabase_sync_timeout (Optional[int], optional): Synchronization timeout (in secs). If set, we will fail hard on synchronization failure; if not set, we will proceed after attempting sync regardless of success. Only valid if sync is enabled. Defaults to None.

BigQuery doesn't have the concept of schema. For this reason, I am getting a lot of errors on the upper() functions applied to NoneType (this happens when retrieving metadata for tables and fields).

I suggest to assume the schema is public when the API returns None.

When using a config.yml file and specifying metabase_use_http: True (tried upper and lower case), I end up with the following SSL error (note: running Metabase via http://localhost:3000):

requests.exceptions.SSLError: HTTPSConnectionPool(host='localhost', port=3000): Max retries exceeded with url: /api/session (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1122)')))

After digging into it, the issue was that it was still sending the request via https. I found that here the self.protocol attribute was still being passed as https resulting in the error.

response = requests.request(
            method, f"{self.protocol}://{self.host}{path}", verify=self.verify, **kwargs
        )

Using --metabase_http as a command line argument fixed the issue for me, I believe it just overrides what's here

@click.option(
    "--metabase_http/--metabase_https",
    "metabase_use_http",
    default=False,
    help="use HTTP or HTTPS to connect to Metabase. Default HTTPS",
)

Currently using dbt-metabase==0.8.5