I would like to customise the name of the y axis, x axis, and series on the chart.

See design on #42

I'd like to be able to change the minimum and maximum value of the ticks on the chart axes.

User acceptance criteria:

• By default, the range on the chart is: [min value, max value]
• if you toggle off auto-range, you get the min + max values that pop up (pre-filled with the auto-range values)
• I can set the Min + Max values for my y-axes and my x-axes
• I can only set the Min + Max values when I have numeric types on my axes (decimal, integer, float)
• I can set only my max or only my min value. If I only set the min value, then the max stays at the default max value, if I only set my max, then the min stays at the default min value.
• If a value has been formatted to a %, then I would need to set min = 0.2 to get a min value at 20%. Or, if it's easier to have a min value = 20 to limit the % to 20%, then let's do it that way! Basically, don't feel like we have to add any more complexity to setting min + max values for percentages. Just use whichever of the two is easiest.

Other tasks:

• add documentation for customizing min/max values
• add tracking for customizing min/max values
  • added to lightdash_server_saved_chart_created + lightdash_server_saved_chart_updated (custom_axis_range as TRUE/FALSE)
  • created new event: lightdash_webapp_custom_axis_range_toggle_clicked

Internal design ticket: https://github.com/lightdash/lightdash-internal-work/issues/299

As suggested by the amazing team at react query: TanStack/query#2363

Currently

The URL encodes the currently selected dimensions, metrics, filters, etc. But that is sometimes counterintuitive because you're more likely to want to share the config for the last set of successful results, rather than an in-flight config that hasn't been run yet.

Better

Changing dimensions, metrics, etc. doesn't change the URL until you hit "run query"

For now we'll document how to use dbt2looker with lightdash

If there are too many tables, there is no scroll feature in order to access the tables below the fold on the left hand side

We should add prettier + eslint + husky ( or another pre-commit hook ) to lint and format the code being committed.
This keeps the code consistent, catches potential bugs, prevents merge conflicts and makes it easier to review PR's.

e.g in slack they currently look like this

A quick win would be to use meta tags. For slack we could have something more custom to improve unfurling.

One potential challenge with this (and with share urls in general) is if the dimensions/metrics/filters contain sensitive information.

What is "more orders"? And why can't "last ordered at" be plotted on a graph?

Let's add rudderstack analytics to the app.

Why?

• So we can track people visiting the demo app https://demo.lightdash.com
• Teams can add their own rudderstack API key and track their usage on their own deployment
• Finally, we will have an opt-in for users to contribute their usage to our analytics. Perhaps we can make this a kind of contribution to the project?

What do you think?

On server start, dbt runs an initial compile but we wait for the first request before compiling the metrics.

It would be better to compile on server start, so production deployments don't need to refresh. This would reduce cold starts on zero downtime deployments too

If a dimension is declared with custom sql: then it can reference other dimensions. These references may not be valid and currently that throws an error at query time.

We can catch these earlier during compilation of all metrics.

We don't have docs showing users how to deploy to a production environment.

Create an example repo and some docs pages

Get error message "unexpected keyword LEFT" when running a query in the UI

I have a model specifying two joins with the jaffle_shop demo. When I remove either of the two joins it works.

When tables are joined together in lightdash, you should be able to see the SQL logic used to join these tables.

https://sketch.cloud/s/d6f235ac-00e3-4414-88cc-c70f88b75224

When a user select a table, they can click on a dependency graph icon and see the upstream + downstream tables.

(jamboard link: https://jamboard.google.com/d/1ecmP5iYwG5dKS5TL005-XVg5KAB9t8rYnHt_u6HzP18/viewer?f=0)

If I change the sort in my query and rerun, my data viz disappears.

Order of columns in the results are fixed sometimes I want to reorder the columns (like put date first not second)

My DBT project accesses the database from localhost. The database itself is remote but the connection is tunnelled to localhost:5433 as it can only be accessed using a VPN from a particular host. Anyway, all this to say that if your ~/.dbt/profiles.yml host key has localhost as a value, then you get an error like the below after it tries and fails to access the database for some time:

DbtError: Error returned from dbt while executing method 'poll': Database Error
  could not connect to server: Connection refused
  	Is the server running on host "localhost" (127.0.0.1) and accepting
  	TCP/IP connections on port 5433?
  could not connect to server: Cannot assign requested address
  	Is the server running on host "localhost" (::1) and accepting
  	TCP/IP connections on port 5433?

I use a mac and I can bypass this by changing localhost to host.docker.internal if running Docker 18.03 or later but it would be nice if there was a better solution that didn't require me changing this. Also not sure what non-mac users need to do in order to access localhost from within the container. This should be mentioned in your docs as it seems you are assuming everyone's database is remote. My particular case is probably niche but many may be using a local database for dev purposes

When you have a bunch of dimensions + measures in your table, you should be able to remove/add them to the visualization (in case you don't want all of them)

Two ways of doing this:

1. we put an 👁️ icon beside the names of the columns. If you click on the eye, it would add a / over the eye and remove it from the visualization. You can then click on the crossed-out eye to make it re-appear in the visualization.

2. We can add a gear symbol beside the name of the column. This could become the settings space for columns (e.g. you could say add filter or sort, etc. within this space eventually). If you click on the gear, there would be a drop-down menu that would let you select hide from chart (or show in chart)

We should have support for dbt cloud.

There are two components to how Lightdash relies on a dbt project:

1. The manifest.json and catalog.json - these are available from dbt cloud and have all the info we need
2. The raw project files. We plan to use this when the user is locally developing to detect changes on disk, and to map Lightdash syntax errors back to source files.

We should be able to support dbt cloud for production use cases, but will be harder for us to provide the "development" type features.

Also we currently don't have a nice way in the UI to distinguish between using "files" as the backend vs "dbt cloud". we might also add "GitHub" here. These are basically different source locations, each with their own features that they support in the UI. Could spiral quickly.

A quick hack would be to take dbt cloud credentials as an environment variable.

When plotting multiple series, you should be able to assign them to separate axes.

Blocked by #1565

What does this look like?

Update the series tab in the chart configuration to give the option to map the series to the left or right axis (for horizontal charts this should be the top or bottom axis)

The axis label tab will need an additional entry too.

it will be really nice specially in complex models to see relationship between tables

thanks, this is a very exciting project.

When joined tables aren't 1:1, metrics can go wrong for users in the frontend (e.g. https://looker.com/blog/aggregate-functions-gone-bad-and-the-joins-who-made-them-that-way)

One solution we've discussed is only allowing star schema type joins. I think this is an interesting solution, which sticks with the principal of moving work to dbt where possible!

Installation guide link:

Demos are fun and all, but to get the full Lightdash experience, we recommend working through our installation guide and hooking it up to your own project 😊

Returns a 404

When the server fails due to dbt config problems, it's not obvious how to fix it.

Add a page to documentation to debug common dbt config problems

lightdash uses backticks ``` to quote field names but this isn't valid in postgres. It should be double quotes "

Summary:
When using BigQuery service-account method [0] the app throws an error because it cannot find the credentials file.

Reproduction steps:
Precondition: dbt project with a profile using BigQuery service-account method
Run lightdash (via docker)

# Go to your project
cd /path/to/my/dbt/project

export DBT_PROJECT_DIR=${PWD}
export DBT_PROFILES_DIR=${HOME}/.dbt
export LIGHTDASH_PORT=8080

docker run -p "${LIGHTDASH_PORT}:8080" -v "${DBT_PROJECT_DIR}:/usr/app/dbt" -v "${DBT_PROFILES_DIR}:/usr/app/profiles" lightdash/lightdash

[0] https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile#service-account-file

If using a bigquery profile and method: oauth (relying on gcloud sdk authentication) and lightdash is run without having a ~/.config/gcloud directory available, then dbt is polled infinitely and eventually errors on timeout

A request was made for cloud run but I don't think Lightdash is well built for server less yet.

Should test this

users can click to see the "source" of a field/metric. When they click on it in lightdash, it opens the dbt model .yml file in a text editor.

https://sketch.cloud/s/785822f3-caf0-41bf-901b-388f2ff3d6ea

It would be great if I could export a chart as a .png etc for external presentation.

This is built into ECharts

Currently, metrics are limited to using one of the (predefined types we have available)[https://docsite.onrender.com/references/metrics#metric-types].

For us to be able to do more complex aggregations (and not have a bajillion pre-defined metric types), users should be able to write custom SQL aggregations to define metrics.

E.g. you'd be able to build the metric for 7d rolling average number of weekly active users

docker compose run lightdash - dbt debug

Docker Desktop on Windows 10.
dbt project and profile.yml working.

Running
docker run -p "8080" -v "C:\projects\myproject" -v "C:\Users\myuser.dbt" lightdash/lightdash

(after downloading docker images) gives the result:

_yarn run v1.22.5
$ node dist/index.js
| | | | | | |
| | | | | | |
| | | | | | |
\ | / \ | / \ | / \ | / \ | / \ | / \ | /
|/ |/ |/ |/ |/ |/ |/

Launch lightdash at http://localhost:8080

/|\ /|\ /|\ /|\ /|\ /|\ /|
/ | \ / | \ / | \ / | \ / | \ / | \ / |
| | | | | | |
| | | | | | |
| | | | | | |
(node:28) UnhandledPromiseRejectionWarning: Error: spawn dbt ENOENT
at Process.ChildProcess.handle.onexit (internal/child_process.js:269:19)
at onErrorNT (internal/child_process.js:467:16)
at processTicksAndRejections (internal/process/task_queues.js:82:21)
(Use node --trace-warnings ... to show where the warning was created)
(node:28) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:28) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

and browsing to localhost:8080 says "This site can’t be reached"

Are there issues with running this on Windows?

When the main content overflows 100vh, the UI becomes scrollable but the sidebar stays at a fixed height.

I'm not sure whether we want the entire UI rendered within 100vh or to make it infinitely scrollable. Either way, the current implementation is inconsistent.

When dbt’s running but it’s failed, it’s not obvious what the problem is. We need to improve these error messages.

The current error message looks something like this:

GET /api/v1/tables 500 283.331 ms - 634
(node:29) UnhandledPromiseRejectionWarning: DbtError: Error returned from dbt while executing method 'status'
    at /usr/app/packages/backend/dist/dbt.js:124:15
    at Generator.next (<anonymous>)
    at fulfilled (/usr/app/packages/backend/dist/dbt.js:5:58)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)

Instead of just having this DbtError: Error returned from dbt while executing method 'status', we should have a bit that says ...and this is why it failed after it.

If you have more than one dimension + one measure in your table, you should be able to pick which things you want to have as the x, y and series.

Currently this happens on the backend:

Error: spawn dbt ENOENT
    at Process.ChildProcess._handle.onexit (node:internal/child_process:282:19)
    at onErrorNT (node:internal/child_process:480:16)
    at processTicksAndRejections (node:internal/process/task_queues:81:21)

Rudderstack is default configured in the docs minisite - but this should only be set on deployment.

This will allow:

• no default rudderstack for people running the docs site themselves
• users to configure their own rudderstack endpoint if they wanted

Right now, we can only build metrics from dimensions. This limits the types of metrics that you can build/the things you can calculate in lightdash.

• We should be able to define a metric with one of the existing types using other metrics and dimensions.
• These other metrics come from the current model, or any metrics in a joined model

I'd imagine it would look something like this in the .yml file:

    columns:
      - name: user_id
        description: 'Unique identifier for a user.'
        tests:
          - unique
          - not_null
        meta:
          metrics:
            num_unique_users:
              type: count_distinct
            num_unique_7d_web_active_users:
              type: count_distinct
              sql: "IF(${is_7d_web_active}, ${user_id}, NULL)"
            num_unique_non_7d_web_active_users:
              type: sum
              sql: "${num_unique_users} - ${num_unique_7d_web_active_users}" # both are metrics coming from the base model
     - name: is_7d_web_active
        meta:
          metrics:
            avg_spend_per_7d_web_active_user:
              type: mean
              sql: "IF(${is_7d_web_active}, ${transactions.total_spend}, NULL" # this is an example where the transactions.total_spend is a metric that is coming from a joined model: transactions. 

when following "Installation from source" section in the readme, there is no info about the DBT_PROFILES_DIR env variable, which is required to setup.

currently, if a user is using BigQuery and has their dbt profile set up with oauth via gcloud, lookdash will fail.

The work around is quite complex for this (and requires users to have a service account/know where to find it).

It would be great if lookdash "just works" for users with oauth via gcloud.

Motivation

We use Google Cloud, BigQuery and dbt. We are looking for a better dbt-friendly BI tool. I love the idea of lightdash. So, if we want to practically use lightdash as a production-grade system, authentication and permission control is necessary. It would be great to support SSO/SAML authentication and fine-grain ACL.

It's quite annoying to select filters, especially if there are a lot of dimensions available.

I would prefer to be able to just click the name of the dimension in the sidebar or table. It would be even cooler if I could click a value in the table to filter on that value!!

Users can choose to see the full tree or just the direct links in the tree of the dependency graph

https://sketch.cloud/s/62ffa882-a034-429e-9bba-312d59130ce7

Add support for sqlserver users.

This may be causing errors with deploying e.g. #37

adapter: https://github.com/dbt-msft/dbt-sqlserver

Todo
[] - Docker image should contain the SQL Server adapter: #37
[] - Sql Builder must be customised to support SQL Server syntax

Multisort behaviour was removed in version 0.2.0 - bring it back!

When a chart can't be plotted or has no data, the empty state doesn't render anything and the configure panel is quite confusing: