Databricks Plugin for Steampipe

Use SQL to query clusters, jobs, users, and more from Databricks.

Get started →
Documentation: Table definitions & examples
Community: Join #steampipe on Slack →
Get involved: Issues

Quick start

Install

Download and install the latest Databricks plugin:

steampipe plugin install databricks

Configure your credentials and config file.

Configure your account details in ~/.steampipe/config/databricks.spc:

connection "databricks" {
  plugin = "databricks"

  # A connection profile specified within .databrickscfg to use instead of DEFAULT.
  # This can also be set via the `DATABRICKS_CONFIG_PROFILE` environment variable.
  # profile = "databricks-dev"

  # The target Databricks account ID.
  # This can also be set via the `DATABRICKS_ACCOUNT_ID` environment variable.
  # See Locate your account ID: https://docs.databricks.com/administration-guide/account-settings/index.html#account-id.
  # account_id = "abcdd0f81-9be0-4425-9e29-3a7d96782373"

  # The target Databricks account SCIM token.
  # See: https://docs.databricks.com/administration-guide/account-settings/index.html#generate-a-scim-token
  # This can also be set via the `DATABRICKS_TOKEN` environment variable.
  # account_token = "dsapi5c72c067b40df73ccb6be3b085d3ba"

  # The target Databricks account console URL, which is typically https://accounts.cloud.databricks.com.
  # This can also be set via the `DATABRICKS_HOST` environment variable.
  # account_host = "https://accounts.cloud.databricks.com/"

  # The target Databricks workspace Personal Access Token.
  # This can also be set via the `DATABRICKS_TOKEN` environment variable.
  # See: https://docs.databricks.com/dev-tools/auth.html#databricks-personal-access-tokens-for-users
  # workspace_token = "dapia865b9d1d41389ed883455032d090ee"

  # The target Databricks workspace URL.
  # See https://docs.databricks.com/workspace/workspace-details.html#workspace-url
  # This can also be set via the `DATABRICKS_HOST` environment variable.
  # workspace_host = "https://dbc-a1b2c3d4-e6f7.cloud.databricks.com"

  # The Databricks username part of basic authentication. Only possible when Host is *.cloud.databricks.com (AWS).
  # This can also be set via the `DATABRICKS_USERNAME` environment variable.
  # username = "[email protected]"

  # The Databricks password part of basic authentication. Only possible when Host is *.cloud.databricks.com (AWS).
  # This can also be set via the `DATABRICKS_PASSWORD` environment variable.
  # password = "password"

  # A non-default location of the Databricks CLI credentials file.
  # This can also be set via the `DATABRICKS_CONFIG_FILE` environment variable.
  # config_file_path = "/Users/username/.databrickscfg"
  
  # OAuth secret client ID of a service principal
  # This can also be set via the `DATABRICKS_CLIENT_ID` environment variable.
  # client_id = "123-456-789"

  # OAuth secret value of a service principal
  # This can also be set via the `DATABRICKS_CLIENT_SECRET` environment variable.
  # client_secret = "dose1234567789abcde"
}

Detailed configuration guide →

Or through environment variables:

export DATABRICKS_CONFIG_PROFILE=user1-test
export DATABRICKS_TOKEN=dsapi5c72c067b40df73ccb6be3b085d3ba
export DATABRICKS_HOST=https://accounts.cloud.databricks.com
export DATABRICKS_ACCOUNT_ID=abcdd0f81-9be0-4425-9e29-3a7d96782373
export [email protected]
export DATABRICKS_PASSWORD=password
export DATABRICKS_CLIENT_ID=123-456-789
export DATABRICKS_CLIENT_SECRET=dose1234567789abcde

Run steampipe:

steampipe query

List details of your Databricks clusters:

select
  cluster_id,
  title,
  cluster_source,
  creator_user_name,
  driver_node_type_id,
  node_type_id,
  state,
  start_time
from
  databricks_compute_cluster;

+----------------------+--------------------------------+----------------+-------------------+---------------------+--------------+------------+---------------------------+
| cluster_id           | title                          | cluster_source | creator_user_name | driver_node_type_id | node_type_id | state      | start_time                |
+----------------------+--------------------------------+----------------+-------------------+---------------------+--------------+------------+---------------------------+
| 1234-141524-10b6dv2h | [default]basic-starter-cluster | "API"          | [email protected]   | i3.xlarge           | i3.xlarge    | TERMINATED | 2023-07-21T19:45:24+05:30 |
| 1234-061816-mvns8mxz | test-cluster-for-ml            | "UI"           | [email protected]   | i3.xlarge           | i3.xlarge    | TERMINATED | 2023-07-28T11:48:16+05:30 |
+----------------------+--------------------------------+----------------+-------------------+---------------------+--------------+------------+---------------------------+

Engines

This plugin is available for the following engines:

Engine	Description
Steampipe	The Steampipe CLI exposes APIs and services as a high-performance relational database, giving you the ability to write SQL-based queries to explore dynamic data. Mods extend Steampipe's capabilities with dashboards, reports, and controls built with simple HCL. The Steampipe CLI is a turnkey solution that includes its own Postgres database, plugin management, and mod support.
Postgres FDW	Steampipe Postgres FDWs are native Postgres Foreign Data Wrappers that translate APIs to foreign tables. Unlike Steampipe CLI, which ships with its own Postgres server instance, the Steampipe Postgres FDWs can be installed in any supported Postgres database version.
SQLite Extension	Steampipe SQLite Extensions provide SQLite virtual tables that translate your queries into API calls, transparently fetching information from your API or service as you request it.
Export	Steampipe Plugin Exporters provide a flexible mechanism for exporting information from cloud services and APIs. Each exporter is a stand-alone binary that allows you to extract data using Steampipe plugins without a database.
Turbot Pipes	Turbot Pipes is the only intelligence, automation & security platform built specifically for DevOps. Pipes provide hosted Steampipe database instances, shared dashboards, snapshots, and more.

Developing

Prerequisites:

Clone:

git clone https://github.com/turbot/steampipe-plugin-databricks.git
cd steampipe-plugin-databricks

Build, which automatically installs the new version to your ~/.steampipe/plugins directory:

make

Configure the plugin:

cp config/* ~/.steampipe/config
vi ~/.steampipe/config/databricks.spc

Try it!

steampipe query
> .inspect databricks

Open Source & Contributing

This repository is published under the Apache 2.0 (source code) and CC BY-NC-ND (docs) licenses. Please see our code of conduct. We look forward to collaborating with you!

Steampipe is a product produced from this open source software, exclusively by Turbot HQ, Inc. It is distributed under our commercial terms. Others are allowed to make their own distribution of the software, but cannot use any of the Turbot trademarks, cloud services, etc. You can learn more in our Open Source FAQ.

Get Involved

Join #steampipe on Slack →

Want to help but don't know where to start? Pick up one of the help wanted issues:

List job run in error if job_id doesn't exist

Describe the bug
When performing a query on databricks_job_run table, if there is a condition onjob_id and the job_id does not exists, an error is thrown.

This can be annoying when performing query on aggregated connection with condition like job_id IN (...)

Steampipe version (steampipe -v)
v0.21.2

Plugin version (steampipe plugin list)
v0.3.0

To reproduce
Steps to reproduce the behavior (please include relevant code and/or commands).
Run a query with a job_id that does not exists like

> select * from databricks.databricks_job_run where job_id = 42

Error: databricks_prod_videoland: Job 42 does not exist. (SQLSTATE HV000)

Expected behavior
If the job does not exist we expect to have no results instead

> select * from databricks.databricks_job_run where run_id = 42
+--------+----------+----------------+------------------+-------------------+----------+--------------------+--------+---------------+-------------------------+--------------+--------------+----------+----------------+------------+---------+----------->
| run_id | run_name | attempt_number | cleanup_duration | creator_user_name | end_time | execution_duration | job_id | number_in_job | original_attempt_run_id | run_duration | run_page_url | run_type | setup_duration | start_time | trigger | cluster_in>
+--------+----------+----------------+------------------+-------------------+----------+--------------------+--------+---------------+-------------------------+--------------+--------------+----------+----------------+------------+---------+----------->
+--------+----------+----------------+------------------+-------------------+----------+--------------------+--------+---------------+-------------------------+--------------+--------------+----------+----------------+------------+---------+----------->

Additional context
This is causes by Databricks SDK. If we provide a JobId (done here for that table), Databricks API return an error if the job does not exists.

A possible solution could be to return nil, nil if we have this exact error

turbot / steampipe-plugin-databricks Goto Github PK

steampipe-plugin-databricks's Introduction

Databricks Plugin for Steampipe

Quick start

Install

Engines

Developing

Open Source & Contributing

Get Involved

steampipe-plugin-databricks's People

Contributors

Stargazers

Watchers

Forkers

steampipe-plugin-databricks's Issues

Describe alternatives you've considered

Recommend Projects

Recommend Topics

Recommend Org