Giter VIP home page Giter VIP logo

turbot / steampipe Goto Github PK

View Code? Open in Web Editor NEW
6.4K 39.0 245.0 22 MB

Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.

Home Page: https://steampipe.io

License: GNU Affero General Public License v3.0

Shell 5.66% JavaScript 2.22% Makefile 0.02% Go 58.11% PLSQL 0.25% HCL 13.84% Dockerfile 0.07% HTML 0.45% CSS 0.32% TypeScript 18.44% Scheme 0.19% Puppet 0.41% Pascal 0.01%
steampipe sql postgresql postgresql-fdw cloud security aws azure cis cnapp

steampipe's Introduction

Steampipe Logo

plugins   slack   maintained by

Steampipe is the zero-ETL solution for getting data directly from APIs and services. We offer these Steampipe engines:

  • Steampipe CLI. Use the Steampipe core engine to translate APIs to tables in the Postgres instance that's bundled with Steampipe.

  • Steampipe Postgres FDWs. Use native Postgres Foreign Data Wrappers to translate APIs to foreign tables.

  • Steampipe SQLite extensions. Use SQLite extensions to translate APIS to SQLite virtual tables.

  • Steampipe export tools. Use standalone binaries that export data from APIs, no database required.

  • Turbot Pipes. Use Turbot Pipes to run Steampipe in the cloud.

Steampipe API plugins

The Steampipe community has grown a suite of plugins that map APIs to database tables. They work with all Steampipe engines.

Install Steampipe

The downloads page shows you how but tl;dr:

Linux or WSL

sudo /bin/sh -c "$(curl -fsSL https://steampipe.io/install/steampipe.sh)"

MacOS

brew tap turbot/tap
brew install steampipe

Install a plugin

Choose a plugin from the hub, for example: Hacker News.

Run the steampipe plugin command to install it.

steampipe plugin install hackernews

Query with a plugin

Run a query using psql — or another Postgres client , or Powerpipe — to query a table provided by the installed plugin.

psql -h localhost -p 9193 -d steampipe -U steampipe
steampipe=> select * from hackernews_new limit 10

Developing

Prerequisites:

  • Golang Version 1.19 or higher.

Clone the repo.

git clone https://github.com/turbot/steampipe
cd steampipe

Build, which automatically installs the new version to your /usr/local/bin/steampipe directory:

make

Check the version.

$ steampipe -v
steampipe version 0.21.1

Install a plugin and run a query as per above.

Open source & contributing

This repository is published under the AGPL 3.0 license. Please see our code of conduct. Contributors must sign our Contributor License Agreement as part of their first pull request. We look forward to collaborating with you!

Steampipe is a product produced from this open source software, exclusively by Turbot HQ, Inc. It is distributed under our commercial terms. Others are allowed to make their own distribution of the software, but cannot use any of the Turbot trademarks, cloud services, etc. You can learn more in our Open Source FAQ.

Turbot Pipes

Want a hosted version of Steampipe? Bring your team to Turbot Pipes.

Get involved

Join #steampipe on Slack →

Want to help but don't know where to start? Pick up one of the help wanted issues:

steampipe's People

Contributors

abhiturbot avatar ajmaradiaga avatar binaek avatar bob-bot avatar cbruno10 avatar chandru89new avatar cjsrkd3321 avatar dboeke avatar dependabot[bot] avatar e-gineer avatar francois2metz avatar gliptak avatar jreyesr avatar judell avatar juneezee avatar kaiba42 avatar kaidaguerre avatar loselarry avatar mafrosis avatar michaelburgess avatar michizhou avatar misraved avatar mr-destructive avatar pdecat avatar peterdavehello avatar ppapishe avatar pskrbasu avatar rupeshpatil20 avatar saumyabhushan avatar testwill avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

steampipe's Issues

Service management improvements

  • steampipe query cannot run if another query session is open which owns service
  • steampipe service cannot start if query owns service

paging fails for very long rows (> 64k chars)

select * from aws_iam_policy for pixie eu-west-2 returns 1 row longer than 64k. This breaks the code which detetcs if paging is necessary.

It can be fixed as follows:

func isPagerNeeded(content string) bool {
	maxCols, maxRow, _ := gows.GetWinSize()
	// let's scan through it instead of iterating over it fully
	sc := bufio.NewScanner(strings.NewReader(content))
	// explicitly allocate a large bugger for the scanner to use - otherwise we may fail for large rows 
	buffSize := 256 * 1024
	buff := make([]byte, buffSize)
	sc.Buffer(buff, buffSize)
	lineCount := 0
	for sc.Scan() {
		line := sc.Text()
		lineCount++
		if lineCount > maxRow {
			return true
		}
		if len(line) > maxCols {
			return true
		}
	}
	return false
}

`steampipe plugin list` not showing correct version of the installed plugins

seems like file ~/.steampipe/internal/versions.json is not getting updated when we install any existing plugin again

in below table for plugin hub.steampipe.io/plugins/turbot/aws@latest Version should be 0.0.10 but it is not showing any version

➜  steampipe-plugin-azure git:(main) steampipe plugin list                                                          
+--------------------------------------------------+---------+-------------+
|                       NAME                       | VERSION | CONNECTIONS |
+--------------------------------------------------+---------+-------------+
| hub.steampipe.io/plugins/turbot/aws@latest       |         | aws         |
| hub.steampipe.io/plugins/turbot/azure@latest     | local   | azure       |
| hub.steampipe.io/plugins/turbot/chaos@latest     |         | chaos       |
| hub.steampipe.io/plugins/turbot/steampipe@latest | 0.0.2   | steampipe   |
+--------------------------------------------------+---------+-------------+

Steampipe plugin install sometimes fails

Version: 0.1.0

 ~ $ steampipe -v
steampipe version 0.1.0
 ~ $ steampipe plugin install steampipe
Error: plugin install failed for plugin 'steampipe'

running the instal again seems to work...

rationalise state management

all states should be contained in one file located in internal/steampipe.json

all necessary state information should be read from here

also, consider removing the key from the versions file and make it an array instead of a map

Inspect should not wrap the comments column text

> .inspect zendesk.zendesk_group
+------------+-----------------------------+--------------------------------+
|   Column   |            Type             |          Description           |
+------------+-----------------------------+--------------------------------+
| created_at | timestamp without time zone | The time the group was created |
| deleted    | boolean                     | True if the group has been     |
|            |                             | deleted                        |
| id         | bigint                      | Unique identifier for the      |
|            |                             | group                          |
| name       | text                        | Name of the group              |
| updated_at | timestamp without time zone | The time of the last update of |
|            |                             | the group                      |
| url        | text                        | API url of the group           |
+------------+-----------------------------+--------------------------------+

steampipe versions file stores the CONFIG digest instead of the IMAGE digest

This is not currently causing an issue - i only noticed because the version is missing (theres already an open issue for that) and i was trying to match up the versions manually to check what version I had installed. It probably WILL be problematic when implementing steampipe plugin update ....

Registry image and manifest:
image

~/.steampipe/internal/versions.json :

  "hub.steampipe.io/plugins/turbot/steampipe@latest": {
      "name": "hub.steampipe.io/plugins/turbot/steampipe@latest",
      "version": "",
      "imageDigest": "sha256:50e6ed50cb04724cb7829ab7a2ba522048162a53b75423f8dc7c46e2fbda7f38",
      "installedFrom": "us-docker.pkg.dev/steampipe/plugins/turbot/steampipe:latest",
      "lastCheckedDate": "2021-01-13T00:10:23Z",
      "installDate": "2021-01-13T00:10:23Z"
    }

write plugin log statements to the console #180

Log statements at the appropriate level should be written to the console. Currently, you can specify the log level with SP_LOG, and log statements at that level and above will be written to the postgres logs. It would be great for debugging/troubleshooting to also print these to the console (possibly with postgres notice?))

Add support for exiting 'steampipe query' environment with Ctrl+C shortcut

Using Ctrl+C in my terminal for other programs allows me to exit quickly, e.g.,

Codys-MacBook-Pro:Desktop cbruno$ node
Welcome to Node.js v14.15.1.
Type ".help" for more information.
>
(To exit, press Ctrl+C again or Ctrl+D or type .exit)
>

For node, it also gives me the command that I can use to exit from the environment as a helper.

Is this something that can be added while inside steampipe query as well?

Command-Right breaks for unicode chars?

I had a query where I tried a unicode domain:

select * from whois_record where domain = 'München.com';

I can Command-Left through the line in history no problem:

Welcome to Steampipe v0.0.14
Type ".inspect" for more information.
> select * from whois_record where domain = 'München.com';

But as soon as I Command-Right over the ü character things break:

~/src/steampipe $ steampipe query
Welcome to Steampipe v0.0.14
Type ".inspect" for more information.
> select * from whois_record where domain =runtime error: slice bounds out of range [:57] with capacity 56

Columns are showing different values in output as json

Query 1:
steampipe query --output json "select * from chaos.all_column_types where id=99 order by id"
Output:

[
 {
  "array_element": {
   "Key": "stringValuesomething-99",
   "Value": "value"
  },
  "boolean_column": false,
  "cidr_column": "MTcyLjMxLjAuMC8xNg==",
  "date_time_column": "2002-05-05T09:23:00Z",
  "double_column": 5.823529411764706,
  "empty_hydrate": "I AM A DEFAULT",
  "id": 99,
  "ipaddress_column": "MTAuMC4wLjc=",
  "json_column": {
   "Id": 99,
   "Name": "stringValuesomething-99",
   "Statement": {
    "Action": "iam:SimulatePrincipalPolicy",
    "Effect": "Deny"
   }
  },
  "string_column": "stringValuesomething-99"
 }
]

Query2:
steampipe query "select * from chaos.all_column_types where id=99 order by id"
Output:
Screenshot 2021-01-13 at 8 05 35 PM

Add support for Postgres 14

Right now, steampipe operates with a stripped down version of pg12

This needs to be updated to pg14

Tasks:

  • Build and Deploy PG14 image to hub.steampipe.io
  • Build and Deploy FDW compatible with PG14 to hub.steampipe.io

Different behavior when using aws_iam_policy with regard to aws_region

MacOS = 10.15.7 / Steampipe = 0.1.0

Running queries against aws.aws_iam_policy return an error:

Error: pq: rpc error: code = Internal desc = hydrate call getCommonColumns failed with panic AWS_REGION must be set to use the aws extension

it is ignoring the default region set in ~/.aws/config.

While running a query against ACM seems to work fine:

Welcome to Steampipe v0.1.0
Type ".inspect" for more information.
> select * from aws_iam_policy
Error: pq: rpc error: code = Internal desc = hydrate call getCommonColumns failed with panic AWS_REGION must be set to use the aws extension
> select title from aws_acm_certificate
+--------------------------------------+
|                title                 |
+--------------------------------------+
| 82bf4707-0a47-4e55-af8b-ece0dd12fa7d |
| 034b902a-18c2-4686-b4c9-83dc3355e891 |
+--------------------------------------+

Incorrect multiline status

After enabling multiline mode while running a query, calling .multi gives the incorrect on/off status:

image

Stream table output instead of buffering it all

currently we wait for all output in order to size the columns

instead, wait for a page of results, and use this for sizing information, wrap any longer columns

NOTE: we may need an indicator when trying to page past the end of the available results

*still fetching*

or something

Passing a bad query as a parameter to `steampipe query` results in session remaining open.

Steampipe Version: v0.1.0
OS Version: MacOS 10.15.7

Issue
When passing a query as a cli parameter, if the query in question is not correct and returns an error, steampipe does not close the DB session and instead leaves it open, resulting in subsequent requests to fail due to steampipe being "open".

Steps to reproduce:

➜  steampipe query "select foo from bar;"
Error: pq: relation "bar" does not exist
➜  sp_demo steampipe query "select foo from bar;"
Error: You already have a steampipe query session open. To run multiple sessions, first run steampipe service start.
To kill existing sessions run steampipe service stop --force

Columns in text output should have a max width

The zendesk_ticket table goes pretty nuts now there is no max width for columns. Sideways scroll goes on a long long way.

Should we have a max column width?
Should it wrap or truncate?
What should it be? 80 chars? 120 chars? 1024 chars?

`runtime error: slice bounds out of range [1:0]` running `.inspect`

The first time i ran inspect on a new install after install a plugin, i got the following error:

> .inspect
+------------+--------------------------------------------------+
| Connection |                      Plugin                      |
+------------+--------------------------------------------------+
| steampipe  | hub.steampipe.io/plugins/turbot/steampipe@latest |
+------------+--------------------------------------------------+
To get information about the tables in a connection, run '.inspect {connection}'
To get information about the columns in a table, run '.inspect {connection}.{table}'
runtime error: slice bounds out of range [1:0]

improve cli feedback/guidance

  • when installing plugins, print a link to the documentation

  • when starting up show more useful message like

Welcome to Steampipe v0.1.0
Type ".help" for more information
Exit with .exit

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.