cloudquery / cq-provider-k8s Goto Github PK

CloudQuery Provider for Kubernetes

License: Mozilla Public License 2.0

Go 95.72% HCL 3.50% Makefile 0.78%

cq-provider-k8s's Introduction

CloudQuery is an open-source, high-performance data integration framework built for developers, with support for a wide range of plugins.

CloudQuery extracts, transforms, and loads configuration from cloud APIs, files or databases to variety of supported destinations such as databases, data lakes, or streaming platforms for further analysis.

Installation

See the Quickstart guide for instructions how to start syncing data with CloudQuery.

Why CloudQuery?

Blazing fast: CloudQuery is optimized for performance, utilizing the excellent Go concurrency model with light-weight goroutines.
Deploy anywhere: CloudQuery plugins are single-binary executables and can be deployed and run anywhere.
Open source: Language-agnostic, extensible plugin architecture using Apache Arrow: develop your own plugins in Go, Python, Java or JavaScript using the CloudQuery SDK.
Pre-built queries: CloudQuery maintains a number of out-of-the-box security and compliance policies for cloud infrastructure.
Unlimited scale: CloudQuery plugins are stateless and can be scaled horizontally on any platform, such as EC2, Kubernetes, batch jobs or any other compute.

Use Cases

Cloud Security Posture Management: Use as an open source CSPM solution to monitor and enforce security policies across your cloud infrastructure for AWS, GCP, Azure and many more.
Cloud Asset Inventory: First-class support for major cloud infrastructure providers such as AWS, GCP and Azure allow you to collect and unify configuration data.
Cloud FinOps: Collect and unify billing data from cloud providers to drive financial accountability.
ELT Platform: With hundreds of plugin combinations and extensible architecture, CloudQuery can be used for reliable, efficient export from any API to any database, or from one database to another.
Attack Surface Management: Open source solution for continuous discovery, analysis and monitoring of potential attack vectors that make up your organization's attack surface.
Eliminate data silos: Eliminate data silos across your organization, unifying data between security, infrastructure, marketing and finance teams.

License

By contributing to CloudQuery you agree that your contributions will be licensed as defined on the LICENSE file.

Hiring

If you are into Go, Backend, Cloud, GCP, AWS - ping us at jobs [at] our domain

Contribution

Feel free to open a pull request for small fixes and changes. For bigger changes and new plugins, please open an issue first to prevent duplicated work and to have the relevant discussions first.

Open source and open core

The CloudQuery framework, SDK and CLI are open source while plugins available under plugins are open core, hence not all contributions to plugins directory will be accepted if they are part of the commercial plugin offering - please file an issue before opening a PR.

cq-provider-k8s's People

Contributors

Stargazers

Watchers

Forkers

francilioaraujo mbenziane amanenk zagronitay bbernays erezrokah hermanschaaf

cq-provider-k8s's Issues

Add k8s Contribution Guide

New Resource

Use Case

Additional context

Add error classifier

Add Support for k8s error classifier to classify access issues

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: Cannot find preset's package (github>cloudquery/.github//.github/renovate-go-default.json5)

k8s provider README is very lacking.

specifically authentication section - you don't actually need a KUBECONFIG environment vairable - cloudquery will just use the default if one isn't sepecified...

Fix integration tests

Describe the Bug

Expected Behavior

CloudQuery Version

Debug Output

Steps to Reproduce

Additional Context

References

Add nightly fetch workflow to k8s

update docs

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

fix(deps): Update tubone24/update_release digest to cf5b1c5
fix(deps): Update leonsteinhaeuser/project-beta-automations action to v1.3.0
fix(deps): Update module go to 1.19

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

fix(deps): Update module k8s.io/client-go to v12

Detected dependencies

github-actions

.github/workflows/issue_to_project.yml

leonsteinhaeuser/project-beta-automations v1.2.1

.github/workflows/lint_doc.yml

actions/checkout v3

dorny/paths-filter v2

actions/setup-go v3

actions/cache v3

.github/workflows/lint_golang.yml

actions/checkout v3

dorny/paths-filter v2

actions/setup-go v3

golangci/golangci-lint-action v3

.github/workflows/pr_title.yml

amannn/action-semantic-pull-request v4

.github/workflows/release-pr.yml

google-github-actions/release-please-action v3

tubone24/update_release 2146f1550a23d883b8ea0c036298ed74cd65eac6

.github/workflows/release.yml

booxmedialtd/ws-action-parse-semver 966a26512c94239a00aa10b1b0c196906f7e1909

actions/checkout v3

actions/setup-go v3

goreleaser/goreleaser-action v3

crazy-max/ghaction-import-gpg v5

goreleaser/goreleaser-action v3

.github/workflows/test_policy_sql.yml

actions/checkout v3

actions/setup-go v3

actions/cache v3

undefined no version found

.github/workflows/test_unit.yml

actions/checkout v3

dorny/paths-filter v2

actions/setup-go v3

actions/cache v3

undefined no version found

.github/workflows/validate_release.yml

actions/checkout v3

actions/setup-go v3

goreleaser/goreleaser-action v3

gomod

go.mod

go 1.18

github.com/cloudquery/cq-provider-sdk v0.14.7

github.com/cloudquery/faker/v3 v3.7.7

github.com/golang/mock v1.6.0

github.com/hashicorp/go-hclog v1.2.2

github.com/thoas/go-funk v0.9.2

k8s.io/api v0.24.3

k8s.io/apimachinery v0.24.3

k8s.io/client-go v0.24.3

terraform

resources/integration_tests/infra/k8s_apps_daemon_sets.tf

nginx 1.23.1

resources/integration_tests/infra/k8s_apps_deployments.tf

nginx 1.23.1

resources/integration_tests/infra/k8s_apps_replica_sets.tf

nginx 1.23.1

resources/integration_tests/infra/k8s_apps_stateful_sets.tf

mongo bionic

resources/integration_tests/infra/k8s_batch_cron_jobs.tf

busybox no version found

resources/integration_tests/infra/k8s_batch_jobs.tf

perl no version found

resources/integration_tests/infra/k8s_core_endpoints.tf

undefined no version found

resources/integration_tests/infra/k8s_core_limit_ranges.tf

undefined no version found

undefined no version found

resources/integration_tests/infra/k8s_core_namespaces.tf

undefined no version found

resources/integration_tests/infra/k8s_core_nodes.tf

nginx 1.23.1

resources/integration_tests/infra/k8s_core_pods.tf

nginx 1.23.1

resources/integration_tests/infra/k8s_core_resource_quotas.tf

undefined no version found

undefined no version found

resources/integration_tests/infra/k8s_core_service_accounts.tf

undefined no version found

undefined no version found

resources/integration_tests/infra/k8s_core_services.tf

undefined no version found

nginx 1.23.1

resources/integration_tests/infra/k8s_networking_network_policies.tf

undefined no version found

resources/integration_tests/infra/k8s_rbac_role_bindings.tf

undefined no version found

resources/integration_tests/infra/k8s_rbac_roles.tf

undefined no version found

resources/integration_tests/infra/provider.tf

kubernetes no version found

resources/integration_tests/infra/terraform.tf

terraform ~> 2.6

Panic when trying to fetch from a specifc context

Describe the Bug

Setting contexts parameter in config.hcl causees panic.

Expected Behavior

cloudquery should fetch from this context.

CloudQuery Version

Version: 0.22.7
Commit: fb5a1447e0851d96af0fb7d522585bf78566ff3f
Date: 2022-04-19T08:55:30Z

cq-provider-k8s version: v0.4.2

Debug Output

See attached file.
log.txt

Steps to Reproduce

config.hcl file:

// Configuration AutoGenerated by CloudQuery CLI
cloudquery {
  plugin_directory = "./cq/providers"
  policy_directory = "./cq/policies"

  provider "k8s" {
    version = "latest"
  }

  connection {
    dsn = "postgres://postgres:pass@localhost:5432/postgres?sslmode=disable"
  }
}

// All Provider Configurations

provider "k8s" {
  configuration {
    // Optional. Set contexts that you want to fetch. If it is not given then all contexts from config are iterated over.
    contexts = ["one"]
  }
  // list of resources to fetch
  resources = [
    "apps.daemon_sets",
    "apps.deployments",
    "apps.replica_sets",
    "apps.stateful_sets",
    "batch.cron_jobs",
    "batch.jobs",
    "core.endpoints",
    "core.limit_ranges",
    "core.namespaces",
    "core.nodes",
    "core.pods",
    "core.resource_quotas",
    "core.service_accounts",
    "core.services",
    "networking.network_policies",
    "rbac.role_bindings",
    "rbac.roles"
  ]
  // enables partial fetching, allowing for any failures to not stop full resource pull
  enable_partial_fetch = true
}

cloudquery fetch

Additional Context

References

update goreleaser to support all platforms

Very weird behaviour when fetching from different clusters/contexts

Describe the Bug

Two separate but related issues here:

Our documentation states that cloudquery fetched from "all available kubernetes contexts in the config file". However, cloudquery only actually fetches from the current-context in the kubeconfig file.
cloudquery doesn't delete old entries if the user switched contexts. This may be related to "purge".

Expected Behavior

only fetching from current context makes sense to me, so I think just changing docs is OK. I actually think fetching from all contexts is a bad idea, because kubeconfig files tend to accumulate a lot of stale contexts (e.g. mine has a bunch of EKS clusters from AWS accounts that no longer exist).
I definitely expect old entries to be deleted if a user switched context. Should the user really explicitly "purge" the old records? I don't love that user experience.

CloudQuery Version

Version: 0.22.7
Commit: fb5a1447e0851d96af0fb7d522585bf78566ff3f
Date: 2022-04-19T08:55:30Z

cq-provider-k8s version: v0.4.2

Steps to Reproduce

Assume kubectl and minikube is installed.

minikube start --profile one
minikube start --profile two
minikube start --profile three

kubectl config use-context one
cloudquery fetch
# k8s_core_nodes now contains 1 row, for profile 'one'
kubectl config use-context two
cloduquery fetch
# k8s_core_nodes now contains 2 rows - node 'two', and node 'one' that wasn't deleted. 
kubectl config use-context three
cloudquery fetch
# k8s_core_nodes now contains 3 rows - node 'three', and nodes 'one' and 'two' that weren't deleted.

Dependency Dashboard

This issue provides visibility into Renovate updates and their statuses. Learn more

This repository currently has no open or pending branches.

Detected dependencies

github-actions

.github/workflows/issue_to_project.yml

leonsteinhaeuser/project-beta-automations v1.2.1

.github/workflows/lint_doc.yml

actions/checkout v3

dorny/paths-filter v2

actions/setup-go v3

actions/cache v3

.github/workflows/lint_golang.yml

actions/checkout v3

dorny/paths-filter v2

actions/setup-go v3

golangci/golangci-lint-action v3

.github/workflows/pr_title.yml

amannn/action-semantic-pull-request v3.4.6

.github/workflows/release-pr.yml

google-github-actions/release-please-action v3

tubone24/update_release undefined

.github/workflows/release.yml

booxmedialtd/ws-action-parse-semver undefined

actions/checkout v2

actions/setup-go v2

goreleaser/goreleaser-action v2

crazy-max/ghaction-import-gpg v3

goreleaser/goreleaser-action v2

.github/workflows/test_policy.yml

actions/checkout v2

actions/setup-go v3

actions/cache v3

.github/workflows/test_unit.yml

actions/checkout v3

dorny/paths-filter v2

actions/setup-go v3

actions/cache v3

.github/workflows/validate_release.yml

actions/checkout v3

actions/setup-go v3

goreleaser/goreleaser-action v2

gomod

go.mod

github.com/cloudquery/cq-provider-sdk v0.9.5

github.com/cloudquery/faker/v3 v3.7.5

github.com/golang/mock v1.6.0

github.com/hashicorp/go-hclog v1.0.0

github.com/thoas/go-funk v0.9.1

k8s.io/api v0.22.1

k8s.io/apimachinery v0.22.1

k8s.io/client-go v0.22.1

terraform

resources/integration_tests/infra/provider.tf

kubernetes undefined

resources/integration_tests/infra/terraform.tf

terraform ~> 2.6

Check this box to trigger a request for Renovate to run again on this repository

add changelog

create e2e testing for k8s

Check api availability before doing any calls to k8s server

K8s server allows to get api schema by calling kClient.OpenAPISchema()
The method returns list of paths available on the server.
The data from response can be used to avoid calls to endpoints that are not supported bu current version of k8s server

Support for querying labels and annotations

It would be tremendously useful to include labels and annotations as something that could be queried. I might have time to make a PR for this at some point. Would you prefer a single polymorphic labels table that is shared across resources, or something like PodLabels?

cloudquery / cq-provider-k8s Goto Github PK

cq-provider-k8s's Introduction

Installation

Why CloudQuery?

Use Cases

Links

License

Hiring

Contribution

Open source and open core

cq-provider-k8s's People

Contributors

Stargazers

Watchers

Forkers

cq-provider-k8s's Issues

New Resource

Use Case

Additional context

Describe the Bug

Expected Behavior

CloudQuery Version

Debug Output

Steps to Reproduce

Additional Context

References

Awaiting Schedule

Ignored or Blocked

Detected dependencies

Describe the Bug

Expected Behavior

CloudQuery Version

Debug Output

Steps to Reproduce

Additional Context

References

Describe the Bug

Expected Behavior

CloudQuery Version

Steps to Reproduce

Detected dependencies

Recommend Projects

Recommend Topics

Recommend Org