Giter VIP home page Giter VIP logo

terraform-provider-pinot's Introduction

terraform-provider-pinot

A Terraform provider for Apache Pinot

Contents

Using the provider

Installation:

terraform {
  required_providers {
    pinot = {
      source = "azaurus1/pinot"
      version = "0.2.0"
    }
  }
}

provider "pinot" {
  controller_url = "http://localhost:9000" //required (can also be set via environment variable PINOT_CONTROLLER_URL)
  auth_token     = "YWRtaW46dmVyeXNlY3JldA" //optional (can also be set via environment variable PINOT_AUTH_TOKEN) 
}

Examples:

Example Schema:

resource "pinot_schema" "block_schema" {
  schema_name = "ethereum_block_headers"
  date_time_field_specs = [{
    data_type   = "LONG",
    name        = "block_timestamp",
    format      = "1:MILLISECONDS:EPOCH",
    granularity = "1:MILLISECONDS",
  }]
  dimension_field_specs = [{
    name      = "block_number",
    data_type = "INT",
    not_null  = true
    },
    {
      name      = "block_hash",
      data_type = "STRING",
      not_null  = true
  }]
  metric_field_specs = [{
    name      = "block_difficulty",
    data_type = "INT",
    not_null  = true
  }]
}

Example Table:

resource "pinot_table" "realtime_table" {

  table_name = "realtime_ethereum_mainnet_block_headers_REALTIME"
  table_type = "REALTIME"
  table      = file("realtime_table_example.json")

Example User:

resource "pinot_user" "test" {
  username  = "user"
  password  = "password"
  component = "BROKER"
  role      = "USER"

  lifecycle {
    ignore_changes = [password]
  }
}

This provider is built on the Terraform Plugin Framework. The template repository built on the Terraform Plugin SDK can be found at terraform-provider-scaffolding.

Requirements

To run this in your local environment be sure to:

  1. run go env GOBIN
  2. Create ~/.terraformrc with the following:
provider_installation {

  dev_overrides {
      "hashicorp.com/edu/pinot" = [Location of your GOBIN]
  }

  # For all other providers, install them directly from their origin provider
  # registries as normal. If you omit this, Terraform will _only_ use
  # the dev_overrides block, and so no other providers will be available.
  direct {}
}

Building The Provider

  1. Clone the repository
  2. Enter the repository directory
  3. Build the provider using the Go install command:
go install

Adding Dependencies

This provider uses Go modules. Please see the Go documentation for the most up to date information about using Go modules.

To add a new dependency github.com/author/dependency to your Terraform provider:

go get github.com/author/dependency
go mod tidy

Then commit the changes to go.mod and go.sum.

Developing the Provider

If you wish to work on the provider, you'll first need Go installed on your machine (see Requirements above).

To compile the provider, run go install. This will build the provider and put the provider binary in the $GOPATH/bin directory.

To generate or update documentation, run go generate.

In order to run the full suite of Acceptance tests, run make testacc.

Note: Acceptance tests create real resources, and often cost money to run.

make testacc

Acknowledgments:

Many thanks to:

terraform-provider-pinot's People

Contributors

azaurus1 avatar hendoxc avatar dependabot[bot] avatar

Stargazers

 avatar Peter Corless avatar Mathew Gardner avatar Vítor avatar Gustavo Vieira avatar Gabriel Custódio avatar  avatar  avatar

Watchers

 avatar

Forkers

guustavov

terraform-provider-pinot's Issues

Improve Schema Resource

  • Improve the schema resource to include more fields, currently it only supports reading files as the schema definition

Add Segment Terraform Resource

Create is not implemented in go-pinot-api

  • Update is not available in the controller API, probably only solution to implementing update would be to Recreate (Delete -> Create)

Add Table Schema Terraform Resource

Context

Add functionality so that a user can:

  • create a new table schema
  • update an existing one
  • destroy a schema

AC

  • create a new table schema
  • update an existing one
  • destroy a schema

Tables: segmentsConfig.timeType should not be mandatory

For creating a table, segments_config.time_type is required:

│   on main.tf line 133, in resource "pinot_table" "realtime_table":
│  133:   segments_config = merge(local.segments_config, {
│  134:     replication = "1"
│  135:   })
│     ├────────────────
│     │ local.segments_config is object with 6 attributes
│ 
│ Inappropriate value for attribute "segments_config": attribute "time_type" is required.

Whereas creating a tables without the "timeType" setting works using curl/the REST API. IMHO the provider should behave like the REST API does and not enforce "timeType" to be set.

Support the `singleValueField` property on dimension fields

Hey @azaurus1, I appreciate the effort you've been putting into this provider. I noticed that while setting up a table schema, some field properties are not supported. I see that you've recently added support for the notNull attribute, which is great. However, our use case requires setting a dimension column as multi-valued. The Pinot schema documentation (https://docs.pinot.apache.org/configuration-reference/schema#dimensionfieldspec) describes the singleValueField property, which is a boolean indicating if the field is a single-valued or a multi-valued column. In our case, a multi-valued column is modeled as a list, where the order of the values is preserved and duplicate values are allowed.

I'm willing to contribute to this if you're open to it. If so, I'm curious whether your intention is to have a single shared struct to define field properties or if you'd prefer to define specific structs for each type of field spec. For example:

type FieldSpec struct {
    Name        string `json:"name"`
    DataType    string `json:"dataType"`
    NotNull     bool   `json:"notNull,omitempty"`
    Format      string `json:"format,omitempty"`
    Granularity string `json:"granularity,omitempty"`
}

type Schema struct {
    SchemaName          string      `json:"schemaName"`
    DimensionFieldSpecs []FieldSpec `json:"dimensionFieldSpecs"`
    MetricFieldSpecs    []FieldSpec `json:"metricFieldSpecs,omitempty"`
    DateTimeFieldSpecs  []FieldSpec `json:"dateTimeFieldSpecs,omitempty"`
    PrimaryKeyColumns   []string    `json:"primaryKeyColumns,omitempty"`
}

If you have any guidance on how to proceed with adding extra properties, I'm ready to work on it.

Improve Table Resource

  • Improve the table resource to include more fields, currently it only supports reading files as the table definition

Create CONTRIBUTING file

Hey! Good initiative! Should we create a CONTRIBUTE.md file to guide new PRs and contributions?

Enable authentication via Bearer token

Hello @azaurus1 and other contributors, I hope you're all doing well!

First off, big kudos on the recent updates – the progress of the provider has been impressive!

I wanted to throw a suggestion into the mix: any chance we could incorporate support for Bearer Token authentication alongside Basic auth? Any chances of this being on your roadmap in the near future? It would be a game-changer for those of us working with managed Apache Pinot services and other similar setups.

Looking forward to hearing your thoughts on this!

Tables: "terraform apply" fails if tenants are not explicitly specified

If you don't explicitly specify tenants in your resource definition, the "terraform plan" works & creates the correct plan, but "terraform apply" fails:

      + table_type         = "OFFLINE"
      + tenants            = {}
    }

Plan: 2 to add, 0 to change, 0 to destroy.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"

➜  tables git:(main) ✗ terraform apply "tfplan"                             
pinot_schema.realtime_table_schema: Creating...
pinot_schema.realtime_table_schema: Creation complete after 0s
pinot_table.realtime_table: Creating...
╷
│ Error: Create Failed: Unable to create table
│ 
│   with pinot_table.realtime_table,
│   on main.tf line 127, in resource "pinot_table" "realtime_table":
│  127: resource "pinot_table" "realtime_table" {
│ 
│ client: request failed: status 400
│ {"code":400,"error":"Invalid table config for table realtime_ethereum_mainnet_block_headers_OFFLINE: Failed to find instances with tag: _BROKER for table: realtime_ethereum_mainnet_block_headers_OFFLINE"}

Whereas if you create a table using curl/REST API calls and having empty tenants, it works. IMHO the provider should behave the same as the REST API in this case.

Crash while running "terraform plan" for tables a second time after creation

The provider (v.0.7.5) currently crashes if a plan is created the second time. This can be reproduced even with the example provided in the repo (I just modified the table from a REALTIME table to a OFFLINE table because I didn't had a Kafka at hand)

steps to reproduce:

  1. Start an new pinot instance locally
  2. go to the examples/tables folder
  3. run terraform init, terraform plan & terraform apply
  4. everything works, i.e. no error messages & schema and table are created.
  5. run terraform plan again

expected behavior: "terraform plan" works & no changes to the plan

actual behavior:

Initializing the backend...

Initializing provider plugins...
- Finding azaurus1/pinot versions matching "0.7.5"...
- Using previously-installed azaurus1/pinot v0.7.5

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
pinot_schema.realtime_table_schema: Refreshing state...
pinot_table.realtime_table: Refreshing state...

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Plugin did not respond
│ 
│ The plugin encountered an error, and failed to respond to the plugin6.(*GRPCProvider).ReadResource call. The plugin logs may contain more details.
╵

Stack trace from the terraform-provider-pinot_v0.7.5 plugin:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x7758829]

goroutine 81 [running]:
terraform-provider-pinot/internal/converter.convertIngestionConfig(0xc000323060)
        terraform-provider-pinot/internal/converter/converter.go:85 +0x29
terraform-provider-pinot/internal/converter.SetStateFromTable({0x7cd1888, 0xc0000afda0}, 0xc0003ecc00, 0xc000323060)
        terraform-provider-pinot/internal/converter/converter.go:22 +0x1c5
terraform-provider-pinot/internal/provider.(*tableResource).Read(0xc00008a000, {0x7cd1888, 0xc0000afda0}, {{{{0x7cd6af8, 0xc0004c7aa0}, {0x7bc9d60, 0xc00020e030}}, {0x7cd9ef0, 0xc000210230}}, 0xc00008a418, ...}, ...)
        terraform-provider-pinot/internal/provider/tables_resource.go:156 +0x247
github.com/hashicorp/terraform-plugin-framework/internal/fwserver.(*Server).ReadResource(0xc0000c16c0, {0x7cd1888, 0xc0000afda0}, 0xc0000afe00, 0xc0003236c8)
        github.com/hashicorp/[email protected]/internal/fwserver/server_readresource.go:101 +0x62e
github.com/hashicorp/terraform-plugin-framework/internal/proto6server.(*Server).ReadResource(0xc0000c16c0, {0x7cd1888?, 0xc0000afcb0?}, 0xc0003b6340)
        github.com/hashicorp/[email protected]/internal/proto6server/server_readresource.go:55 +0x38e
github.com/hashicorp/terraform-plugin-go/tfprotov6/tf6server.(*server).ReadResource(0xc0001d86e0, {0x7cd1888?, 0xc0000aebd0?}, 0xc0001ca2a0)
        github.com/hashicorp/[email protected]/tfprotov6/tf6server/server.go:784 +0x309
github.com/hashicorp/terraform-plugin-go/tfprotov6/internal/tfplugin6._Provider_ReadResource_Handler({0x7ca17a0, 0xc0001d86e0}, {0x7cd1888, 0xc0000aebd0}, 0xc0003a2c00, 0x0)
        github.com/hashicorp/[email protected]/tfprotov6/internal/tfplugin6/tfplugin6_grpc.pb.go:482 +0x1a6
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000195000, {0x7cd1888, 0xc0000ae7e0}, {0x7cd81d8, 0xc0003fe000}, 0xc00056e120, 0xc0002fdad0, 0x81d4fc8, 0x0)
        google.golang.org/[email protected]/server.go:1369 +0xdf8
google.golang.org/grpc.(*Server).handleStream(0xc000195000, {0x7cd81d8, 0xc0003fe000}, 0xc00056e120)
        google.golang.org/[email protected]/server.go:1780 +0xe8b
google.golang.org/grpc.(*Server).serveStreams.func2.1()
        google.golang.org/[email protected]/server.go:1019 +0x8b
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 23
        google.golang.org/[email protected]/server.go:1030 +0x125

Error: The terraform-provider-pinot_v0.7.5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.