azaurus1 / terraform-provider-pinot Goto Github PK
View Code? Open in Web Editor NEWA terraform provider for Apache Pinot
Home Page: https://registry.terraform.io/providers/azaurus1/pinot/latest
License: Mozilla Public License 2.0
A terraform provider for Apache Pinot
Home Page: https://registry.terraform.io/providers/azaurus1/pinot/latest
License: Mozilla Public License 2.0
Hi,
Thanks for this providor! Is it possible to deploy schema from json file instead?
Add acceptance testing for users
Create the Table resource in Terraform
Add a terraform data source for tenants
I want to create users which have permissions on certain tables only. The current version of the provider doesn't seem to support that. Do you have any plans to add this?
Add config_sensitve like this:
config_sensitive = {
"connection.password" = "this-should-never-appear-unmasked"
}
Hey @azaurus1, I appreciate the effort you've been putting into this provider. I noticed that while setting up a table schema, some field properties are not supported. I see that you've recently added support for the notNull
attribute, which is great. However, our use case requires setting a dimension column as multi-valued. The Pinot schema documentation (https://docs.pinot.apache.org/configuration-reference/schema#dimensionfieldspec) describes the singleValueField
property, which is a boolean indicating if the field is a single-valued or a multi-valued column. In our case, a multi-valued column is modeled as a list, where the order of the values is preserved and duplicate values are allowed.
I'm willing to contribute to this if you're open to it. If so, I'm curious whether your intention is to have a single shared struct to define field properties or if you'd prefer to define specific structs for each type of field spec. For example:
type FieldSpec struct {
Name string `json:"name"`
DataType string `json:"dataType"`
NotNull bool `json:"notNull,omitempty"`
Format string `json:"format,omitempty"`
Granularity string `json:"granularity,omitempty"`
}
type Schema struct {
SchemaName string `json:"schemaName"`
DimensionFieldSpecs []FieldSpec `json:"dimensionFieldSpecs"`
MetricFieldSpecs []FieldSpec `json:"metricFieldSpecs,omitempty"`
DateTimeFieldSpecs []FieldSpec `json:"dateTimeFieldSpecs,omitempty"`
PrimaryKeyColumns []string `json:"primaryKeyColumns,omitempty"`
}
If you have any guidance on how to proceed with adding extra properties, I'm ready to work on it.
Guides can be created in the docs directory under the sub-directory guides
example in this repo: https://github.com/Telmate/terraform-provider-proxmox
Add a Cluster terraform Data source
Hey! Good initiative! Should we create a CONTRIBUTE.md file to guide new PRs and contributions?
Create is not implemented in go-pinot-api
For creating a table, segments_config.time_type is required:
│ on main.tf line 133, in resource "pinot_table" "realtime_table":
│ 133: segments_config = merge(local.segments_config, {
│ 134: replication = "1"
│ 135: })
│ ├────────────────
│ │ local.segments_config is object with 6 attributes
│
│ Inappropriate value for attribute "segments_config": attribute "time_type" is required.
Whereas creating a tables without the "timeType" setting works using curl/the REST API. IMHO the provider should behave like the REST API does and not enforce "timeType" to be set.
The provider (v.0.7.5) currently crashes if a plan is created the second time. This can be reproduced even with the example provided in the repo (I just modified the table from a REALTIME table to a OFFLINE table because I didn't had a Kafka at hand)
steps to reproduce:
expected behavior: "terraform plan" works & no changes to the plan
actual behavior:
Initializing the backend...
Initializing provider plugins...
- Finding azaurus1/pinot versions matching "0.7.5"...
- Using previously-installed azaurus1/pinot v0.7.5
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
pinot_schema.realtime_table_schema: Refreshing state...
pinot_table.realtime_table: Refreshing state...
Planning failed. Terraform encountered an error while generating this plan.
╷
│ Error: Plugin did not respond
│
│ The plugin encountered an error, and failed to respond to the plugin6.(*GRPCProvider).ReadResource call. The plugin logs may contain more details.
╵
Stack trace from the terraform-provider-pinot_v0.7.5 plugin:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x7758829]
goroutine 81 [running]:
terraform-provider-pinot/internal/converter.convertIngestionConfig(0xc000323060)
terraform-provider-pinot/internal/converter/converter.go:85 +0x29
terraform-provider-pinot/internal/converter.SetStateFromTable({0x7cd1888, 0xc0000afda0}, 0xc0003ecc00, 0xc000323060)
terraform-provider-pinot/internal/converter/converter.go:22 +0x1c5
terraform-provider-pinot/internal/provider.(*tableResource).Read(0xc00008a000, {0x7cd1888, 0xc0000afda0}, {{{{0x7cd6af8, 0xc0004c7aa0}, {0x7bc9d60, 0xc00020e030}}, {0x7cd9ef0, 0xc000210230}}, 0xc00008a418, ...}, ...)
terraform-provider-pinot/internal/provider/tables_resource.go:156 +0x247
github.com/hashicorp/terraform-plugin-framework/internal/fwserver.(*Server).ReadResource(0xc0000c16c0, {0x7cd1888, 0xc0000afda0}, 0xc0000afe00, 0xc0003236c8)
github.com/hashicorp/[email protected]/internal/fwserver/server_readresource.go:101 +0x62e
github.com/hashicorp/terraform-plugin-framework/internal/proto6server.(*Server).ReadResource(0xc0000c16c0, {0x7cd1888?, 0xc0000afcb0?}, 0xc0003b6340)
github.com/hashicorp/[email protected]/internal/proto6server/server_readresource.go:55 +0x38e
github.com/hashicorp/terraform-plugin-go/tfprotov6/tf6server.(*server).ReadResource(0xc0001d86e0, {0x7cd1888?, 0xc0000aebd0?}, 0xc0001ca2a0)
github.com/hashicorp/[email protected]/tfprotov6/tf6server/server.go:784 +0x309
github.com/hashicorp/terraform-plugin-go/tfprotov6/internal/tfplugin6._Provider_ReadResource_Handler({0x7ca17a0, 0xc0001d86e0}, {0x7cd1888, 0xc0000aebd0}, 0xc0003a2c00, 0x0)
github.com/hashicorp/[email protected]/tfprotov6/internal/tfplugin6/tfplugin6_grpc.pb.go:482 +0x1a6
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000195000, {0x7cd1888, 0xc0000ae7e0}, {0x7cd81d8, 0xc0003fe000}, 0xc00056e120, 0xc0002fdad0, 0x81d4fc8, 0x0)
google.golang.org/[email protected]/server.go:1369 +0xdf8
google.golang.org/grpc.(*Server).handleStream(0xc000195000, {0x7cd81d8, 0xc0003fe000}, 0xc00056e120)
google.golang.org/[email protected]/server.go:1780 +0xe8b
google.golang.org/grpc.(*Server).serveStreams.func2.1()
google.golang.org/[email protected]/server.go:1019 +0x8b
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 23
google.golang.org/[email protected]/server.go:1030 +0x125
Error: The terraform-provider-pinot_v0.7.5 plugin crashed!
This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.
Add functionality so that a user can:
Hello @azaurus1 and other contributors, I hope you're all doing well!
First off, big kudos on the recent updates – the progress of the provider has been impressive!
I wanted to throw a suggestion into the mix: any chance we could incorporate support for Bearer Token authentication alongside Basic auth? Any chances of this being on your roadmap in the near future? It would be a game-changer for those of us working with managed Apache Pinot services and other similar setups.
Looking forward to hearing your thoughts on this!
Implement Integration testing
Run go generate
for the provider
If you don't explicitly specify tenants in your resource definition, the "terraform plan" works & creates the correct plan, but "terraform apply" fails:
+ table_type = "OFFLINE"
+ tenants = {}
}
Plan: 2 to add, 0 to change, 0 to destroy.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Saved the plan to: tfplan
To perform exactly these actions, run the following command to apply:
terraform apply "tfplan"
➜ tables git:(main) ✗ terraform apply "tfplan"
pinot_schema.realtime_table_schema: Creating...
pinot_schema.realtime_table_schema: Creation complete after 0s
pinot_table.realtime_table: Creating...
╷
│ Error: Create Failed: Unable to create table
│
│ with pinot_table.realtime_table,
│ on main.tf line 127, in resource "pinot_table" "realtime_table":
│ 127: resource "pinot_table" "realtime_table" {
│
│ client: request failed: status 400
│ {"code":400,"error":"Invalid table config for table realtime_ethereum_mainnet_block_headers_OFFLINE: Failed to find instances with tag: _BROKER for table: realtime_ethereum_mainnet_block_headers_OFFLINE"}
Whereas if you create a table using curl/REST API calls and having empty tenants, it works. IMHO the provider should behave the same as the REST API in this case.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.