walterwanderley / sqlc-grpc Goto Github PK

View Code? Open in Web Editor NEW

125.0 125.0 13.0 900 KB

Create a gRPC server from code generated by sqlc

License: MIT License

Go 96.60% Shell 1.07% Dockerfile 0.94% PLpgSQL 1.39%

grpc sqlc sqlite

sqlc-grpc's People

Contributors

Stargazers

Watchers

Forkers

kokizzu niiniyare omirror cesargdd koolay libchaos hash167 jamesarthurholland kainsk doraa7 adoublef-tmp enigmatic-smile codeharik

sqlc-grpc's Issues

Is there a way to change the proto file or make additions without them being overwritten?

I have the following schema:

--pricing_schema.sql

CREATE TABLE pricing_schema (
    id UUID PRIMARY KEY,
    name varchar(255) NOT NULL,
    organization_id UUID NOT NULL,
    created_at timestamp NOT NULL DEFAULT current_timestamp,
    updated_at timestamp NOT NULL DEFAULT current_timestamp
);

CREATE TABLE price_category (
    id UUID PRIMARY KEY,
    name varchar(255) NOT NULL,
    pricing_schema_id UUID NOT NULL,
    created_at timestamp NOT NULL DEFAULT current_timestamp,
    updated_at timestamp NOT NULL DEFAULT current_timestamp
);

CREATE TABLE price (
    id UUID PRIMARY KEY,
    name varchar(255) NOT NULL,
    description varchar(255) NOT NULL,
    price_category_id UUID NOT NULL,
    price_in_smallest_donimation int NOT NULL,
    created_at timestamp NOT NULL DEFAULT current_timestamp,
    updated_at timestamp NOT NULL DEFAULT current_timestamp
);

I want to be able to use transactions to save a full pricing schema at once, including the nested parts.

schema
-> category1
----> item1
-> category2
----> item1
----> item2

I have the individual methods / queries for saving, but how do I create the addition to the proto file / grpc schema without having to write a sql query for the insert?

Implement as a sqlc WASM plugin using `sqlc-dev/plugin-sdk-go`

A suggestion from this Twitter thread is to implement the project as a sqlc WASM plugin. This would allow users to get the grpc codegen output from this project directly from sqlc generate. An example sqlc.yaml config might look like:

version: '2'
plugins:
- name: walterwanderley-sqlc-grpc
  wasm:
    url: https://some.cdn/walterwanderley-sqlc-grpc.wasm
    sha256: ""
sql:
- schema: schema.sql
  queries: query.sql
  engine: postgresql
  codegen:
  - plugin: walterwanderley-sqlc-grpc
    out: sqlc
    options:
      module: "my-module"
      tracing: true
      litefs: true
  gen:
    go:
      out: sqlc
      sql_package: pgx/v5

Running sqlc generate with this config would put both the typical sqlc Go codegen output and the grpc output from this project in the sqlc directory.

There's a sqlc plugin sdk for Go which should make the implementation a little more straightforward.

The only hiccup I can think of would be that sqlc's GenerateRequest proto message that sqlc hands to a plugin may not have enough information to support the grpc codegen work as you've currently implemented it.

authors example has some issues

Grapha and minio is up

Swagger not up.

I ran gen.sh first and then docker-compose up.

I use podman with the docker and docker compose alias's.

I on on v0.15.4. I tried main also. Exactly same problem

any ideas ?

sqlite && docker-compose up
[+] Running 5/0
 ✔ Container sqlite-prometheus-1     Created                                                                        0.0s 
 ✔ Container sqlite-minio-1          Created                                                                        0.0s 
 ✔ Container sqlite-grafana-1        Created                                                                        0.0s 
 ✔ Container sqlite-app-1            Created                                                                        0.0s 
 ✔ Container sqlite-createbuckets-1  Created                                                                        0.0s 
Attaching to sqlite-app-1, sqlite-createbuckets-1, sqlite-grafana-1, sqlite-minio-1, sqlite-prometheus-1
sqlite-app-1            | [00] Starting service
sqlite-createbuckets-1  | mc: <ERROR> Unable to initialize new alias from the provided credentials. Get "http://minio:9000/probe-bucket-sign-k9wtvvkvzzli/?location=": dial tcp 10.89.0.5:9000: connect: connection refused.
sqlite-prometheus-1     | ts=2023-04-10T12:22:13.489Z caller=main.go:520 level=info msg="No time or size retention was set so using the default time retention" duration=15d
sqlite-prometheus-1     | ts=2023-04-10T12:22:13.509Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)"
sqlite-prometheus-1     | ts=2023-04-10T12:22:13.513Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)"
sqlite-prometheus-1     | ts=2023-04-10T12:22:13.517Z caller=main.go:570 level=info host_details="(Linux 6.2.8-200.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 22 19:11:02 UTC 2023 x86_64 9ef338ce7d0a (none))"
sqlite-prometheus-1     | ts=2023-04-10T12:22:13.518Z caller=main.go:571 level=info fd_limits="(soft=524288, hard=524288)"
sqlite-prometheus-1     | ts=2023-04-10T12:22:13.534Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)"
sqlite-prometheus-1     | ts=2023-04-10T12:22:13.976Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
sqlite-prometheus-1     | ts=2023-04-10T12:22:14.054Z caller=main.go:1005 level=info msg="Starting TSDB ..."
sqlite-prometheus-1     | ts=2023-04-10T12:22:14.149Z caller=dir_locker.go:77 level=warn component=tsdb msg="A lockfile from a previous execution already existed. It was replaced" file=/prometheus/data/lock
sqlite-prometheus-1     | ts=2023-04-10T12:22:14.313Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9090
sqlite-prometheus-1     | ts=2023-04-10T12:22:14.321Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9090
sqlite-prometheus-1     | ts=2023-04-10T12:22:14.591Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
sqlite-prometheus-1     | ts=2023-04-10T12:22:14.610Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=100.999µs
sqlite-prometheus-1     | ts=2023-04-10T12:22:14.617Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while"
sqlite-prometheus-1     | ts=2023-04-10T12:22:15.005Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2
sqlite-prometheus-1     | ts=2023-04-10T12:22:15.032Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2
sqlite-prometheus-1     | ts=2023-04-10T12:22:15.037Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2
sqlite-prometheus-1     | ts=2023-04-10T12:22:15.055Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=157.999µs wal_replay_duration=437.259916ms wbl_replay_duration=15µs total_replay_duration=445.196878ms
sqlite-prometheus-1     | ts=2023-04-10T12:22:15.097Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC
sqlite-prometheus-1     | ts=2023-04-10T12:22:15.105Z caller=main.go:1029 level=info msg="TSDB started"
sqlite-prometheus-1     | ts=2023-04-10T12:22:15.175Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/config/prometheus.yml
sqlite-prometheus-1     | ts=2023-04-10T12:22:15.446Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/config/prometheus.yml totalDuration=271.058708ms db_storage=26µs remote_storage=34µs web_handler=14µs query_engine=15µs scrape=40.914805ms scrape_sd=100.999µs notify=25µs notify_sd=18µs rules=25µs tracing=133.999µs
sqlite-prometheus-1     | ts=2023-04-10T12:22:15.508Z caller=main.go:990 level=info msg="Server is ready to receive web requests."
sqlite-prometheus-1     | ts=2023-04-10T12:22:15.523Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..."
sqlite-createbuckets-1  | Bucket created successfully `myminio/somebucketname`.
sqlite-createbuckets-1  | mc: Please use 'mc anonymous'
sqlite-minio-1          | MinIO Object Storage Server
sqlite-minio-1          | Copyright: 2015-2023 MinIO, Inc.
sqlite-minio-1          | License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
sqlite-minio-1          | Version: RELEASE.2023-04-07T05-28-58Z (go1.20.3 linux/amd64)
sqlite-minio-1          | 
sqlite-minio-1          | Status:         1 Online, 0 Offline. 
sqlite-minio-1          | API: http://10.89.0.5:9000  http://127.0.0.1:9000     
sqlite-minio-1          | Console: http://10.89.0.5:9001 http://127.0.0.1:9001   
sqlite-minio-1          | 
sqlite-minio-1          | Documentation: https://min.io/docs/minio/linux/index.html
sqlite-minio-1          | Warning: The standard parity is set to 0. This can lead to data loss.
sqlite-grafana-1        | logger=settings t=2023-04-10T12:22:20.165294853Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z
sqlite-grafana-1        | logger=settings t=2023-04-10T12:22:20.198932692Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead."
sqlite-grafana-1        | logger=settings t=2023-04-10T12:22:20.20144368Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
sqlite-grafana-1        | logger=settings t=2023-04-10T12:22:20.207613651Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
sqlite-grafana-1        | logger=settings t=2023-04-10T12:22:20.210742636Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
sqlite-grafana-1        | logger=settings t=2023-04-10T12:22:20.211534632Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
sqlite-grafana-1        | logger=settings t=2023-04-10T12:22:20.238635503Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
sqlite-grafana-1        | logger=settings t=2023-04-10T12:22:20.244856473Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
sqlite-grafana-1        | logger=settings t=2023-04-10T12:22:20.249893449Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
sqlite-grafana-1        | logger=settings t=2023-04-10T12:22:20.251173443Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
sqlite-grafana-1        | logger=settings t=2023-04-10T12:22:20.25820341Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
sqlite-grafana-1        | logger=settings t=2023-04-10T12:22:20.265415375Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
sqlite-grafana-1        | logger=settings t=2023-04-10T12:22:20.266438371Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
sqlite-grafana-1        | logger=settings t=2023-04-10T12:22:20.267039368Z level=info msg="Path Home" path=/usr/share/grafana
sqlite-grafana-1        | logger=settings t=2023-04-10T12:22:20.267884364Z level=info msg="Path Data" path=/var/lib/grafana
sqlite-grafana-1        | logger=settings t=2023-04-10T12:22:20.286100277Z level=info msg="Path Logs" path=/var/log/grafana
sqlite-grafana-1        | logger=settings t=2023-04-10T12:22:20.287173272Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
sqlite-grafana-1        | logger=settings t=2023-04-10T12:22:20.288096267Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
sqlite-grafana-1        | logger=settings t=2023-04-10T12:22:20.288862264Z level=info msg="App mode production"
sqlite-grafana-1        | logger=sqlstore t=2023-04-10T12:22:20.306623179Z level=info msg="Connecting to DB" dbtype=sqlite3
sqlite-grafana-1        | logger=migrator t=2023-04-10T12:22:21.23584875Z level=info msg="Starting DB migrations"
sqlite-grafana-1        | logger=migrator t=2023-04-10T12:22:21.625210895Z level=info msg="migrations completed" performed=0 skipped=481 duration=184.423121ms
sqlite-grafana-1        | logger=plugin.finder t=2023-04-10T12:22:23.254275131Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled
sqlite-grafana-1        | logger=secrets t=2023-04-10T12:22:23.260041103Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
sqlite-grafana-1        | logger=query_data t=2023-04-10T12:22:23.618365395Z level=info msg="Query Service initialization"
sqlite-createbuckets-1 exited with code 0
sqlite-grafana-1        | logger=live.push_http t=2023-04-10T12:22:23.894026081Z level=info msg="Live Push Gateway initialization"
sqlite-grafana-1        | logger=infra.usagestats.collector t=2023-04-10T12:22:35.102585868Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2
sqlite-grafana-1        | logger=provisioning.datasources t=2023-04-10T12:22:35.157692268Z level=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" filename=/etc/grafana/provisioning/datasources/datasource.yml
sqlite-grafana-1        | logger=provisioning.alerting t=2023-04-10T12:22:35.449101672Z level=info msg="starting to provision alerting"
sqlite-grafana-1        | logger=provisioning.alerting t=2023-04-10T12:22:35.449879691Z level=info msg="finished to provision alerting"
sqlite-grafana-1        | logger=ngalert.state.manager t=2023-04-10T12:22:35.465129079Z level=info msg="Warming state cache for startup"
sqlite-grafana-1        | logger=grafanaStorageLogger t=2023-04-10T12:22:35.559340473Z level=info msg="storage starting"
sqlite-grafana-1        | logger=ngalert.state.manager t=2023-04-10T12:22:35.753192398Z level=info msg="State cache has been initialized" states=0 duration=288.053318ms
sqlite-grafana-1        | logger=ticker t=2023-04-10T12:22:35.756696487Z level=info msg=starting first_tick=2023-04-10T12:22:40Z
sqlite-grafana-1        | logger=ngalert.multiorg.alertmanager t=2023-04-10T12:22:35.800006587Z level=info msg="starting MultiOrg Alertmanager"
sqlite-grafana-1        | logger=http.server t=2023-04-10T12:22:35.876121521Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl= socket=
sqlite-app-1            | [00] go: downloading go.uber.org/automaxprocs v1.5.2
sqlite-app-1            | [00] go: downloading google.golang.org/grpc v1.54.0
sqlite-app-1            | [00] go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.15.2
sqlite-app-1            | [00] go: downloading google.golang.org/genproto v0.0.0-20230223222841-637eb2293923
sqlite-app-1            | [00] go: downloading google.golang.org/protobuf v1.30.0
sqlite-app-1            | [00] go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0
sqlite-app-1            | [00] go: downloading golang.org/x/net v0.9.0
sqlite-app-1            | [00] go: downloading github.com/hashicorp/raft v1.4.0
sqlite-app-1            | [00] go: downloading go.uber.org/multierr v1.11.0
sqlite-app-1            | [00] go: downloading github.com/golang/protobuf v1.5.3
sqlite-app-1            | [00] go: downloading github.com/armon/go-metrics v0.4.1
sqlite-app-1            | [00] go: downloading golang.org/x/sys v0.7.0
sqlite-app-1            | [00] go: downloading github.com/cespare/xxhash/v2 v2.2.0
sqlite-app-1            | [00] go: downloading golang.org/x/text v0.9.0
sqlite-app-1            | [00] # go.uber.org/multierr
sqlite-app-1            | [00] /go/pkg/mod/go.uber.org/[email protected]/error.go:209:13: undefined: atomic.Bool
sqlite-app-1            | [00] note: module requires Go 1.19
sqlite-app-1            | [00] # github.com/superfly/litefs
sqlite-app-1            | [00] /go/pkg/mod/github.com/superfly/[email protected]/db.go:1476:14: db.shmMu.TryLock undefined (type sync.Mutex has no field or method TryLock)
sqlite-app-1            | [00] /go/pkg/mod/github.com/superfly/[email protected]/store.go:918:34: undefined: any
sqlite-app-1            | [00] note: module requires Go 1.19
sqlite-grafana-1        | logger=context userId=1 orgId=1 uname=admin t=2023-04-10T12:27:25.042645591Z level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=10.89.0.4 time_ms=70 duration=70.086907ms size=0 referer= handler=/api/live/ws
sqlite-grafana-1        | logger=live t=2023-04-10T12:27:26.404472643Z level=info msg="Initialized channel handler" channel=grafana/dashboard/uid/7_VGtoLma address=grafana/dashboard/uid/7_VGtoLma
sqlite-app-1            | [00] (error exit: exit status 2)

Error executing command

Getting this error - panic: invalid type *ast.FuncType when I run sqlc-grpc -m "path"

Underscore in package name causes error

sqlc.yaml

version: "1"
packages:
  - path: "internal/package_name"

package_name instead of packagename causes an unfound error for unimplemented server.

Dunno if that's an sqlc issue or from here.

gogo protobufs deprecated

Hi,

gogo/protobuf has been deprecated back in 2022 and it would be good to move to std Protobuf v3, at this point.

You can see the announcement (and talk here)
https://x.com/awalterschulze/status/158455305610005708

regenerating after sql file update overwrites swagger options

Steps to reproduce

write sql query/schema
run sqlc generate
run sqlc-grpc -m "<path>"
edit swagger API details in proto/model/v1/model.proto to replace the placeholder entries
modify a sql file
per the instructions, run sqlc generate && go generate

The custom swagger details in proto/model/v1/model.proto are now reverted to the placeholder entries, rather than preserving our own details.

Masking SQLC queries in the resulting GRPC proto

When writing transactions, I write a few sqlc queries then write a custom RPC in the proto in order to run the queries as a transaction.

The issue I find is that all of these subqueries become externally accessible via the RPC because they all get added.

Have you thought of a way to encapsulate these types of internal queries so they aren't exported to the proto?

It would keep the proto cleaner and also means (as part of a larger team) that team members don't need to be aware of the hidden dangers of calling the wrong RPC individually when it was meant only to run as part of a larger transaction.

Proto reflect

This is really a nice database first approach .

do you think it’s possible to extend it to use proto reflect ?

It would essentially be pulling the dB meta data and holding it in memory and then using reflection at run time to provide a GRPC service

Outside docker ?

fuse probably means its linux only right ?

there is https://github.com/maxpert/marmot which also works that is 100% portable.

Its uses NATS which has similar properties as consul. 100% golang, no CGO.

speaking of CGO, we would also use non CGO sqlite. Its much slower currently...

Transactions

Is there a way to use transactions?

If you're familiar with gqlgen, the resolvers file is able to be editted, despite it being generated.

Would be good for the author service to have this functionality, or else have a "controller" that wraps the services that could have this functionality. Similar to the resolver pattern in graphql.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.