Giter VIP home page Giter VIP logo

gorsk's Introduction

GORSK - GO(lang) Restful Starter Kit

Build Status codecov Go Report Card Maintainability

Gorsk V2 is released - read more about it

Gorsk is a Golang starter kit for developing RESTful services. It is designed to help you kickstart your project, skipping the 'setting-up part' and jumping straight to writing business logic.

Previously Gorsk was built using Gin. Gorsk using Gin is available HERE.

Gorsk follows SOLID principles, with package design being inspired by several package designs, including Ben Johnson's Standard Package Layout, Go Standard Package Layout with my own ideas applied to both. The idea for building this project and its readme structure was inspired by this.

This starter kit currently provides:

  • Fully featured RESTful endpoints for authentication, changing password and CRUD operations on the user entity
  • JWT authentication and session
  • Application configuration via config file (yaml)
  • RBAC (role-based access control)
  • Structured logging
  • Great performance
  • Request marshaling and data validation
  • API Docs using SwaggerUI
  • Mocking using stdlib
  • Complete test coverage
  • Containerized database query tests

The following dependencies are used in this project (generated using Glice):

|-------------------------------------|--------------------------------------------|--------------|
|             DEPENDENCY              |                  REPOURL                   |   LICENSE    |
|-------------------------------------|--------------------------------------------|--------------|
| github.com/labstack/echo            | https://github.com/labstack/echo           | MIT          |
| github.com/go-pg/pg                 | https://github.com/go-pg/pg                | bsd-2-clause |
| github.com/dgrijalva/jwt-go         | https://github.com/dgrijalva/jwt-go        | MIT          |
| github.com/rs/zerolog               | https://github.com/rs/zerolog              | MIT          |
| golang.org/x/crypto/bcrypt          | https://github.com/golang/crypto           |              |
| gopkg.in/yaml.v2                    | https://github.com/go-yaml/yaml            |              |
| gopkg.in/go-playground/validator.v8 | https://github.com/go-playground/validator | MIT          |
| github.com/lib/pq                   | https://github.com/lib/pq                  | Other        |
| github.com/nbutton23/zxcvbn-go      | https://github.com/nbutton23/zxcvbn-go     | MIT          |
| github.com/fortytw2/dockertest      | https://github.com/fortytw2/dockertest     | MIT          |
| github.com/stretchr/testify         | https://github.com/stretchr/testify        | Other        |
|-------------------------------------|--------------------------------------------|--------------|
  1. Echo - HTTP 'framework'.
  2. Go-Pg - PostgreSQL ORM
  3. JWT-Go - JWT Authentication
  4. Zerolog - Structured logging
  5. Bcrypt - Password hashing
  6. Yaml - Unmarshalling YAML config file
  7. Validator - Request validation.
  8. lib/pq - PostgreSQL driver
  9. zxcvbn-go - Password strength checker
  10. DockerTest - Testing database queries
  11. Testify/Assert - Asserting test results

Most of these can easily be replaced with your own choices since their usage is abstracted and localized.

Getting started

Using Gorsk requires having Go 1.7 or above. Once you downloaded Gorsk (either using Git or go get) you need to configure the following:

  1. To use Gorsk as a starting point of a real project whose package name is something like github.com/author/project, move the directory $GOPATH/github.com/ribice/gorsk to $GOPATH/github.com/author/project. We use task as a task runner. Run task relocate to rename the packages, e.g. task relocate TARGET=github.com/author/project

  2. Change the configuration file according to your needs, or create a new one.

  3. Set the ("ENVIRONMENT_NAME") environment variable, either using terminal or os.Setenv("ENVIRONMENT_NAME","dev").

  4. Set the JWT secret env var ("JWT_SECRET")

  5. In cmd/migration/main.go set up psn variable and then run it (go run main.go). It will create all tables, and necessery data, with a new account username/password admin/admin.

  6. Run the app using:

go run cmd/api/main.go

The application runs as an HTTP server at port 8080. It provides the following RESTful endpoints:

  • POST /login: accepts username/passwords and returns jwt token and refresh token
  • GET /refresh/:token: refreshes sessions and returns jwt token
  • GET /me: returns info about currently logged in user
  • GET /swaggerui/ (with trailing slash): launches swaggerui in browser
  • GET /v1/users: returns list of users
  • GET /v1/users/:id: returns single user
  • POST /v1/users: creates a new user
  • PATCH /v1/password/:id: changes password for a user
  • DELETE /v1/users/:id: deletes a user

You can log in as admin to the application by sending a post request to localhost:8080/login with username admin and password admin in JSON body.

Implementing CRUD of another table

Let's say you have a table named 'cars' that handles employee's cars. To implement CRUD on this table you need:

  1. Inside pkg/utl/model create a new file named car.go. Inside put your entity (struct), and methods on the struct if you need them.

  2. Create a new car folder in the (micro)service where your service will be located, most probably inside api. Inside create a file/service named car.go and test file for it (car/car.go and car/car_test.go). You can test your code without writing a single query by mocking the database logic inside /mock/mockdb folder. If you have complex queries interfering with other entities, you can create in this folder other files such as car_users.go or car_templates.go for example.

  3. Inside car folder, create folders named platform, transport and logging.

  4. Code for interacting with a platform like database (postgresql) should be placed under car/platform/pgsql. (pkg/api/car/platform/pgsql/car.go)

  5. In pkg/api/car/transport create a new file named http.go. This is where your handlers are located. Under the same location create http_test.go to test your API.

  6. In logging directory create a file named car.go and copy the logic from another service. This serves as request/response logging.

  7. In pkg/api/api.go wire up all the logic, by instantiating car service, passing it to the logging and transport service afterwards.

Implementing other platforms

Similarly to implementing APIs relying only on a database, you can implement other platforms by:

  1. In the service package, in car.go add interface that corresponds to the platform, for example, Indexer or Reporter.

  2. Rest of the procedure is same, except that in /platform you would create a new folder for your platform, for example, elastic.

  3. Once the new platform logic is implemented, create an instance of it in main.go (for example elastic.Client) and pass it as an argument to car service (pkg/api/car/car.go).

Running database queries in transaction

To use a transaction, before interacting with db create a new transaction:

err := s.db.RunInTransaction(func (tx *pg.Tx) error{
    // Application service here
})

Instead of passing database client as s.db , inside this function pass it as tx. Handle the error accordingly.

Project Structure

  1. Root directory contains things not related to code directly, e.g. docker-compose, CI/CD, readme, bash scripts etc. It should also contain vendor folder, Gopkg.toml and Gopkg.lock if dep is being used.

  2. Cmd package contains code for starting applications (main packages). The directory name for each application should match the name of the executable you want to have. Gorsk is structured as a monolith application but can be easily restructured to contain multiple microservices. An application may produce multiple binaries, therefore Gorsk uses the Go convention of placing main package as a subdirectory of the cmd package. As an example, scheduler application's binary would be located under cmd/cron. It also loads the necessery configuration and passes it to the service initializers.

  3. Rest of the code is located under /pkg. The pkg directory contains utl and 'microservice' directories.

  4. Microservice directories, like api (naming corresponds to cmd/ folder naming) contains multiple folders for each domain it interacts with, for example: user, car, appointment etc.

  5. Domain directories, like user, contain all application/business logic and two additional directories: platform and transport.

  6. Platform folder contains various packages that provide support for things like databases, authentication or even marshaling. Most of the packages located under platform are decoupled by using interfaces. Every platform has its own package, for example, postgres, elastic, redis, memcache etc.

  7. Transport package contains HTTP handlers. The package receives the requests, marshals, validates then passes it to the corresponding service.

  8. Utl directory contains helper packages and models. Packages such as mock, middleware, configuration, server are located here.

License

gorsk is licensed under the MIT license. Check the LICENSE file for details.

Author

Emir Ribic

gorsk's People

Contributors

dependabot[bot] avatar dougwinsby avatar elvischiang avatar harund avatar julienkosinski avatar labzy88 avatar michael2008s avatar nehagup avatar noartem avatar priom avatar ribice avatar slayerjain avatar wingkwong avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gorsk's Issues

Design question: Your various application services..

... have methods that takes in echo.Context as one of its parameters:

func (u User) Create(c echo.Context, req gorsk.User) (gorsk.User, error) {

func (a Auth) Authenticate(c echo.Context, user, pass string) (gorsk.AuthToken, error) {

func (p Password) Change(c echo.Context, userID int, oldPass, newPass string) error {

I noticed the same trend in gorsk-gin:
here

among others..

Why do you have it as a parameter if it is never used within the methods? Is its inclusion a placeholder for edge use cases?

Thanks for publishing this code. I find it immensely useful to think about how to go about structuring API in Go.

Docker and Go modules

  • I feel adding docker and docker-compose setup will make local development way easier and could replace the replace-all package steps while getting started.
  • Also, how about using Go modules?

A makefile target can also be used instead of manual package name change. See this.

Would be happy to submit PRs if above sound useful.

P.S.: Nice work on v2 πŸ‘

Move config test files from files directory

When testing config.go, config.invalid.yml and config.testdata.yml are being used.

They are located at same directory as other config files, which should not be the case due to clutter. They need to be moved to separate folder, e.g. testdata.

microservice and docker

Hi, please introduce my name is Naray Citra from Indonesia. I learned Golang since 2 weeks ago, same as you my programming experience is Java with SpringBoot :). I found your started kit by searching google :). I like your stater kit, but i still newbie in Golang, i have some questions for you. How we separate the pkg into microservice and containerized it? Can you give an example for that?

Use testify for testing

While verifying test results using standard library only, I felt that it was way too verbose and the output wasn't as nice.

https://github.com/stretchr/testify provides assert method for verifying test results, which is written in a single line and writes a meaningful output to stdout.

I did not want to add another dependency, but this one feel justifiable.

EnforceLocation should query for location's company

When enforcing certain location, currently all users with role company_admin can do the change.

In order to EnforceLocation, company_id needs to be retrieved (from locations table) and then matched if user is company_admin.

Add security middleware

What comes to my mind is rate limiting (link) and other security related middleware (e.g. XSS, DNSPrefetchControl etc.)

Most of them are available in Helmet package, but it will be better to write them manually after #2.

Rename the Packages

I tried to rename but failed, something wrong?
task relocate TARGET=github.com/author/gorsk

task relocate TARGET=github.com/author/gorsk                                                                                                                                               ─╯
 *** Relocating packages to github.com\/author\/gorsk ***
sed: can't read s/gorsk/gorsk/g: No such file or directory
sed: can't read s/gorsk/gorsk/g: No such file or directory
sed: can't read s/gorsk/gorsk/g: No such file or directory
sed: can't read s/gorsk/gorsk/g: No such file or directory
sed: can't read s/gorsk/gorsk/g: No such file or directory
sed: can't read s/gorsk/gorsk/g: No such file or directory
sed: can't read s/gorsk/gorsk/g: No such file or directory
....

Handle sessions using Redis

Currently payload is stored in JWT Claims, and in middleware it is then passed to context.

I don't really like this implementation. Also by removing it the project will be less dependant on Gin.

Token should have only expiry info and be used for authentication. For authorization purposes and getting user info, details should be fetched from Redis with JWT token as key.

Additionally this will require use of docker-compose, for running Redis.

Improve database migration

Initially I added a .sql file for database migration. I didn't like the fact that it had to be executed manually, so I added another executable located at cmd/migration.

Currently it creates necessary tables and data, without any constraints or relations.

go-pg has support for these (CreateTableOptions).

This also needs to be changed for integration tests.

Improve tests

Database tests needs to be improved.

UserDB Update tests updates all columns, instead of only mentioned ones. Manually works fine.

Tests on structs package should be completed.

Improve logging

Gorsk currently only logs DB errors.

Perhaps logging middleware should be added with requestID generation, being passed to logger.

Also requests/responses could be logged.

Error use of internal package not allowed

Trying to run go run cmd/api/main.go, but get problem with "internal package not allowed"
I clone project under $GOPATH/src, then

=> ~/go/bin/dep ensure -v
=> go run cmd/api/main.go
cmd/api/main.go:44:2: use of internal package not allowed
cmd/api/main.go:45:2: use of internal package not allowed
cmd/api/main.go:36:2: use of internal package not allowed
cmd/api/main.go:46:2: use of internal package not allowed
cmd/api/main.go:47:2: use of internal package not allowed 

After removed github.com/ribice from import in all go files, so

import (
        ...
	"gorsk/internal"
        ...
)

Everything goes well.

Viper for configuration management

How about using https://github.com/spf13/viper for configuration management? We won't be restricted to YAML files for configurations.

Would be happy to contribute if this looks like a suitable enhancement.

P.S.: Loved gorsk. Saved me a lot of effort in standardising things. πŸ‘ :)

Questions about routes

I think this would be more logical to have routes like below because user should be singular when manipulating only one entry don't you think?

POST /login: accepts username/passwords and returns jwt token and refresh token
GET /refresh/:token: refreshes sessions and returns jwt token
GET /me: returns info about currently logged in user
GET /swaggerui/: launches swaggerui in browser
GET /v1/users: returns list of users
GET /v1/user/:id: returns single user
POST /v1/user: creates a new user
PATCH /v1/user/:id/password: changes password for a user
DELETE /v1/user/:id: deletes a user

Also, I'm wondering about route if version should be on every routes or not.

Improve get queries

Currently select queries are retrieved using:

var entity model.User{} 
db.Model(&entity).Where("id = ?", id).Select()

It should be replaced with

entity := model.User{ID: id}
db.Select(&entity)

Using Gorsk

Hi Mr Ribice,
My name is Mouhamed, I’m a software engineer and I’m new to Golang.
I decide to use it to build Restful API, and after few research on which framework to use, I found your GitHub repository about Gorsk. For me it’s an amazing work and I decided to use it.
But can I have more details about the workflow on gorsk for example the migrations. What is the best way to write them ? In the main file or create differences packages inside ? (I come from Django rest framework so please understand :))
Also can I have a project sample to explore before starting and doing messy things :).

swagger.json doesn't sync with code

Reproduce

By using latest go-swagger

swagger generate spec -o assets/swaggerui/swagger.json --scan-models -b github.com/ribice/gorsk/pkg/api

check assets/swaggerui/swagger.json

Keep track on user actions

My problem might be basic, I'm new to go, sorry.

I'd like to add in the database multiple common field defined in pkg/utl/model/model.go like that:

// Base contains common fields for all tables
type Base struct {
	ID        int       `json:"id"`
	CreatedAt time.Time `json:"created_at"`
	UpdatedAt time.Time `json:"updated_at"`
	DeletedAt time.Time `json:"deleted_at,omitempty" pg:",soft_delete"`
	CreatedBy int `json:"created_by"`
	UpdatedBy int `json:"updated_by"`
	DeletedBy int `json:"deleted_by,omitempty"`
}

The idea is fill the fields with the ID of the authenticated user like it fills times with the use of go-pg hooks functions.

So still in pkg/utl/model/model.go I'd like to be able to do something like:

// BeforeInsert hooks into insert operations, setting createdAt and updatedAt to current time
func (b *Base) BeforeInsert(_ orm.DB) error {
	now := time.Now()
	b.CreatedAt = now
	b.UpdatedAt = now

	b.CreatedByID = contextUserID
	b.UpdatedByID = contextUserID

	return nil
}

// BeforeUpdate hooks into update operations, setting updatedAt to current time
func (b *Base) BeforeUpdate(_ orm.DB) error {
	b.UpdatedAt = time.Now()

	b.UpdatedByID = contextUserID

	return nil
}

// BeforeUpdate hooks into delete operations
func (b *Base) BeforeDelete(_ orm.DB) error {
	
	b.DeletedByID = contextUserID

	return nil
}

In this file how can I access echo.Context.Get("id") to fill contextUserID?

(Merry Christmas ;))

Readme inconsistency: Model files at root level vs pkg/utl/model

README references pkg/utl/model folder, but it seems such files were moved to root level in d89f430

I was wondering if this (having the files at root level) is generally the preferred approach, or if it's better to use the folder as suggested in the readme?

Inside pkg/utl/model create a new file named car.go. Inside put your entity (struct), and methods on the struct if you need them.

ENVIRONMENT_NAME not found in the code

Hi

I cannot see where ENVIRONMENT_NAME is use id. I can see JWT_SECRET user as well as "psn" (DATABASE_URL) in the code
it seem setting ENVIRONMENT_NAME to "dev" it is not working.

I am having certificate error like I am in PROD.
go: downloading github.com/segmentio/encoding v0.1.10
pkg/api/auth/auth.go:8:2: found packages auth (auth.go) and gorsk (company.go) in /home/go/github.com/buscando/gocar
../../../pkg/mod/github.com/go-pg/pg/[email protected]/pgjson/json.go:7:2: github.com/segmentio/[email protected]:
Get "https://storage.googleapis.com/proxy-golang-org-prod/ xxx": x509: certificate signed by unknown authority

Any help is appreciated

Thank you
Marcelo

The /swaggerui endpoint returns a 404 Not Found Error

I have setup the server according to the docs and running it normally via: go run cmd/api/main.go. Requesting the /swaggerui endpoint returns the following JSON error message:

$ curl http://localhost:8080/swaggerui/ 
"code=404, message=Not Found"

The error occurs regardless of the trailing / character in the URL.

Replace Gin with Echo

I feel Echo is superior to Gin in many ways.

  1. Echo's handler returns an error, which is much nicer compared to empty returns.
  2. Echo's documentation is far superior to Gin's README
  3. More active development - Issues on Gin barely have any replies.

It would be easier to move to Echo once #1 is resolved.

Swagger docs are not being generated

Due to 'bug' in go-swagger, swagger docs are currently not being generated.

The main reason is usage /golang.org/x/crypto package, which has Linter/Code issues. Go-swagger complains with error:

swagger generate spec -o ./swagger.json --scan-models
/usr/local/go/src/crypto/x509/root_cgo_darwin.go:210:30: cannot convert nil (untyped nil value) to _Ctype_CFDataRef
/usr/local/go/src/crypto/x509/root_cgo_darwin.go:211:39: cannot convert nil (untyped nil value) to _Ctype_CFDataRef
/usr/local/go/src/crypto/x509/root_cgo_darwin.go:221:22: cannot convert nil (untyped nil value) to _Ctype_CFDataRef
/Users/ribice/go/src/github.com/ugorji/go/codec/goversion_makemap_gte_go19.go:12:10: MakeMapWithSize not declared by package reflect
/Users/ribice/go/src/github.com/ugorji/go/codec/goversion_makemap_gte_go19.go:14:9: MakeMapWithSize not declared by package reflect
couldn't load packages due to errors: github.com/ugorji/go/codec, crypto/x509

Per author's words:

I think it migh tmake it consider less packages
otherwise the error is probably a setting here: https://github.com/go-swagger/go-swagger/blob/master/scan/scanner.go#L205-L216
or a bug here: https://github.com/go-swagger/go-swagger/blob/master/scan/classifier.go#L72:30

Also I installed go-swagger using brew, which downloads latest release (6 months old). Perhaps download from master using go-get and running the binary would solve the issue.

Authorization header does not work with Bearer keyword

Currently authorization token is passed as "Authorization:$token", as I had some issues with Bearer keyword and jwt-go library.

Will need to fix this, but I'm curious are there any drawbacks of using it like this (no Bearer keyword)?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.