Giter VIP home page Giter VIP logo

golang-http-template's Introduction

OpenFaaS Golang HTTP templates

This repository contains two Golang templates for OpenFaaS which give additional control over the HTTP request and response. They will both handle higher throughput than the classic watchdog due to the process being kept warm.

Our recommended template for Go developers is golang-middleware.

You'll find a chapter dedicated to writing functions with Go in Everyday Golang by Alex Ellis

Using the templates:

faas-cli template store pull golang-http
faas-cli template store pull golang-middleware

Or:

$ faas template pull https://github.com/openfaas/golang-http-template
$ faas new --list

Languages available as templates:
- golang-http
- golang-middleware

The two templates are very similar:

  • golang-middleware implements a http.HandleFunc from Go's stdlib.
  • golang-http uses a structured request/response object

Dependencies

You can manage dependencies in one of the following ways:

  • To use Go modules without vendoring, the default already is set GO111MODULE=on but you also can make that explicit by adding --build-arg GO111MODULE=on to faas-cli up, you can also use --build-arg GOPROXY=https:// if you want to use your own mirror for the modules
  • You can also Go modules with vendoring, run go mod vendor in your function folder and add --build-arg GO111MODULE=off --build-arg GOFLAGS='-mod=vendor' to faas-cli up
  • If you have a private module dependency, we recommend using the vendoring technique from above.

SSH authentication for private Git repositories and modules

If you do not wish to, or cannot use vendoring for some reason, then we provide an alternative set of templates for OpenFaaS Pro customers:

1.0 golang-middleware (recommended template)

This is one of the fastest templates available for Go available. Its signature is a http.HandlerFunc, instead of a traditional request and response that you may expect from a function.

The user has complete control over the HTTP request and response.

Get the template

$ faas template store pull golang-middleware

# Or

$ faas template pull https://github.com/openfaas/golang-http-template
$ faas new --lang golang-middleware <fn-name>

Example usage

Example writing a JSON response:

package function

import (
	"encoding/json"
	"fmt"
	"io"
	"net/http"
	"os"
)

func Handle(w http.ResponseWriter, r *http.Request) {
	var input []byte

	if r.Body != nil {
		defer r.Body.Close()

		// read request payload
		reqBody, err := io.ReadAll(r.Body)

		if err != nil {
			http.Error(w, err.Error(), http.StatusInternalServerError)
			return

		input = reqBody
		}
	}

	// log to stdout
	fmt.Printf("request body: %s", string(input))

	response := struct {
		Payload     string              `json:"payload"`
		Headers     map[string][]string `json:"headers"`
		Environment []string            `json:"environment"`
	}{
		Payload:     string(input),
		Headers:     r.Header,
		Environment: os.Environ(),
	}

	resBody, err := json.Marshal(response)
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

    // write result
	w.WriteHeader(http.StatusOK)
	w.Write(resBody)
}

Example persistent database connection pool between function calls:

package function

import (
	"database/sql"
	"fmt"
	"io"
	"net/http"
	"strings"
	_ "github.com/go-sql-driver/mysql"
)

// db pool shared between function calls
var db *sql.DB

func init() {
	var err error
	db, err = sql.Open("mysql", "user:password@/dbname")
	if err != nil {
		panic(err.Error())
	}

	err = db.Ping()
	if err != nil {
		panic(err.Error())
	}
}

func Handle(w http.ResponseWriter, r *http.Request) {
	var query string
	ctx := r.Context()

	if r.Body != nil {
		defer r.Body.Close()

		// read request payload
		body, err := io.ReadAll(r.Body)

		if err != nil {
			http.Error(w, err.Error(), http.StatusInternalServerError)
			return
		}

		query = string(body)
	}

	// log to stdout
	fmt.Printf("Executing query: %s", query)

	rows, err := db.QueryContext(ctx, query)
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}
	defer rows.Close()

	ids := make([]string, 0)
	for rows.Next() {
		if e := ctx.Err(); e != nil {
			http.Error(w, e, http.StatusBadRequest)
			return
		}
		var id int
		if err := rows.Scan(&id); err != nil {
			http.Error(w, err.Error(), http.StatusInternalServerError)
			return
		}
		ids = append(ids, string(id))
	}
	if err := rows.Err(); err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

	result := fmt.Sprintf("ids %s", strings.Join(ids, ", "))

	// write result
	w.WriteHeader(http.StatusOK)
	w.Write([]byte(result))
}

Example retrieving request query strings

package function
import (
	"fmt"
	"net/http"
)
func Handle(w http.ResponseWriter, r *http.Request) {
	// Parses RawQuery and returns the corresponding
	// values as a map[string][]string
	// for more info https://golang.org/pkg/net/url/#URL.Query
	query := r.URL.Query()
	w.WriteHeader(http.StatusOK)
	w.Write([]byte(fmt.Sprintf("id: %s", query.Get("id"))))
}

Adding static files to your image

If a folder named static is found in the root of your function's source code, it will be copied into the final image published for your function.

To read this back at runtime, you can do the following:

package function

import (
    "net/http"
    "os"
)

func Handle(w http.ResponseWriter, r *http.Request) {

    data, err := os.ReadFile("./static/file.txt")

    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
    }

    w.Write(data)
}

2.0 golang-http

This template provides additional context and control over the HTTP response from your function.

Status of the template

Like the golang-middleware template, this template is highly performant and suitable for production.

Get the template

$ faas template store pull golang-http

# Or
$ faas template pull https://github.com/openfaas/golang-http-template
$ faas new --lang golang-http <fn-name>

Example usage

Example writing a successful message:

package function

import (
	"fmt"
	"net/http"

		handler "github.com/openfaas/templates-sdk/go-http"
)

// Handle a function invocation
func Handle(req handler.Request) (handler.Response, error) {
	var err error

	message := fmt.Sprintf("Hello world, input was: %s", string(req.Body))

	return handler.Response{
		Body:       []byte(message),
    }, err
}

Example writing a custom status code

package function

import (
	"fmt"
	"net/http"

		handler "github.com/openfaas/templates-sdk/go-http"
)

// Handle a function invocation
func Handle(req handler.Request) (handler.Response, error) {
	var err error

	return handler.Response{
		Body:       []byte("Your workload was accepted"),
		StatusCode: http.StatusAccepted,
	}, err
}

Example writing an error / failure.

package function

import (
	"fmt"
	"net/http"

		handler "github.com/openfaas/templates-sdk/go-http"
)

// Handle a function invocation
func Handle(req handler.Request) (handler.Response, error) {
	var err error

	return handler.Response{
        Body: []byte("the input was invalid")
	}, fmt.Errorf("invalid input")
}

The error will be logged to stderr and the body will be written to the client along with a HTTP 500 status code.

Example reading a header.

package function

import (
	"log"

		handler "github.com/openfaas/templates-sdk/go-http"
)

// Handle a function invocation
func Handle(req handler.Request) (handler.Response, error) {
	var err error

	log.Println(req.Header) // Check function logs for the request headers

	return handler.Response{
		Body: []byte("This is the response"),
		Header: map[string][]string{
			"X-Served-By": []string{"My Awesome Function"},
		},
	}, err
}

Example responding to an aborted request.

The Request object provides access to the request context. This allows you to check if the request has been cancelled by using the context's done channel req.Context().Done() or the context's error req.Context().Err()

package function

import (
	"fmt"
	"net/http"

	handler "github.com/openfaas/templates-sdk/go-http"
)

// Handle a function invocation
func Handle(req handler.Request) (handler.Response, error) {
	var err error

	for i := 0; i < 10000; i++ {
		if req.Context().Err() != nil  {
			return handler.Response{}, fmt.Errorf("request cancelled")
		}
		fmt.Printf("count %d\n", i)
	}

	message := fmt.Sprintf("Hello world, input was: %s", string(req.Body))
	return handler.Response{
		Body:       []byte(message),
		StatusCode: http.StatusOK,
	}, err
}

This context can also be passed to other methods so that they can respond to the cancellation as well, for example db.ExecContext(req.Context())

Advanced usage

Sub-packages

It is often natural to organize your code into sub-packages, for example you may have a function with the following structure

./echo
├── go.mod
├── go.sum
├── handler.go
└── pkg
    └── version
        └── version.go

Now if you want to reference theversion sub-package, import it as

import "handler/function/pkg/version"

This works like any local Go project.

Go sub-modules

Sub-modules (meaning sub-folders with a go.mod) are not supported.

golang-http-template's People

Contributors

alexellis avatar amalkh5 avatar bmcustodio avatar cconger avatar cpanato avatar ewilde avatar koeng101 avatar lucasroesler avatar marcusnoble avatar martindekov avatar mrwormhole avatar nitishkumar71 avatar omrishtam avatar rdimitrov avatar stefanprodan avatar utsavanand2 avatar viveksyngh avatar welteki avatar zbblanton avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

golang-http-template's Issues

Cannot publish Function after updating template

Needed to make use of Go 1.17 modules. Followed guide to upgrade the template:

Inside of func folder did the following:

rm -rf template/golang-http template/golang-middleware

Install new templates
faas-cli template store pull golang-http

Tidy modules
go mod tidy

Try to publish function and get the following error:

#24 1.350 handler.go:17:2: ambiguous import: found package github.com/openfaas/templates-sdk/go-http in multiple modules:
#24 1.350 	github.com/openfaas/templates-sdk v0.0.0-20200723110415-a699ec277c12 (/go/pkg/mod/github.com/openfaas/[email protected]/go-http)
#24 1.350 	github.com/openfaas/templates-sdk/go-http v0.0.0-20220408082716-5981c545cb03 (/go/pkg/mod/github.com/openfaas/templates-sdk/[email protected])
#24 ERROR: process "/bin/sh -c GOOS=${TARGETOS} GOARCH=${TARGETARCH} go test ./... -cover" did not complete successfully: exit code: 1

Expected Behaviour

Upgrade template and publish app

Your Environment

  • FaaS-CLI version ( Full output from: faas-cli version ):

  • Docker version docker version (e.g. Docker 17.0.05 ):
    Client:
    Cloud integration: v1.0.22
    Version: 20.10.12
    API version: 1.41
    Go version: go1.16.12
    Git commit: e91ed57
    Built: Mon Dec 13 11:46:56 2021
    OS/Arch: darwin/arm64
    Context: default
    Experimental: true

Server: Docker Engine - Community
Engine:
Version: 20.10.12
API version: 1.41 (minimum version 1.12)
Go version: go1.16.12
Git commit: 459d0df
Built: Mon Dec 13 11:43:07 2021
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.4.12
GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc:
Version: 1.0.2
GitCommit: v1.0.2-0-g52b36a2
docker-init:
Version: 0.19.0
GitCommit: de40ad0

  • Are you using Docker Swarm or Kubernetes (FaaS-netes)?
    K8s
  • Operating System and version (e.g. Linux, Windows, MacOS):
    MacOS
  • Code example or link to GitHub repo or gist to reproduce problem:
    NA

Unable to use vendoring with golang-http

Why do you need this?

I need to be able to use vendoring with the golang-http template because it is how I get private Go code into my functions.

Expected Behaviour

This should work without error, when using a vendor folder and any other stack.yml flags or build-args required:

faas-cli build

Current Behaviour

Step 29/42 : RUN ${GO} build --ldflags "-s -w" -a -installsuffix cgo -o handler .
 ---> Running in b9e5d187d314
Setting vendor mode env variables
main.go:18:2: cannot find package "github.com/openfaas/templates-sdk/go-http" in any of:
        /usr/local/go/src/github.com/openfaas/templates-sdk/go-http (from $GOROOT)
        /go/src/github.com/openfaas/templates-sdk/go-http (from $GOPATH)
The command '/bin/sh -c ${GO} build --ldflags "-s -w" -a -installsuffix cgo -o handler .' returned a non-zero code: 1
[0] < Building go1 done in 5.91s.
[0] Worker done.

Total build time: 5.91s
Errors received during build:
- [go1] received non-zero exit code from build, error: The command '/bin/sh -c ${GO} build --ldflags "-s -w" -a -installsuffix cgo -o handler .' returned a non-zero code: 1

List All Possible Solutions and Workarounds

  1. Create a new separate template golang-http-vendoring that only works with vendoring
  2. Stop maintaining a golang-http template at all, migrate everyone to golang-middleware
  3. Use interfaces instead of structs, so that the main.go module doesn't have to reference the SDK, which seems to be the source of the problem - full example, and it's a breaking change: https://github.com/alexellis/golang-http-template
  4. Consider a new direction that looks like AWS Lambda's Go handler - https://docs.aws.amazon.com/lambda/latest/dg/golang-handler.html https://docs.aws.amazon.com/lambda/latest/dg/golang-handler.html
  5. Use Go's plugin capability, to build the function and entry point separately, and link them after the fact. Unfortunately this seems to require CGO and breaks portability - golang/go#19569

For 4), the function's handler owns the main() entrypoint instead of the hidden underlying code, and calls a utility method to set up the HTTP server etc.

package main

import (
        "fmt"
        "context"
        "github.com/openfaas/golang-http/entrypoint"
)

type MyEvent struct {
        Name string `json:"name"`
}

func HandleRequest(ctx context.Context, name MyEvent) (string, error) {
        return fmt.Sprintf("Hello %s!", name.Name ), nil
}

func main() {
        entrypoint.Start(HandleRequest)
}

For 3), the code would end up looking like this for a handler:

(this can be compared to the second code sample to see what's different)

package function

import (
    "fmt"
    "net/http"

    gohttp "github.com/alexellis/templates-sdk/go-http"
    logrus "github.com/sirupsen/logrus"
)

// Handle a function invocation
func Handle(req gohttp.Request) (gohttp.Response, error) {
    var err error

    logrus.Info("Value of body: ", string(req.GetBody()))

    message := fmt.Sprintf("Body: %s", string(req.GetBody()))

    return &gohttp.FunctionResponse{
        Body:       []byte(message),
        StatusCode: http.StatusOK,
    }, err
}

What is your preferred solution?

I like 3) and to lessen its impact to existing users, we could consider promoting golang-http into the primary community templates repository and deprecating the one we use here.

Alternatively, we could call it "go-http1.18orgolang-http1.18as a new template, withgolang-http1.19` and so on coming after that.

Steps to Reproduce (for bugs)

faas-cli template store pull golang-http-template
faas-cli new --lang golang-http  --prefix alexellis2 withlogrus

Edit withlogrus/handler.go:

package function

import (
	"fmt"
	"io"
	"net/http"

	"github.com/sirupsen/logrus"
)

func Handle(w http.ResponseWriter, r *http.Request) {
	var input []byte

	if r.Body != nil {
		defer r.Body.Close()

		body, _ := io.ReadAll(r.Body)

		input = body
	}

	logrus.Debug(fmt.Sprintf("Handle; %q", input))

	w.WriteHeader(http.StatusOK)
	w.Write([]byte(fmt.Sprintf("Body: %q", string(input))))
}

Then edit withlogrus.yml, adding:

    build_args:
      GO111MODULE: off
      GOFLAGS: "-mod=vendor"

Then try a build:

cd golang-http-template

go get
go mod vendor

cd ..
faas-cli build -f withlogrus.yml

Cannot build with go modules

My actions before raising this issue

I would like to use go modules in my openfaas function.

Expected Behaviour

I whould be able to build my function by enabling GO111MODULE.

Current Behaviour

Building fail with this error:

handler.go:8:2: no required module provides package golang.org/x/example/stringutil; to add it:
        go get golang.org/x/example/stringutil

Steps to Reproduce (for bugs)

We are going to build a function with "golang.org/x/example/stringutil" module

  1. faas new --lang=golang-middleware test-module
  2. Edit test-module/handler.go
package function

import (
	"fmt"
	"io/ioutil"
	"net/http"

	"golang.org/x/example/stringutil"
)

func Handle(w http.ResponseWriter, r *http.Request) {
	var input []byte

	if r.Body != nil {
		defer r.Body.Close()

		body, _ := ioutil.ReadAll(r.Body)

		input = body
	}

	w.WriteHeader(http.StatusOK)
	w.Write([]byte(stringutil.Reverse("!selpmaxe oG ,olleH")))
}
  1. faas-cli build -f test-module.yml
Step 21/36 : RUN GOOS=${TARGETOS} GOARCH=${TARGETARCH} go test ./... -cover
 ---> Running in e673616d5614
handler.go:8:2: no required module provides package golang.org/x/example/stringutil; to add it:
        go get golang.org/x/example/stringutil
The command '/bin/sh -c GOOS=${TARGETOS} GOARCH=${TARGETARCH} go test ./... -cover' returned a non-zero code: 1

Context

Pulled latest golang-middleware template.

GO111MODULE is now enabled by default.

I tried with build-args option on the cli and in test-module.yml without success.

Your Environment

  • FaaS-CLI version ( Full output from: faas-cli version ):

CLI:
commit: 72816d486cf76c3089b915dfb0b66b85cf096634
version: 0.13.13

  • Docker version docker version (e.g. Docker 17.0.05 ): 20.10.12

  • Are you using Docker Swarm or Kubernetes (FaaS-netes)? Irrelevant

  • Operating System and version (e.g. Linux, Windows, MacOS): Linux

  • Code example or link to GitHub repo or gist to reproduce problem: Irrelevant

  • Other diagnostic information / logs from troubleshooting guide

Next steps

You may join Slack for community support.

Replace ioutil

My actions before raising this issue

Any possibility to replace ioutil with non deprecated method? I can just use io package which is the most direct & explicit way after 1.16v, I can also see some variable naming a bit annoying like bErr, cErr, dErr instead of being just err

I am also a bit confused about having request and response OpenFaaS model in a separate repository over here: github.com/openfaas/templates-sdk/go-http

Is there a reason why it is there but not in this repository? The files seem very minimal to have it somewhere else. For the maintainability sake, I would pull that repo's go-http here and delete this openfaas/templates-sdk dependecy for Go templates

Expected Behaviour

No behaviour change, just a maintainability and a convention

Current Behaviour

No behaviour change, just a maintainability and a convention

Possible Solution

No solution needed, tidy up task

Steps to Reproduce (for bugs)

N/A

Context

N/A

Your Environment

  • FaaS-CLI version ( Full output from: faas-cli version ):

  • Docker version docker version (e.g. Docker 17.0.05 ):

  • Are you using Docker Swarm or Kubernetes (FaaS-netes)?
    No

  • Operating System and version (e.g. Linux, Windows, MacOS):
    Linux

  • Code example or link to GitHub repo or gist to reproduce problem:
    N/A

  • Other diagnostic information / logs from troubleshooting guide

Next steps

You may join Slack for community support.

go-http template query string not available in function

Summary

In the go-http template headers, body and query string are copied over to the request used to call the function, however the query string is not.

https://github.com/openfaas-incubator/golang-http-template/blob/7bce4849a2657e94d00b0411f2da3ae8a902f3ff/template/golang-http-armhf/main.go#L33-L39

Proposal

  1. Copy query string

or

  1. Document in the template why this is not desired

Why?

As a function developer, I found it confusing that the query string was blank, in order to implement my business code I had to use the alternative template golang-middleware. Also took some time debugging the template code and function code in order to discover this.

My preference would be to copy query string. Happy to pick this up once design is approved

Timeout should match value in read/write_timeout

Expected

Timeout value in main.go should match value in read/write_timeout env-vars (these are Golang durations)

Actual

Hard-coded to 3 seconds

This is the same in both variants of the template.

Merge criteria

For a merge this will need:

  • demonstrated (with console output) end-to-end testing showing timeout met and cancelled, a request within timeout limits
  • golang-http and golang-middleware both need to be updated including armhf versions

No static file support in golang-middleware

Expected behaviour

Static files such as Golang HTML templates or similar assets can be read and served by the function

Current behaviour

Static files such as Golang HTML templates are not present in the function image

golang-middleware: Body not closed

In the default handler for golang-middleware the body is not closed of the request. This is generally done when reading the body.

I'd expect to see something like:

defer r.Body.Close()

Enable Go modules as a default

Description

Enable Go modules as a default

Update the templates to use GO111MODULE=on as a default

Then clarify in the docs or the README here how to turn it off, for those who need to vendor private code.

Testing

Test end to end to show that the build arg to turn off modules still works

Show output that also shows the default is working without additional settings.

Feature: Add Busybox Binary Version

My actions before raising this issue

Requesting a new feature which allows for a busybox:glibc based docker image. Here is one that I have come up with:
https://github.com/brianwyka/openfaas-templates

The reasons for using busybox:glibc are defined in the README.md, but here is a summary:

  • Resolve DNS resolution errors in alpine caused by muslc alternative to glibc

The template also allows for binary executables to be copied to the function into a bin directory, for the golang function handler to call using go exec.

The busybox:glibc image is essentially the same size (just as small) as alpine image as well.

chore: remove unused atomic int

My actions before raising this issue

Why do you need this?

I don't need this, I just realized this value is never been used, I am trying to make sense of it for the last 2 weeks, hopefully, this is not an awkward issue to bother people's time. I remember this variable being used in the past when I started learning faasd (around Go version 1.15 1.16 templates)

Expected Behaviour

The variable is set and it should be used

Current Behaviour

The variable is set but never used

https://github.com/openfaas/golang-http-template/blob/master/template/golang-http/main.go#L74
https://github.com/openfaas/golang-http-template/blob/master/template/golang-middleware/main.go#L72

List All Possible Solutions and Workarounds

the solution is just removing the unused variable

Steps to Reproduce (for bugs)

it is not a bug really.

Your Environment

  • FaaS-CLI version ( Full output from: faas-cli version ):
    0.15.4

  • Docker version docker version (e.g. Docker 17.0.05 ):
    Docker version 20.10.21

  • Are you using Docker Swarm or Kubernetes (FaaS-netes)?
    faasd 0.16.7

  • Operating System and version (e.g. Linux, Windows, MacOS):
    linux

  • Code example or link to GitHub repo or gist to reproduce problem:
    N/A

  • Other diagnostic information / logs from troubleshooting guide
    N/A

Next steps

You may join Slack for community support.

Cannot find package

Hi, when I create the project and generate the image I get this error
Do you have any idea why?

Step 21/36 : RUN GOOS=${TARGETOS} GOARCH=${TARGETARCH} go test ./... -cover
 ---> Running in f56f8b3d9eee
handler.go:7:2: cannot find package "github.com/openfaas/templates-sdk/go-http" in any of:
	/usr/local/go/src/github.com/openfaas/templates-sdk/go-http (from $GOROOT)
	/go/src/github.com/openfaas/templates-sdk/go-http (from $GOPATH)
The command '/bin/sh -c GOOS=${TARGETOS} GOARCH=${TARGETARCH} go test ./... -cover' returned a non-zero code: 1

Getting `go: inconsistent vendoring` error when a module is located within function

Expected Behaviour

It should work? 🤷‍♂️

Current Behaviour

Step 17/32 : RUN CGO_ENABLED=${CGO_ENABLED} GOOS=${TARGETOS} GOARCH=${TARGETARCH}     go build --ldflags "-s -w" -a -installsuffix cgo -o handler .
 ---> Running in 56939001fb88
go: inconsistent vendoring in /go/src/handler:
	github.com/openfaas/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	handler/function: is replaced in go.mod, but not marked as replaced in vendor/modules.txt

run 'go mod vendor' to sync, or use -mod=mod or -mod=readonly to ignore the vendor directory
The command '/bin/sh -c CGO_ENABLED=${CGO_ENABLED} GOOS=${TARGETOS} GOARCH=${TARGETARCH}     go build --ldflags "-s -w" -a -installsuffix cgo -o handler .' returned a non-zero code: 1
[0] < Building notifier-fn done in 2.64s.
[0] Worker done.

Total build time: 2.64s
Errors received during build:
- [notifier-fn] received non-zero exit code from build, error: The command '/bin/sh -c CGO_ENABLED=${CGO_ENABLED} GOOS=${TARGETOS} GOARCH=${TARGETARCH}     go build --ldflags "-s -w" -a -installsuffix cgo -o handler .' returned a non-zero code: 1

Steps to Reproduce (for bugs)

We were following the template documentation guide as described here.

  1. $ faas-cli template store pull golang-http
  2. $ faas-cli new notifier-fn --lang golang-http --prefix Dentrax
  3. $ cd notifier-fn
  4. $ go mod init github.com/Dentrax/foo
  5. Use a dependency (i.e. github.com/slack-go/slack) inside handler.go

Context

go.mod

module github.com/Dentrax/foo

go 1.16

require (
    github.com/openfaas/templates-sdk v0.0.0-20200723092016-0ebf61253625
    github.com/slack-go/slack v0.8.1
)

notifier-fn.yml:

version: 1.0
provider:
  name: openfaas
  gateway: http://foo
functions:
  notifier-fn:
    lang: golang-http
    handler: ./notifier-fn
    image: Dentrax/notifier-fn:latest
    build_args:
      GO111MODULE: on
    environment:
      SLACK_WEBHOOK_URL: https://bar

What we have tried so far that did not work

Each step we applied these in sequential:

  • faas-cli up -f notifier-fn.yml

  • faas-cli up -f notifier-fn.yml --build-arg GO111MODULE=on

  • Single handler.go file:

  • $ go mod init ...

  • $ go mod vendor + $ go mod tidy? (w/ vendor dir)

  • Using GO_REPLACE.txt in parent dir

Your Environment

FaaS-CLI version CLI: commit: 6b5e7a14a598527063c7c05fb4187739b6eb6d38 version: 0.13.6
Docker version Client: Docker Engine - Community Cloud integration: 1.0.7 Version: 20.10.2 API version: 1.41 Go version: go1.13.15 Git commit: 2291f61 Built: Mon Dec 28 16:12:42 2020 OS/Arch: darwin/amd64 Context: default Experimental: true

Server: Docker Engine - Community
Engine:
Version: 20.10.2
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 8891c58
Built: Mon Dec 28 16:15:28 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.3
GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc:
Version: 1.0.0-rc92
GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version: 0.19.0
GitCommit: de40ad0

  • Are you using Docker Swarm (FaaS-swarm ) or Kubernetes (FaaS-netes)? NO

  • Operating System and version (e.g. Linux, Windows, MacOS): macOS 11.0.1

We are not sure why the given template documentation does not work as we expected. I think we are missing some important detail on the documentation. Actually, the command that failing above is passing on our local workspace: $ CGO_ENABLED=${CGO_ENABLED} GOOS=${TARGETOS} GOARCH=${TARGETARCH} go build --ldflags "-s -w" -a -installsuffix cgo -o handler . Any ideas?

cc: @developer-guy

Improve caching for sub-package cache

I'm currently using golang-middleware and following the Advanced use-case where you have your own go package in a subfolder and make use of GO_REPLACE.txt.

The template works for me so far but build/deploy is very slow because it downloads dependencies every time.

Is there a way we could improve the Dockerfile so that a go mod download could be cached?

implicit redirect with go-middleware template

my usecase is to have a function that receives a URL as parameter, and after doing some processing redirects to that URL.

Now the redirect is working, but the URL in the browser still points to the original function URL.

I believe it can be fixed using client.CheckRedirect thing in golang, but i am not exactly sure where will that change be needed in openfaas cloud.

To reproduce:
https://rajatjindal.o6s.io/redirect-test -> this returns OK, no redirect
https://rajatjindal.o6s.io/redirect-test?redirect=https://google.com -> this get o/p from google.com, but the URL in browser still remains the same.

source code of this function: https://github.com/rajatjindal/redirect-test

Feature: replace alpine Docker image with scratch

Hi, I was looking at the golang template and noticed the image being based on an Alpine image.
Wouldn't it make more sense to base ourselves on a scratch image with non-root user?
e.g. look at https://github.com/ironpeakservices/iron-scratch/blob/master/Dockerfile

I can imagine smaller images (and no shell!) greatly reduce the size again and the attack surface.
Additional templates that require some additional binaries can use a full linux distribution, but the default behavior should be as secure/optimized as possible.

If you want I can open a PR for this.

Go modules - unable to get a sub-package to work

Description

When adding a sub-package to a function using golang-middleware and Go modules, I'm unable to get it to build. The unit tests fails immediately, then the go build fails if the test step is commented out.

Step 17/29 : RUN go build --ldflags "-s -w" -a -installsuffix cgo -o handler .
 ---> Running in cd54e2e66fb9
build handler: cannot load github.com/alexellis/vend/vend/handlers: git ls-remote -q origin in /go/pkg/mod/cache/vcs/15c84a3c22a2d4367cca03dcf38500ba216f1f2e95fa3fdd4255f8434c417d89: exit status 128:
        fatal: could not read Username for 'https://github.com': terminal prompts disabled
Confirm the import path was entered correctly.
If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.

Sample repo

https://github.com/alexellis/golang-middleware-relative-import

The top-level entry-point is main.go

main.go is a module and references a folder "function" which is also a module.

The basic path of a function handler with no other packages seems to work without Go modules, but when adding a sub-package i.e. handlers and referencing that, the tests and build fail.

main -> function -> handlers (sub-package)

This should have been a basic happy path scenario tested in #24, so I'm surprised to see that this didn't work. Pinging the people who tested / reviewed the PR.

@ewilde @LucasRoesler @bmcstdio @jonatasbaldin @stefanprodan

I imagine this is the same issue for go and golang-http also.

When using dep and a vendor folder this can be worked-around so that it works in the container, but not on a local system. The ideal resolution would have OpenFaaS functions with sub-packages working well in an IDE with highlighting and goreturns etc all working fine, and also working in a docker build.

Graceful shutdown for Kubernetes

There appears to be an issue with graceful shutdowns. This occurs when Kubernetes sends a SIGTERM to our function's Pod.

We have two processes:

  • of-watchdog
  • Go handler / HTTP server

The of-watchdog catches the signal and waits for wait_timeout env-var, but the child process appears to die immediately

We need to copy the code / approach of the of-watchdog into the Go handler for all templates and test.

Task

  • Show the issue by adding a SIGTERM handler and echo message in the entry point for one of the templates
  • Deploy in a container
  • Scale to --replicas=0
  • Look at the logs

Now fix the code and repeat

You should see that both processes wait at least wait_timeout before exiting.

Add a GitHub Action to do a test build for both templates

Add a GitHub Action to do a test build for both templates

Expected Behaviour

We should create a function with and without a Go template with both templates during CI and validate that they can build with docker or faas-cli build.

Current Behaviour

Manual testing

Possible Solution

Add a bash / contrib script to do this and a GitHub Action to run on push/pr.

Context

Backwards compatibility, preventing issues for users.

Recover for panics in function Handler for HTTP mode

In current code we are calling the http function handler as

result, resultErr := function.Handle(req)

In case there is a panic inside the handler the handler server process will exit, failing the future requests for the same.

Optionally it can be handled with a recover as:

defer func() {
       if recoverLog := recover(); recoverLog != nil {
               log.Print(recoverLog)
               w.WriteHeader(http.StatusInternalServerError)
       }
}()
result, resultErr := function.Handle(req)

In this case the handler server keep running

Broken link to golang-middleware section of README.md

Under the section at 2.0 golang-middleware, there is a command that results in an error.

The command is

faas template pull https://github.com/openfaas-incubator/golang-middleware-template

The output from running that command is

❯ faas template pull https://github.com/openfaas-incubator/golang-middleware-template
Fetch templates from repository: https://github.com/openfaas-incubator/golang-middleware-template at master
2020/03/06 09:17:36 Attempting to expand templates from https://github.com/openfaas-incubator/golang-middleware-template
Cloning into '/var/folders/zj/vc4ln5h14djg9svz7x_t1d0r0000gq/T/openFaasTemplates019767266'...
remote: Repository not found.
fatal: repository 'https://github.com/openfaas-incubator/golang-middleware-template/' not found
Error while fetching templates: exit status 128

golang-http does not allow graceful request cancelling

Because the go-function-sdk does not propagate the original request context in the Request object, the handler can not correctly respond to the request being cancelled by the user. In the Go stdlib this information is generally conveyed by the checking if request.Context() is done.

If the handler has an expensive operation that should be canceled and reverted, e.g. a DB transaction, this would be impossible because the handler can not check for the early cancellation.

We should be able to see this by creating a handler that prints/counts to a large number, issuing a request and then immediately cancelling. The handler should continue to count even after the user aborts the request.

For example, using this

# stack.yml
version: 1.0
provider:
  name: openfaas
  gateway: http://127.0.0.1:808

configuration:
  templates:
    - name: golang-http
      source: https://github.com/openfaas-incubator/golang-http-template

functions:
  counter:
    lang: golang-http
    handler: ./counter
    image: counter:latest
# counter/handler.go
package function

import (
	"fmt"
	"net/http"

	handler "github.com/openfaas-incubator/go-function-sdk"
)

// Handle a function invocation
func Handle(req handler.Request) (handler.Response, error) {
	var err error

	for i := 0; i < 1000; i++ {
		fmt.Printf("count %d\n", i)
	}

	message := fmt.Sprintf("Hello world, input was: %s", string(req.Body))
	return handler.Response{
		Body:       []byte(message),
		StatusCode: http.StatusOK,
	}, err
}

Build and run the function

faas-cli build -f stack.yml
docker run --rm -p 8080:8080 counter:latest 

Then in a new terminal

curl localhost:8080 -d "anything"
<ctrl>-c

You will see the function start and then continue even after you ctrl-c the curl request. This is because the handler has not even attempted to check for the abort. But it is not currently possible to do this because the context is not available.

Provide armhf variants

Provide duplicated versions of the two templates which end with "-armhf" as the name.

This is to add better support for our ARM users.

faas-cli build error with dep

faas-cli build ... fails to build the image when the function contains external libraries vendored with dep.

Steps to reproduce:

faas template pull https://github.com/openfaas-incubator/golang-http-template
faas-cli new --lang golang-http test-fn
cd test-fn
# now create a sample function (handler.go) using an external library, example below
# make sure when you save that the openfaas-incubator... import is not removed, e.g. due to go imports formating
dep init # assumption here is that dep should be run from within the handler folder (?)
tree -L 2 vendor
vendor
└── github.com
    ├── openfaas-incubator
    └── pkg
# dep correctly did its job
cd ..
faas-cli build -f test-fn.yml

Then it fails at Dockerfile step 7:

Step 7/17 : RUN CGO_ENABLED=0 GOOS=linux     go build --ldflags "-s -w" -a -installsuffix cgo -o handler . &&     go test $(go list ./... | grep -v /vendor/) -cover
 ---> Running in eb8face14c7b
# handler
./main.go:37:39: cannot use req (type "handler/vendor/github.com/openfaas-incubator/go-function-sdk".Request) as type "handler/function/vendor/github.com/openfaas-incubator/go-function-sdk".Request in argument to function.Handle
The command '/bin/sh -c CGO_ENABLED=0 GOOS=linux     go build --ldflags "-s -w" -a -installsuffix cgo -o handler . &&     go test $(go list ./... | grep -v /vendor/) -cover' returned a non-zero code: 2
2019/01/17 22:23:33 ERROR - Could not execute command: [docker build -t test-fn:latest .]

Took me a while to figure out how to work around that. Before faas-cli build remove the folder openfaas-incubator in the handler.go vendor folder. Then build works fine (as it will pull in the dependency from the template).

Example function:

package function

import (
	"fmt"
	"net/http"

	"github.com/openfaas-incubator/go-function-sdk"
	"github.com/pkg/errors"
)

// Handle a function invocation
func Handle(req handler.Request) (handler.Response, error) {
	var err error
	err = errors.Wrap(err, "hit a bug")

	message := fmt.Sprintf("Hello world, input was: %s", string(req.Body))

	return handler.Response{
		Body:       []byte(message),
		StatusCode: http.StatusOK,
	}, err

}

cc/ @alexellis

received non-zero exit code from build, error: invalid platform architecture "amd64}"

when I build the template, like faas-cli build -f golang-middleware-test.yml

I get the following output

[0] > Building golang-middleware-test.
Clearing temporary build folder: ./build/golang-middleware-test/
Preparing: ./golang-middleware-test/ build/golang-middleware-test/function
Building: golang-middleware-test:latest with golang-middleware template. Please wait..
Sending build context to Docker daemon  11.26kB
Step 1/33 : FROM --platform=${TARGETPLATFORM:-linux/amd64} openfaas/of-watchdog:0.8.0 as watchdog
invalid platform architecture "amd64}"
[0] < Building golang-middleware-test done in 0.29s.
[0] Worker done.

Total build time: 0.29s
Errors received during build:
- [golang-middleware-test] received non-zero exit code from build, error: invalid platform architecture "amd64}"

My Environment

  • FaaS-CLI version ( Full output from: faas-cli version ):
CLI:
 commit:  ea687659ecf14931a29be46c4d2866899d36c282
 version: 0.11.8
Your faas-cli version (0.11.8) may be out of date. Version: 0.12.14 is now available on GitHub.
  • Docker version docker version (e.g. Docker 17.0.05 ):
Client:
 Version:	18.04.0-ce
 API version:	1.37
 Go version:	go1.9.4
 Git commit:	3d479c0
 Built:	Tue Apr 10 18:20:32 2018
 OS/Arch:	linux/amd64
 Experimental:	false
 Orchestrator:	swarm

Server:
 Engine:
  Version:	18.04.0-ce
  API version:	1.37 (minimum version 1.12)
  Go version:	go1.9.4
  Git commit:	3d479c0
  Built:	Tue Apr 10 18:18:40 2018
  OS/Arch:	linux/amd64
  Experimental:	false
  • Are you using Docker Swarm or Kubernetes (FaaS-netes)?
Kubernetes 
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:32:58Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
  • Operating System and version (e.g. Linux, Windows, MacOS):
    Linux version 4.12.0-041200-generic (kernel@gloin) (gcc version 6.3.0 20170618 (Ubuntu 6.3.0-19ubuntu1) )

Update Docker build layer to go 1.11.13

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.