Giter VIP home page Giter VIP logo

cloud-functions-go's Introduction

Status: Archived

status: inactive

This repository has been archived and is no longer maintained.

We recommend you use:


Unofficial Native Go Runtime for Google Cloud Functions

Disclaimer: This is not an official Google product. It is not and will not be maintained by Google, and is not part of Google Cloud Functions project. There is no guarantee of any kind, including that it will work or continue to work, or that it will supported in any way.

Background

When this doc was first written, Google Cloud Functions only supported a Node.js runtime. Since then, new official runtimes have been added. Using an official runtime (such as the official Go runtime) is the recommended approach for almost all users.

But if you're feeling adventurous, you can run non-JavaScript code by using native Node modules and running subprocesses. Node will always be handling the requests. Wrapping Go code in a native Node module will require writing complicated foreign function interfaces (FFIs). FFIs are notoriously difficult to get right and are an easy way to introduce memory leaks as well as security and stability issues. Using a subprocess is much easier to get right, but it introduces the overhead of interprocess communication (IPC) and context switches. In both cases, the requests are proxied between Node and Go which typically involves extra copies and translations.

100% Pure Go Request Path

After an initial bootstrap step, the only thing running is a pure Go binary. This project works by completely replacing the Node process and lets a Go process directly handle requests. There is no cgo and no proxying. This means that it is faster and potentially more secure than other projects which typically to use both.

How It Works

A native module calls execve(2) without the normal preceding fork(2) call when the user module is imported. Open socket FDs are preserved and handled by the new Go process to ensure a smooth transition. The new Go process then pretends to be Node.

Requirements

There are four supported environments:

Native

For advanced users who don't like VirtualBox.

  • Linux or macOS development environment (The native Node module depends on Linux/macOS specific behavior.)
  • Go 1.5 or above
  • Node.js v0.10 or above, npm and node-gyp
  • Make and GCC
  • If you are using macOS then as of right now you need the FULL xcode installed. Command line tools alone will not work.

Vagrant

Compatible with Windows, macOS and Linux and easier than native development.

In the cloud-functions-go directory, run vagrant up to start the environment and vagrant ssh to get a shell. Support for other Vagrant providers besides VirtualBox is in the works.

Windows with Cygwin

  • Go for Windows
  • Node.js for Windows
  • Cygwin with make and zip (you may also want git and an editor like vim or nano)

Use the Cygwin Terminal to run the commands as described below. Note that make test won't work under Windows.

Windows with Powershell 5.0

  • Go 1.5 or above
  • Node.js (optional)

The commands described below may be run as-is using Command Prompt, or prefixed with ./ using Windows PowerShell (i.e. ./make or ./make godev). Note that make test won't work under Windows.

External Dependencies

The events sub-package depends on the following libraries:

  • google.golang.org/api/pubsub/v1
  • google.golang.org/api/storage/v1

Hello, world!

A demo hello world example is included. To try it out, simply skip to the Deployment section.

Making Changes

For normal development, it should only be necessary to modify the main.go file. Although this file probably shouldn't live in your GOPATH, feel free to import libraries from your GOPATH (including libraries downloaded with go get).

Local Testing

Run make test to compile your code and start the test server. Open http://localhost:8080/execute in your browser. The page should display User function is ready. Refresh the page to talk to your code.

When using a go-only or non-unix environment, run make godev instead.

Logging

The logger may be used directly:

nodego.InfoLogger.Println("Hello World!")
nodego.ErrorLogger.Println("Something went wrong!")

or via the log package after calling nodego.OverrideLogger():

func init() {
	nodego.OverrideLogger()
}

...

log.Println("Hello World!")

Use the nodego.WithLogger() or nodego.WithLoggerFunc() middleware for your logs to reference the specific execution:

http.WithLoggerFunc(func(w http.ResponseWriter, r *http.Request) {
	log.Println("Hello World!")
})

A full example is included in examples/logging.go.

Deployment

Run make to compile and package your code. Upload the generated function.zip file to Google Cloud Functions as an HTTP trigger function.

Vagrant

Run vagrant up to start the envirement. Run vagrant ssh to connect to the envirement. Run cd /vagrant to access the respority files. The instructions in Local Testing and Deployment should now work.

Troubleshooting

Some versions of Node.js (especially those packaged for Ubuntu) name their main binary nodejs instead of node. The symptom of this problem is an error about the node binary not being found in the path even though Node.js is installed. This can be fixed with sudo ln -s $(which nodejs) /usr/local/bin/node. There's also a package called nodejs-legacy that can be installed in some Debian and Ubuntu distros using apt that creates a symlink node in /usr/bin/

It may also be helpful to understand the nodejs environment that your Google Cloud Function runs in (e.g. how does regular console.log send logs to Stackdriver, whereas fmt.Println doesn't?). You can find more details in the WORKERJS.md file.

License

Copyright 2017, Google, Inc.

Licensed under the Apache License, Version 2.0

See LICENSE.

cloud-functions-go's People

Contributors

choonkeat avatar grant avatar iangudger avatar jasonaheron avatar prayashm avatar salrashid123 avatar sethvargo avatar ssttevee avatar stewartreichling avatar theist avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cloud-functions-go's Issues

Official Golang support plans?

Not sure where to look for info on this, are there any plans for native support of Golang in google cloud functions? Feels a bit silly that AWS Lambda beat them to this.

If this isn't the right place to ask, any ideas about where we should would be great.

currently only allow one main.go

I have tried to compile multiple .go files with make & make test and throw a error instead if I use go run main.go my_other_code.go work ok, so I think is a matter of make stript.
Thanks

PubSub trigger `json: cannot unmarshal object`

Hi!

I'm trying to deploy a Go Cloud Function, with PubSub trigger, but I'm always hitting this error:

{
 insertId:  "000000-0e31e000-5a9c-41f7-b40e-5af2eba5cdab"   
 labels: {
  execution_id:  ""    
 }
 logName:  "projects/infrasnukture/logs/cloudfunctions.googleapis.com%2Fcloud-functions"   
 receiveTimestamp:  "2018-07-26T11:27:35.175212742Z"   
 resource: {
  labels: {
   function_name:  "test-go1"     
   project_id:  "infrasnukture"     
   region:  "europe-west1"     
  }
  type:  "cloud_function"    
 }
 severity:  "ERROR"   
 textPayload:  "Failed to decode event: json: cannot unmarshal object into Go struct field EventContext.resource of type string
"   
 timestamp:  "2018-07-26T11:27:28.982Z"   
}

This is the payload of the PubSub message:

{
"data":{
    "data":"dGVzdA==",
    "attributes":{
      "age":"22"
    },
    "@type":"type.googleapis.com/google.pubsub.v1.PubsubMessage"
  }
}

And my main.go looks like this:

package main

import (
	"flag"
	"net/http"

	"./events"
	"./nodego"
)

func main() {
	flag.Parse()

	http.HandleFunc(nodego.PubSubTrigger, events.Handler(func(event *events.Event) error {
		nodego.InfoLogger.Printf("PubSub triggered Go function!")

		msg, err := event.PubSubMessage()
		nodego.InfoLogger.Printf("Your message: %s", msg.Data)
		if err != nil {
			return err
		}
		return nil
	}))

	nodego.TakeOver()
}

BTW, if I run it locally on my machine, it works fine.

Appreciate some help!

NodeJS, PassportJS or JWT in Google Cloud Functions

Hello, I would like to know if it is possible to make an authentication system using nodejs with passportjs or JWT integrated with Google Cloud Functions, which I want to know exactly if doing the authentication in GCF I do not have to authenticate every time I do an action and yes only authenticate once

Deployment failure: Operation interrupted.

When I tried to deploy function.zip I got the following error message on Google Cloud Functions:

Deployment failure: Operation interrupted.

This is what I did:

At first, clone and build:

$ git clone https://github.com/GoogleCloudPlatform/cloud-functions-go.git && cd cloud-functions-go
$ make

Then uploaded to Google Cloud Functions using the GCP web UI.

The following error entry is in the logs:

{
 insertId:  "-uie6zkd1h36"  
 logName:  "projects/my-project-id/logs/cloudaudit.googleapis.com%2Factivity"  
 operation: {
  id:  "operations/aXBpbmZvLTE4MTYyMC91cy1jZW50cmFsMS9mdW5jdGlvbi0xL1YyWERCd1Z4ZGdr"   
  last:  true   
  producer:  "cloudfunctions.googleapis.com"   
 }
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"   
  methodName:  "google.cloud.functions.v1.CloudFunctionsService.CreateFunction"   
  resourceName:  "projects/my-project-id/locations/us-central1/functions/function-1"   
  serviceName:  "cloudfunctions.googleapis.com"   
  status: {
   code:  10    
   message:  "Operation interrupted."    
  }
 }
 receiveTimestamp:  "2018-02-19T12:27:44.296923939Z"  
 resource: {
  labels: {
   function_name:  "function-1"    
   project_id:  "my-project-id"    
   region:  "us-central1"    
  }
  type:  "cloud_function"   
 }
 severity:  "ERROR"  
 timestamp:  "2018-02-19T12:27:43.703Z"  
}

Function always dies when issuing HTTP(S)-Requests with nodego.OverrideLogger()

I've been having a bad time trying to issue HTTP(S)-Requests from a Cloud Function that uses this package.

Consider this absolute minimal repro:

package main

import (
    "flag"
    "net/http"

    "github.com/GoogleCloudPlatform/cloud-functions-go/nodego"
)

func init() {
    nodego.OverrideLogger()
}

func main() {
    flag.Parse()

    http.HandleFunc(nodego.HTTPTrigger, nodego.WithLoggerFunc(func(w http.ResponseWriter, r *http.Request) {
        http.Get("http://example.com/")
    }))

    nodego.TakeOver()
}

When called, it always dies:

Function execution took 296 ms, finished with status: 'connection error'
Error: function crashed out of request scope Function invocation was interrupted.

Just when writing this, it occured to me to leave out the line nodego.OverrideLogger().
Now the Cloud Function finally does what it is supposed to.

Does anyone have a clue what's wrong here?

Error: could not handle the request

Getting this plain text response every 30 or so requests:

Error: could not handle the request

Checking the server logs I found this:

severity:  "ERROR"  
textPayload:  "Error: function crashed out of request scope
 Function invocation was interrupted.
" 

Deployment issues

I've had two different issues with deployment I've since worked around.
Leaving this here for documentation, maybe it'll help others.

Allow setting environment variables while launching runtime

Suggest using

use execve instead of execv so that users can set and launch a runtime which may require specific .so files or other env variables

For example:

	char *envp[] =
	{
			"LD_LIBRARY_PATH=/user_code/lib:$LD_LIBRARY_PATH",
			NULL
	};

	execve(bin, const_cast<char* const*>(&args[0]), envp);  

where put my code

I tested and work very nice but I have a doubt where should I place my code and how in other to be isolate of runtime code.

nodego.go ?
main.go ?
other

cross projects permissions

I have tried to performance a query from one project A to a BigQuery in other project B and I am getting a error connection. (i am using a service-account)

Same app running locally works propertly. I have spent a lot time with permision, service account, token, credentials etc, etc.

Someone can confirm me that current version on c-f-go support cross projects environments.

thanks

ps. apoint to example will be perfect.

projectID := "xxxx"
ctx := context.Background()
jwt, err := google.JWTConfigFromJSON(getServiceAccount(), bigquery.Scope)
if err != nil {
	log.Fatal(err)
}
client, err := bigquery.NewClient(
        ctx,
	projectID,
	option.WithTokenSource(jwt.TokenSource(ctx)),
	)
if err != nil {
log.Fatal(err)
	}
```

Can't get this package.

I can't get this package for following reason.

$ go get github.com/GoogleCloudPlatform/cloud-functions-go
../../../github.com/GoogleCloudPlatform/cloud-functions-go/main.go:22:2: local import "./nodego" in non-local package

How to use this package?

Pass through the file descriptors with options denoting what they are for

Right now the runtime binary is executed with the file descriptors as positional arguments and the application runtime has to check/interpret what the FD's are for.

Suggest adding in options to args for the parameter's intents

execv(bin, const_cast<char* const*>(&args[0]));

eg.
instead of
./main file_descriptor

you'd get

./main --listenfd file_descriptor

that way, i can opt-parse the arguments in main.go if i wanted to extend it.

Why vendor folder is ignored (go project)

I am trying to deploy my go/http project as a function with a command like

gcloud functions deploy table-viewer --verbosity=debug --entry-point LambdaX --runtime go111 --trigger-http --set-env-vars AUTH_LOGIN=login,...

.gcloudignore contains both

go.sum
go.mod

From the output :

INFO: Using .gcloudignore file at [.../table-viewer/.gcloudignore].
DEBUG: Skipping file [go.mod]
DEBUG: Skipping file [go.sum]
DEBUG: Skipping file [.gitignore]
DEBUG: Skipping file [vendor]
DEBUG: Skipping file [vendor/golang.org]
DEBUG: Skipping file [vendor/golang.org/x]
DEBUG: Skipping file [vendor/golang.org/x/sys]
...

Why it ignores vendor folder?

about function-go

I have tried and it works very well and I love the idea of build my code using this architecture but I am concern about two stuff.
There will be a cloud-functions-go supported by google soon?
My environment is windows OS . Exists a workaround to build on windows box.

Thanks

Deployment failure without invisibility!

I have just started using Cloud function - go to create a new application. On my first deployment, It took more than 10 minutes and at the end showed me the following error:

ERROR: (gcloud.functions.deploy) OperationError: code=10, message=Operation interrupted.

I have a couple of issues with the deployment so far:

  • There is no way or at least I couldn't find it in documentation to see the deployment log as it happens
  • It fails without any further information!

Would be great if someone can shed a light on this issue as I am currently stuck

there are any update?

We are using cloud-functions but lately it is apparently stopped someone can tell us some updated

thanks

Use docker_file to build node_modules and acquire additional libraries

This is a longer term FR:

At the moment, the build and test steps requires gcc,make nodejs. However, those are only needed if you want to deploy with functions.zip or run the test node server.

Suggestion is two part:

allow users to run cloud function alone locally using native toolchains and not deal with node and the execv.
The advantage of this is you don't need node, make, gcc or anything and can elect to acquire them during deployment by other means.

By other means i mean like build node_modules and execve entirely in docker and then "copy" them to your workstation just prior to creating functions.zip.

This would require the runtime binary to accept both sockets as arguments as well as listen on its own (eg:)

http.ListenAndServe(":8080", nil)

The following dockerfile is an example of this 'just in time' library/module support:

Makefile

all: node_modules lib bin_from_local
	zip -FS -r $(OUT) bin/ lib/ node_modules index.js package.json -x *build*

bin_from_local:
	<use go to compile ./main into bin/>

node_modules:
	docker build -t docker_tmp -f dockerfiles/Dockerfile_node_modules .
	docker cp `docker create docker_tmp`:/user_code/node_modules .
	docker rmi -f docker_tmp

lib: 
	docker build -t docker_tmp -f dockerfiles/Dockerfile_lib .
	docker cp `docker create docker_tmp`:/user_code/lib .
	docker rmi -f docker_tmp

To compile node_modules:

Dockerfile_node_modules

FROM debian:jessie
ADD . /user_code
RUN apt-get update && apt-get install -y --no-install-recommends \
        curl \
        gcc \
        npm \
        build-essential \
    && rm -rf /var/lib/apt/lists/*

RUN curl -sL https://deb.nodesource.com/setup_6.x | bash -  && \ 
    apt-get update && apt-get install -y nodejs

WORKDIR /user_code
RUN npm install --save local_modules/execer

To acquire additional shared objects (.so) files and copy them to lib/

FROM debian:jessie

RUN apt-get update \
    && apt-get install -y --no-install-recommends \
        ca-certificates \
        libc6 \
        libcurl3 \
        libgcc1 \
    && rm -rf /var/lib/apt/lists/*

RUN mkdir -p /user_code/lib && \
    for i in `dpkg -L libc6 libcurl3 libgcc1  | egrep "^/usr/lib/x86_64-linux-gnu/.*\.so\."`; do cp $i /user_code/lib; done

How is lifecycle

I have read how work this piece of code and it is very powerfu. I would like know how work lifecycle of instance, connection, reuse objects etc.

Thanks in advance

Can't deploy function to Google Cloud Platform

Hello,
Thank you for this wonderful initiative to add Go support to Cloud Functions! Unfortunately, I cannot seem to be able to deploy the example function. It works fine locally, but after uploading the zip file to the new function it just gets stuck during deployment and it eventually gets cut off by the quota limit.
status: { code: 10 message: "Operation interrupted." }

Any ideas of what might be going wrong? Thanks again!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.