Giter VIP home page Giter VIP logo

terraform-google-scheduled-function's Introduction

Terraform Google Scheduled Functions Module

This modules makes it easy to set up a scheduled job to trigger events/run functions.

Compatibility

This module is meant for use with Terraform 0.13+ and tested using Terraform 1.0+. If you find incompatibilities using Terraform >=0.13, please open an issue. If you haven't upgraded and need a Terraform 0.12.x-compatible version of this module, the last released version intended for Terraform 0.12.x is v1.6.0.

Usage

You can go to the examples folder, however the usage of the module could be like this in your own main.tf file:

module "scheduled-function" {
  source  = "terraform-google-modules/scheduled-function/google"
  version = "0.1.0"
  project_id   = "<PROJECT ID>"
  job_name="<NAME_OF_JOB>"
  job_schedule="<CRON_SYNTAX_SCHEDULE>"
  function_entry_point="<NAME_OF_FUNCTION>"
  function_source_directory="<DIRECTORY_OF_FUNCTION_SOURCE>"
  function_name="<RESOURCE_NAMES>"
  region="<REGION>"
}

Then perform the following commands on the root folder:

  • terraform init to get the plugins
  • terraform plan to see the infrastructure plan
  • terraform apply to apply the infrastructure build
  • terraform destroy to destroy the built infrastructure

Inputs

Name Description Type Default Required
bucket_force_destroy When deleting the GCS bucket containing the cloud function, delete all objects in the bucket first. bool true no
bucket_name The name to apply to the bucket. Will default to a string of -scheduled-function-XXXX> with XXXX being random characters. string "" no
create_bucket Create bucket (default). Set to false to use existing one bool true no
function_available_memory_mb The amount of memory in megabytes allotted for the function to use. number 256 no
function_description The description of the function. string "Processes log export events provided through a Pub/Sub topic subscription." no
function_docker_registry Docker Registry to use for storing the function's Docker images. Allowed values are CONTAINER_REGISTRY (default) and ARTIFACT_REGISTRY. string null no
function_docker_repository User managed repository created in Artifact Registry optionally with a customer managed encryption key. If specified, deployments will use Artifact Registry. string null no
function_entry_point The name of a method in the function source which will be invoked when the function is executed. string n/a yes
function_environment_variables A set of key/value environment variable pairs to assign to the function. map(string) {} no
function_event_trigger_failure_policy_retry A toggle to determine if the function should be retried on failure. bool false no
function_kms_key_name Resource name of a KMS crypto key (managed by the user) used to encrypt/decrypt function resources. string null no
function_labels A set of key/value label pairs to assign to the function. map(string) {} no
function_max_instances The maximum number of parallel executions of the function. number null no
function_name The name to apply to the function string n/a yes
function_runtime The runtime in which the function will be executed. string "nodejs10" no
function_secret_environment_variables A list of maps which contains key, project_id, secret_name (not the full secret id) and version to assign to the function as a set of secret environment variables. list(map(string)) [] no
function_service_account_email The service account to run the function as. string "" no
function_source_archive_bucket_labels A set of key/value label pairs to assign to the function source archive bucket. map(string) {} no
function_source_dependent_files A list of any terraform created local_files that the module will wait for before creating the archive.
list(object({
filename = string
id = string
}))
[] no
function_source_directory The contents of this directory will be archived and used as the function source. string n/a yes
function_timeout_s The amount of time in seconds allotted for the execution of the function. number 60 no
grant_token_creator Specify true if you want to add token creator role to the default Pub/Sub SA bool false no
ingress_settings The ingress settings for the function. Allowed values are ALLOW_ALL, ALLOW_INTERNAL_AND_GCLB and ALLOW_INTERNAL_ONLY. Changes to this field will recreate the cloud function. string null no
job_description Addition text to describe the job string "" no
job_name The name of the scheduled job to run string null no
job_schedule The job frequency, in cron syntax string "*/2 * * * *" no
message_data The data to send in the topic message. string "dGVzdA==" no
project_id The ID of the project where the resources will be created string n/a yes
region The region in which resources will be applied. string n/a yes
scheduler_job An existing Cloud Scheduler job instance object({ name = string }) null no
time_zone The timezone to use in scheduler string "Etc/UTC" no
topic_kms_key_name The resource name of the Cloud KMS CryptoKey to be used to protect access to messages published on this topic. string null no
topic_labels A set of key/value label pairs to assign to the pubsub topic. map(string) {} no
topic_name Name of pubsub topic connecting the scheduled job and the function string "test-topic" no
vpc_connector The VPC Network Connector that this cloud function can connect to. It should be set up as fully-qualified URI. The format of this field is projects//locations//connectors/*. string null no
vpc_connector_egress_settings The egress settings for the connector, controlling what traffic is diverted through it. Allowed values are ALL_TRAFFIC and PRIVATE_RANGES_ONLY. If unset, this field preserves the previously set value. string null no

Outputs

Name Description
name The name of the job created
pubsub_topic_name PubSub topic name
scheduler_job The Cloud Scheduler job instance

Requirements

These sections describe requirements for using this module.

Software

The following dependencies must be available:

App Engine

Note that this module requires App Engine being configured in the specified project/region. This is because Google Cloud Scheduler is dependent on the project being configured with App Engine. Refer to the Google Cloud Scheduler documentation information on the App Engine dependency.

The recommended way to create projects with App Engine enabled is via the Project Factory module. There is an example of how to create the project within that module

Service Account

A service account with the following roles must be used to provision the resources of this module:

  • Storage Admin: roles/storage.admin
  • PubSub Editor: roles/pubsub.editor
  • Cloudscheduler Admin: roles/cloudscheduler.admin
  • Cloudfunctions Developer: roles/cloudfunctions.developer
  • IAM ServiceAccount User: roles/iam.serviceAccountUser

The Project Factory module and the IAM module may be used in combination to provision a service account with the necessary roles applied.

APIs

A project with the following APIs enabled must be used to host the resources of this module:

  • Cloud Scheduler API: cloudscheduler.googleapis.com
  • Cloud PubSub API: pubsub.googleapis.com
  • Cloud Functions API: cloudfunctions.googleapis.com
  • Cloud Build API: cloudbuild.googleapis.com
  • App Engine Admin API: appengine.googleapis.com

The Project Factory module can be used to provision a project with the necessary APIs enabled.

Contributing

Refer to the contribution guidelines for information on contributing to this module.

terraform-google-scheduled-function's People

Contributors

aaron-lane avatar agustinramirodiaz avatar alexrohv avatar andolsizied avatar apeabody avatar averbuks avatar bharathkkb avatar bkamin29 avatar cloud-foundation-bot avatar daniel-cit avatar darkwanderer avatar dependabot[bot] avatar eeaton avatar erjohnso avatar fpoon avatar franciscocpg avatar franviera92 avatar g-awmalik avatar jberlinsky avatar luketaverne avatar moreal avatar morgante avatar nick4fake avatar omazin avatar paulpalamarchuk avatar release-please[bot] avatar renato-rudnicki avatar renovate[bot] avatar taylorludwig avatar zachberger avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

terraform-google-scheduled-function's Issues

project cleaner: bug in folder cleanup

TL;DR

Looks like there maybe a bug in folder cleanup resulting in some folders not getting cleaned up even when empty.

Expected behavior

No response

Observed behavior

No response

Terraform Configuration

N/A

Terraform Version

N/A

Additional information

No response

project cleaner fails to delete org-level tag keys and values when associated with resources

TL;DR

removeTagKeys := func(organization string) {

This functionality in project_cleanup attempts to delete tags without removing their association. This fails if the tag is associated with any existing or deleted resources.

Expected behavior

When the cleanup function is referenced with clean_up_org_level_tag_keys set to true, it should be able to delete tag keys and values.

Observed behavior

The delete method fails when tags still have a resource association. This occurs even when the associated resource has been deleted.

ERROR 2024-06-04T14:18:26.666958Z [protoPayload.serviceName: cloudresourcemanager.googleapis.com] [protoPayload.methodName: google.cloud.resourcemanager.v3.TagValues.DeleteTagValue] [protoPayload.resourceName: tagValues/281484311440934] [protoPayload.authenticationInfo.principalEmail: project-cleaner-function@cft-project-manager-e86e.iam.gserviceaccount.com] Cannot delete tag value, tagValues/281484311440934, because it is still attached to resources in 'global' region. To delete this tag value, remove all holds and then remove it from all resources. At least one binding was found to an active or deleted resource in the 'global' region.

Terraform Configuration

n/a, the issue is in go code

Terraform Version

n/a, the issue is in go code

Additional information

No response

Support for create_bucket argument.

TL;DR

The event-function module has a create_bucket input argument that should also be supported by this module.
The inclusion of this input argument permits the use case of using a bucket that already exists for the storage of function deployment sources when create_bucket is false and bucket_name points to an existing bucket.

Terraform Resources

https://registry.terraform.io/modules/terraform-google-modules/event-function/google/latest?tab=inputs

Detailed design

In main.tf:

module "main" {
    ...
    create_bucket = var.function_create_bucket
    ...
}

In variables.tf:

variable "function_create_bucket" {
    type = bool
    default = true
    description = "Whether to create a new bucket or use an existing one. If false, `bucket_name` should reference the name of the alternate bucket to use."
}

Additional information

I am more than happy to create a PR for this, and I think this should be supported since finer control is always nice, especially since that finer control is already supported by the underlying event functions module this module makes use of.

Support for max_instances argument

TL;DR

Hi,
It seems that this module lacks support for cloud function max_instances argument, though the underlying event-function module seems to support it.
Adding this would protect users against unlimited scaling in case of pubsub malfunction (e.g. constant failing to acknowledge message and redelivering it, which might result in increasing number of cloud function instances)

Terraform Resources

No response

Detailed design

main.tf:
...
module "main" {
  ...
  max_instances = var.function_max_instances
  ...
}
...

variables.tf:
...
variable "function_max_intances" {
  type        = number
  default     = 0
  description = "The maximum number of parallel executions of the function."
}
...

Additional information

If you are ok with this proposal, I'll be happy to contribute code :)

Release for terraform 0.12?

Hi. It looks like this module is updated for terraform 0.12, but there has not been a release. Is it possible to make a release supporting 0.12?

Add CAI asset feeds to project cleanup

TL;DR

add cai asset feeds to project cleanup at the org level

Terraform Resources

No response

Detailed design

No response

Additional information

No response

Ability to use existing Cloud Scheduler

It would be nice if we could pass in an existing Cloud Scheduler job instead of creating one every time: this is useful when we have multiple functions that need to be synchronized (i.e: triggered by the same Cloud Scheduler job).

Is that something doable ?

google_cloudfunctions_function.main : Error : file main.py [...] doesn't exist.

Hello,
I'm systematically encountered an error when running a first terraform apply command from my laptop and after adding a new terraform-google-scheduled-function resource.
I encounter this error by using terraform-google-slo/slo

Trace :

[...]
module.slo-generator.module.slo-hp-fr-latency-1024ms.module.slo-cloud-function.module.main.google_cloudfunctions_function.main: Still creating... [1m40s elapsed]

Error: Error waiting for Creating CloudFunctions Function: Error code 3, message: Function failed on loading user code. Error message: File main.py that is expected to define function doesn't exist

  on .terraform/modules/slo-generator.slo-hp-fr-latency-1024ms.slo-cloud-function.main/terraform-google-modules-terraform-google-event-function-32a9ef5/main.tf line 41, in resource "google_cloudfunctions_function" "main":
  41: resource "google_cloudfunctions_function" "main" {

After re-executing a terraform apply command, the new cloud function is created successfully.

Maybe we don't wait long enough between the event where we push "code" in the storage bucket and the moment where the Clund Function is in creating.
So i have try to add a manual resource dependency (depends_on=) in google_cloudfunctions_function. main, but it's not a success..

Could you please create a new release/version with the output fixe ?

When using this sub_module with terraform-google-slo, i have an error concerning the output scheduler_job :

Error: Inconsistent conditional result types

  on .terraform/modules/slo_generator.slo-hp-fr-availability.slo_cloud_function/outputs.tf line 23, in output "scheduler_job":
  23:   value       = var.scheduler_job == null ? google_cloud_scheduler_job.job : var.scheduler_job
    |----------------
    | google_cloud_scheduler_job.job is empty tuple
    | var.scheduler_job is object with 1 attribute "name"

The true and false result expressions must have consistent types. The given
expressions are tuple and object, respectively.

I think the value is not good, google_cloud_scheduler_job.job have many element, it's list. In our case the output have to be :

output "scheduler_job" { value = var.scheduler_job == null ? google_cloud_scheduler_job.job[0] : var.scheduler_job description = "The Cloud Scheduler job instance" }

I have test and it's work fine now

Bucket name configurability

Docs state that bucket_name defaults to <project-id>-scheduled-function-XXXX>, however that does not seem to be the case.

In particular, deploying an instance of project-cleaner submodule errors out as it wants to use the function name old-project-cleaner as the bucket name (which is expected behavior from event-function) and it already exists.

Tests are flaky

When executing tests, it looks like functions fail to converge:

       module.pubsub_scheduled_example.module.pubsub_scheduled_example.module.main.google_cloudfunctions_function.main: Still creating... [10s elapsed]
       
       Error: Error waiting for Creating CloudFunctions Function: Error code 3, message: Failed to retrieve function source code
       
         on .terraform/modules/pubsub_scheduled_example.pubsub_scheduled_example.main/terraform-google-event-function-1.2.0/main.tf line 68, in resource "google_cloudfunctions_function" "main":
         68: resource "google_cloudfunctions_function" "main" {

Module with http endpoint support

TL;DR

Request a new module to support a cloud schedule job calling an http endpoint

Terraform Resources

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_scheduler_job#example-usage---scheduler-job-http

Detailed design

Biggest thing seems to be supporting http_target block

Additional information

No response

project_cleanup: removal of folder should respect the same time constraint used for projects

TL;DR

A folder should only be remove if it is older that the resource creation cut off time

Expected behavior

an empty folder is only deleted if it was created before the resource creation cut off time

Observed behavior

between the time a folder was created and another folder was created under it the first folder was removed. the frist folder was a few minutes old.

 Error: Error creating folder 'fldr-bootstrap' in 'folders/1083015950553': Error waiting for creating folder: Error code 9, message: Folder reservation failed for parent [folders/1083015950553], folder [] due to constraint: The destination resource has been deleted.
 
   with google_folder.bootstrap,
   on main.tf line 27, in resource "google_folder" "bootstrap":
   27: resource "google_folder" "bootstrap" {

Terraform Configuration

N/A

Terraform Version

N/A

Additional information

No response

Module broken

Last change from #26 (changing variable scheduler_job from any to object({name=string}) has broken the module.

I'm getting the error:

Error: Inconsistent conditional result types

  on .terraform/modules/slo-app-search-latency-64ms.slo_cloud_function/terraform-google-modules-terraform-google-scheduled-function-7aacc62/outputs.tf line 23, in output "scheduler_job":
  23:   value       = var.scheduler_job == null ? google_cloud_scheduler_job.job : var.scheduler_job
    |----------------
    | google_cloud_scheduler_job.job is tuple with 1 element
    | var.scheduler_job is null

The true and false result expressions must have consistent types. The given
expressions are tuple and object, respectively.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

github-actions
.github/workflows/go-lint.yaml
  • actions/checkout v4.1.7@692973e3d937129bcbf40652eb9f2f61becf3332
  • actions/setup-go v5.0.2@0a12ed9d6a96ab950c8f026ed9f722fe0da7ef32
  • golangci/golangci-lint-action v6.0.1@a4f60bb28d35aeee14e6880718e0c85ff1882e64
gomod
modules/project_cleanup/function_source/go.mod
  • go 1.21
  • cloud.google.com/go/asset v1.19.5
  • cloud.google.com/go/container v1.38.0
  • cloud.google.com/go/securitycenter v1.33.1
  • golang.org/x/net v0.27.0
  • golang.org/x/oauth2 v0.21.0
  • google.golang.org/api v0.189.0
test/integration/go.mod
  • go 1.22
  • go 1.22.5
  • github.com/GoogleCloudPlatform/cloud-foundation-toolkit/infra/blueprint-test v0.16.1
  • github.com/stretchr/testify v1.9.0
npm
examples/pubsub_scheduled/function_source/package.json
  • googleapis ^137.0.0
pip_requirements
examples/logs-slack-alerts/function_source/requirements.txt
  • google-cloud-bigquery ==3.23.1
  • requests ==2.32.2
regex
Makefile
  • cft/developer-tools 1.22
build/int.cloudbuild.yaml
  • cft/developer-tools 1.22
build/lint.cloudbuild.yaml
  • cft/developer-tools 1.22
terraform
examples/logs-slack-alerts/main.tf
  • terraform-google-modules/scheduled-function/google ~> 4.0
examples/logs-slack-alerts/versions.tf
  • hashicorp/terraform >= 0.13
examples/pubsub_scheduled/main.tf
  • terraform-google-modules/scheduled-function/google ~> 4.0
examples/pubsub_scheduled/versions.tf
  • hashicorp/terraform >= 0.13
examples/pubsub_scheduled_multiple/main.tf
  • terraform-google-modules/scheduled-function/google ~> 4.0
  • terraform-google-modules/scheduled-function/google ~> 4.0
examples/pubsub_scheduled_multiple/versions.tf
  • hashicorp/terraform >= 0.13
main.tf
  • terraform-google-modules/event-function/google ~> 3.0
  • terraform-google-modules/pubsub/google ~> 6.0
modules/project_cleanup/main.tf
modules/project_cleanup/versions.tf
  • google >= 3.53, < 6
  • hashicorp/terraform >= 0.13
test/setup/main.tf
  • terraform-google-modules/project-factory/google ~> 14
test/setup/versions.tf
  • google >= 3.53, < 6
  • google-beta >= 3.35, < 6
  • hashicorp/terraform >= 0.13
versions.tf
  • google >= 4.23, < 6
  • random >= 2.1, < 4.0
  • hashicorp/terraform >= 0.13

  • Check this box to trigger a request for Renovate to run again on this repository

Terraform apply always has an update

When re-running the Terraform module multiple times, there is always something to update.

Here is the update in-place that Terraform wants to re-do everytime:

# module.slo.module.slo-cloud-function.module.pubsub_topic.google_pubsub_topic.topic will be updated in-place
  ~ resource "google_pubsub_topic" "topic" {
        id      = "projects/rnm-shared-slo-6c95/topics/slo-svc-pubsub-pubsub-ack"
        labels  = {}
        name    = "slo-svc-pubsub-pubsub-ack"
        project = "rnm-shared-slo-6c95"


      - message_storage_policy {
          - allowed_persistence_regions = [
              - "asia-east1",
              - "asia-east2",
              - "asia-northeast1",
              - "asia-northeast2",
              - "asia-south1",
              - "asia-southeast1",
              - "australia-southeast1",
              - "europe-north1",
              - "europe-west1",
              - "europe-west2",
              - "europe-west3",
              - "europe-west4",
              - "europe-west5",
              - "europe-west6",
              - "northamerica-northeast1",
              - "southamerica-east1",
              - "us-central1",
              - "us-central2",
              - "us-east1",
              - "us-east4",
              - "us-west1",
              - "us-west2",
            ] -> null
        }
    }

Maybe we can set a default for the allowed_persistence_regions list so that Terraform doesn't trigger an update everytime ?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.