Giter VIP home page Giter VIP logo

srevinsaju / togomak Goto Github PK

View Code? Open in Web Editor NEW
143.0 3.0 4.0 5.45 MB

A declarative pipeline orchestrator with the magic of HCL as a configuration language, inspired from Terraform's architecture.

Home Page: https://togomak.srev.in

License: Mozilla Public License 2.0

Go 96.99% Dockerfile 0.28% HCL 2.73%
continuous-delivery continuous-integration hacktoberfest cicd concurrency go golang hcl orchestration terraform

togomak's Introduction

togomak

GitHub go.mod Go version (branch) codecov

Screenshot of togomak build pipeline for togomak using togomak, dogfooding?

togomak is a declarative pipeline orchestration tool, powered by HashiCorp Configuration Language (HCL) built on top of Terraform's architecture and features. It supports modules, custom HCL expressions.

togomak {
  version = 2
}
stage "hello" {
  script = "echo hello world"
}
$ togomak
0000 • togomak (version=dev)
0000 • [stage.hello]  hello world
0000 • took 2ms

Contents

Installation

Download a compatible binary for your operating system and architecture from Releases. The v2.x tag has the latest, bleeding edge features which are more relevant.

Or, using go:

go install github.com/srevinsaju/togomak/v2@latest

Building and installing from source

  1. Clone the repository, https://github.com/srevinsaju/togomak
  2. cd cmd/togomak
go build .
./togomak

Docker containers

Consider using the ghcr.io/srevinsaju/togomak:v2alpine image. Other flavors are available are available here.

Getting Started

togomak {
  version = 2
}

locals {
  validate = toset(["vet", "fmt", "test"])
}

variable "service" {
  type = string
  description = "Name of the service"
}

stage "build" {
  depends_on = [stage.validate]
  script = "echo Running go build ... for ${var.service}"
  post_hook {
    stage {
      script = "echo send slack message, build completed for ${upper(var.service)} with status: ${this.status}"
    }
  }
}

stage "validate" {
  for_each = local.validate
  script = "echo Running go ${each.value} ... for ${var.service}"
}

Screenshot of the above execution

Installation

Check out the releases page for the v2.0.0-alpha.* release binaries, and other pre-built packages for your desired platform.

Building from Source

cd cmd/togomak 
go build

Building using togomak (what!)

togomak

Features

In togomak, a stage is a single independent task, and a module is a group of stages. Here is a non-exhaustive list of features. See the work in progress documentation or examples for a list of examples. These examples also run as part of an integration test, using tests/togomak.hcl.

  • Concurrency: All stages and modules run in parallel by default.
  • Modular: Create reusable parts of your CI/CD pipeline and use them from git, https, s3 buckets or gcs buckets. 
    # modules/togomak.hcl
...
variable "name" {
  type = string
  description = "Name of person"
}
stage "hello" {
  script = "echo ${var.name}"
}
    # togomak.hcl
...
module {
  source = "./modules"
  name = "srev"
}
  • Conditional: Run stages or modules by specifying a HCL expression in if meta-arguments. 
    stage "hello" {
  if = env("USER") == "builder"
  ...
}
  • Built on top of Terraform's Architecture: supports most of Terraform's functions, block types, language, syntax and expression, keeping it a low learning curve for new and experienced users alike. Supports many familiar features like
    • local.<name> and locals {}
    • var.<name> and variable {}
    • Functions like sum(), flatten(), toset(), upper(), fileset(), setunion() and so on
    • for_each to iterate over a local, var to perform a stage or a module over different configurations.
  • Lifecycles and Rule Engine: Configure how your pipeline behaves when you type togomak deploy or togomak build, or when you would like to allow-list a specific stage with togomak deploy +stage.some_stage and block a specific stage with togomak build ^stage.slack_hook. See Usage on how togomak uses them. 
    ❯ togomak
0000 • togomak (version=dev)
0000 • [stage.alice]
0000 • [stage.bob]  skipped
0000 • [stage.eve]
0000 • [stage.eve]  hello im default
0000 • [stage.alice]  hello im default, deploy
0000 • took 2ms
❯ togomak deploy
0000 • togomak (version=dev)
0000 • [stage.alice]
0000 • [stage.bob]
0000 • [stage.eve]  skipped
0000 • [stage.bob]  hello im deploy
0000 • [stage.alice]  hello im default, deploy
0000 • took 2ms
❯ togomak all
0000 • togomak (version=dev)
0000 • [stage.bob]
0000 • [stage.eve]
0000 • [stage.alice]
0000 • [stage.eve]  hello im default
0000 • [stage.alice]  hello im default, deploy
0000 • [stage.bob]  hello im deploy
0000 • took 2ms
  • Pre and Post Hooks: Runs commands or modules before and after the execution of a stage as well as an entire pipeline.
  • Query Engine: Pass custom hcl expressions to togomak to choose which stages can run, and which cannot using the --query parameter on the command line.
  • Comprehensive terraform-like error diagnostics with references to contents of the file where the error originates from.
  • Terraform Support: Use terraform data blocks (or resource) blocks as data sources in your CI/CD pipeline. See examples/terraform for an example.

And don't forget to format your CICD files with togomak fmt.

Contributing

Contributions are welcome, and encouraged. Please check out the contributing guide for more information.

License

togomak is licensed under the MPL License v2.0

togomak's People

Contributors

dependabot[bot] avatar srevinsaju avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar

Forkers

sorcero dorucioclea doytsujin

togomak's Issues

use sensible defaults for the git provider when no data is found 

INFO[0147] [stage=deploy] ERRO took 2m25.238s
INFO[0147] [stage=deploy] Error: Invalid template interpolation value
INFO[0147] [stage=deploy] 
INFO[0147] [stage=deploy]   on /builds/xxx/.togomak/pipelines/tmp/e29e2611-1ea6-4fa4-b002-79788588f2302885812944/deploy/publish.hcl line 81, in stage "echo":
INFO[0147] [stage=deploy]   79:   script = <<EOT
INFO[0147] [stage=deploy]   80:   echo "branch=${data.env.branch.value}"
INFO[0147] [stage=deploy]   81:   echo "commits_since_last_tag=${data.git.this.commits_since_last_tag}"
INFO[0147] [stage=deploy]   82:   echo "last_tag=${data.git.this.last_tag}"
INFO[0147] [stage=deploy]   83:   echo "timestamp=${local.timestamp}"
INFO[0147] [stage=deploy]   84:   echo "gcp: $GOOGLE_APPLICATION_CREDENTIALS"
INFO[0147] [stage=deploy]   85:   EOT
INFO[0147] [stage=deploy] 
INFO[0147] [stage=deploy] with data.git.this.commits_since_last_tag set to null.
INFO[0147] [stage=deploy] 
INFO[0147] [stage=deploy] The expression result is null. Cannot include a null value in a string template.
INFO[0147] [stage=deploy] 
ERRO[0147] <nil>: failed to run command (stage.deploy); exit status 1

if fields don't exist, return sensible defaults instead of null.

togmak should fail if the tag specified on the git data source does not exist

the following config should throw an error b/c the tag doesn't exist:

data git cicd {
  url   = "..."
  tag = "asdf"
  files = [
    "python/togomak/build.hcl", "python/togomak/docs.hcl", "python/togomak/publish.hcl"
  ]
}

output:

togomak root +build
INFO[0000] togomak (version=1.2.1)
INFO[0000] [provider=git] Still pulling git repo data.git... [0s elapsed]
INFO[0001] [provider=git] Completed pulling git repo data.git in [1s elapsed]

Panic when Ctrl+C on watchdog 

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x11eabf8]

goroutine 43 [running]:
github.com/srevinsaju/togomak/v1/internal/ci.(*Conductor).Logger(...)
        /home/srev/src/github.com/srevinsaju/togomak/internal/ci/conductor.go:197
github.com/srevinsaju/togomak/v1/internal/ci.(*Stage).Terminate(0xc00127c000, 0x0, 0x0)
        /home/srev/src/github.com/srevinsaju/togomak/internal/ci/stage_terminate.go:15 +0x58
github.com/srevinsaju/togomak/v1/internal/ci.(*Handler).Interrupt(0xc0004653b0)
        /home/srev/src/github.com/srevinsaju/togomak/internal/ci/handler.go:303 +0x4d3
created by github.com/srevinsaju/togomak/v1/internal/ci.StartHandlers in goroutine 42
        /home/srev/src/github.com/srevinsaju/togomak/internal/ci/pipeline_run.go:19 +0x189

togomak doesn't abort after exit when multiple stages run 

INFO[0000] [provider=git] Still pulling git repo data.git... [0s elapsed]
INFO[0001] [provider=git] Completed pulling git repo data.git in [2s elapsed]
INFO[0001] [+] deploy
INFO[0001] [+] build
INFO[0001] [+] docs
INFO[0001] [stage=build] INFO [+] ci-init (skipped)
INFO[0001] [stage=build] INFO [+] local-init
INFO[0001] [stage=docs] INFO [+] init (skipped)
INFO[0001] [stage=docs] INFO [+] docs (skipped)
INFO[0001] [stage=docs] INFO took 2ms
INFO[0001] [stage=deploy] INFO [+] version
INFO[0001] [stage=deploy] INFO [provider=git] Still pulling git repo data.git... [0s elapsed]
INFO[0001] [stage=deploy] INFO [stage=version] git repo is dirty
INFO[0001] [stage=deploy] ERRO <nil>: failed to run command (stage.version); exit status 1
INFO[0002] [stage=deploy] INFO [provider=git] Completed pulling git repo data.git in [0s elapsed]
INFO[0002] [stage=deploy] ERRO took 410ms
INFO[0002] [stage=deploy] Error: failed to run command (stage.version)
INFO[0002] [stage=deploy] 
INFO[0002] [stage=deploy] exit status 1
INFO[0002] [stage=deploy] 
ERRO[0002] <nil>: failed to run command (stage.deploy); exit status 1
INFO[0002] [stage=build] WARN [stage=local-init] Warning: In a future version of Poetry, PyPI will be disabled automatically if at least one custom source is configured with another priority than 'explicit'. In order to avoid a breaking change and make your pyproject.toml forward compatible, add PyPI explicitly via 'poetry source add pypi'. By the way, this has the advantage that you can set the priority of PyPI as with any other source.
INFO[0003] [stage=build] INFO [stage=local-init] Installing dependencies from lock file
INFO[0004] [stage=build] WARN [stage=local-init] Warning: poetry.lock is not consistent with pyproject.toml. You may be getting improper dependencies. Run `poetry lock [--no-update]` to fix it.```

this should have aborted after the `exit status 1`

Panic when required argument was not defined 

0002 • [module.deploy]
panic: /tmp/togomak790217308/modules/deploy/variables.hcl:5,31-31: Missing required argument; The argument "key" is required, but no definition was found.

goroutine 32 [running]:
github.com/srevinsaju/togomak/v1/internal/blocks/data.Variables({0x1a9a1c0, 0xc0013114a0}, {0x1a93d50, 0xc0012e53f0})
        /home/srev/src/github.com/srevinsaju/togomak/internal/blocks/data/provider.go:62 +0x1fa
github.com/srevinsaju/togomak/v1/internal/ci.(*Data).Variables(0xc001334f78)
        /home/srev/src/github.com/srevinsaju/togomak/internal/ci/data_hcl.go:16 +0xff
github.com/srevinsaju/togomak/v1/internal/ci.GraphTopoSort(0xc000191b80, 0xc0010d9d00)
        /home/srev/src/github.com/srevinsaju/togomak/internal/ci/graph.go:94 +0x85b
github.com/srevinsaju/togomak/v1/internal/ci.(*Pipeline).Run(0xc000191b80?, 0xc000191b80)
        /home/srev/src/github.com/srevinsaju/togomak/internal/ci/pipeline_run.go:66 +0x565
github.com/srevinsaju/togomak/v1/internal/ci.(*Module).run(0xc001188fc0, 0xc000cf4c60, {0xc0000481e0, 0x4d}, 0xc00172ca50, {0xc00158b010, 0x2, 0x2})
        /home/srev/src/github.com/srevinsaju/togomak/internal/ci/module_run.go:238 +0x105a
github.com/srevinsaju/togomak/v1/internal/ci.(*Module).Run(0xc001188fc0, 0xc000cf4c60, {0xc00158b010, 0x2, 0x2})
        /home/srev/src/github.com/srevinsaju/togomak/internal/ci/module_run.go:82 +0xa56
github.com/srevinsaju/togomak/v1/internal/ci.BlockRunWithRetries(0x0?, {0xc000014d00, 0xd}, {0x1a9f640, 0xc001188fc0}, 0xc000cf8460, {0x1acd760?, 0xc000cc4800?}, {0xc00158b010, 0x2, ...})
        /home/srev/src/github.com/srevinsaju/togomak/internal/ci/run.go:16 +0x143
created by github.com/srevinsaju/togomak/v1/internal/ci.(*Pipeline).Run in goroutine 1
        /home/srev/src/github.com/srevinsaju/togomak/internal/ci/pipeline_run.go:127 +0xcf

we should be providing a good error message instead.

concurrent map read/write error 

WARN[0001] [stage=deploy] fatal error: concurrent map read and map write
INFO[0001] [stage=deploy] INFO [+] init
WARN[0001] [stage=deploy] 
WARN[0001] [stage=deploy] goroutine 69 [running]:
WARN[0001] [stage=deploy] github.com/hashicorp/hcl/v2.Traversal.TraverseAbs({0xc000520780?, 0x14c2ec8?, 0xc0000440e5?}, 0xc00050f8f0)
WARN[0001] [stage=deploy] 	/home/runner/go/pkg/mod/github.com/hashicorp/hcl/[email protected]/traversal.go:82 +0x2c5
WARN[0001] [stage=deploy] github.com/hashicorp/hcl/v2/hclsyntax.(*ScopeTraversalExpr).Value(0xc0010c19e0, 0xc00050f8f0)
WARN[0001] [stage=deploy] 	/home/runner/go/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclsyntax/expression.go:139 +0x3b
WARN[0001] [stage=deploy] github.com/srevinsaju/togomak/v1/pkg/blocks/data.(*EnvProvider).DecodeBody(0xc00056a900, {0x14c24b8, 0xc0010d86e0})
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/pkg/blocks/data/env.go:98 +0x2fb
WARN[0001] [stage=deploy] github.com/srevinsaju/togomak/v1/pkg/ci.Data.Run({{0xc00104c424, 0x3}, {0xc00104c427, 0x3}, {0x0, 0x0}, {0x0, 0x0}, {0x14c24b8, 0xc0010d86e0}}, ...)
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/pkg/ci/data_run.go:43 +0x30a
WARN[0001] [stage=deploy] github.com/srevinsaju/togomak/v1/pkg/orchestra.Orchestra.func1({0xc0010df030, 0xc})
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/pkg/orchestra/orchestra.go:306 +0xce
WARN[0001] [stage=deploy] created by github.com/srevinsaju/togomak/v1/pkg/orchestra.Orchestra
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/pkg/orchestra/orchestra.go:305 +0x1f3e
WARN[0001] [stage=deploy] 
WARN[0001] [stage=deploy] goroutine 1 [semacquire]:
WARN[0001] [stage=deploy] sync.runtime_Semacquire(0xc000568380?)
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/runtime/sema.go:62 +0x27
WARN[0001] [stage=deploy] sync.(*WaitGroup).Wait(0xc0000ca180?)
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/sync/waitgroup.go:116 +0x4b
WARN[0001] [stage=deploy] github.com/srevinsaju/togomak/v1/pkg/orchestra.Orchestra({{0xc00003c274, 0x21}, {0x7fffbe2313d2, 0x21}, 0x1, 0x1, 0x1, {0x7fffbe23145d, 0x6}, {0x0, ...}, ...})
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/pkg/orchestra/orchestra.go:380 +0x15b3
WARN[0001] [stage=deploy] main.run(0xc0000c9340?)
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/cmd/togomak/main.go:211 +0x98
WARN[0001] [stage=deploy] github.com/urfave/cli/v2.(*Command).Run(0xc0000c9340, 0xc0002e7cc0, {0xc0000340a0, 0xa, 0xa})
WARN[0001] [stage=deploy] 	/home/runner/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:274 +0x9eb
WARN[0001] [stage=deploy] github.com/urfave/cli/v2.(*App).RunContext(0xc000fea1e0, {0x14c22f8?, 0xc000044088}, {0xc0000340a0, 0xa, 0xa})
WARN[0001] [stage=deploy] 	/home/runner/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:332 +0x616
WARN[0001] [stage=deploy] github.com/urfave/cli/v2.(*App).Run(...)
WARN[0001] [stage=deploy] 	/home/runner/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:309
WARN[0001] [stage=deploy] main.main()
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/cmd/togomak/main.go:142 +0x118f
WARN[0001] [stage=deploy] 
WARN[0001] [stage=deploy] goroutine 6 [select]:
WARN[0001] [stage=deploy] code.gitea.io/gitea/modules/log.(*ChannelledLog).Start(0xc0002b96c0)
WARN[0001] [stage=deploy] 	/home/runner/go/pkg/mod/code.gitea.io/[email protected]/modules/log/event.go:79 +0xd6
WARN[0001] [stage=deploy] created by code.gitea.io/gitea/modules/log.NewChannelledLog
WARN[0001] [stage=deploy] 	/home/runner/go/pkg/mod/code.gitea.io/[email protected]/modules/log/event.go:68 +0x385
WARN[0001] [stage=deploy] 
WARN[0001] [stage=deploy] goroutine 7 [select]:
WARN[0001] [stage=deploy] code.gitea.io/gitea/modules/log.(*MultiChannelledLog).Start(0xc0000d4120)
WARN[0001] [stage=deploy] 	/home/runner/go/pkg/mod/code.gitea.io/[email protected]/modules/log/event.go:340 +0x2bc
WARN[0001] [stage=deploy] created by code.gitea.io/gitea/modules/log.(*MultiChannelledLog).AddLogger
WARN[0001] [stage=deploy] 	/home/runner/go/pkg/mod/code.gitea.io/[email protected]/modules/log/event.go:221 +0x176
WARN[0001] [stage=deploy] 
WARN[0001] [stage=deploy] goroutine 18 [syscall]:
WARN[0001] [stage=deploy] os/signal.signal_recv()
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/runtime/sigqueue.go:152 +0x2f
WARN[0001] [stage=deploy] os/signal.loop()
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/os/signal/signal_unix.go:23 +0x19
WARN[0001] [stage=deploy] created by os/signal.Notify.func1.1
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/os/signal/signal.go:151 +0x2a
WARN[0001] [stage=deploy] 
WARN[0001] [stage=deploy] goroutine 33 [select]:
WARN[0001] [stage=deploy] github.com/srevinsaju/togomak/v1/pkg/orchestra.InterruptHandler({0x14c2368, 0xc0010e2fc0}, 0xc001048c40, 0xc0010ea4e0, 0xc000011170)
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/pkg/orchestra/interrupt.go:16 +0xe5
WARN[0001] [stage=deploy] created by github.com/srevinsaju/togomak/v1/pkg/orchestra.Orchestra
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/pkg/orchestra/orchestra.go:180 +0x101d
WARN[0001] [stage=deploy] 
WARN[0001] [stage=deploy] goroutine 34 [select]:
WARN[0001] [stage=deploy] github.com/srevinsaju/togomak/v1/pkg/orchestra.KillHandler({0x14c2368, 0xc0010e2fc0}, 0xc001048c40, 0xc0010ea540, 0xc000011170)
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/pkg/orchestra/interrupt.go:56 +0xe5
WARN[0001] [stage=deploy] created by github.com/srevinsaju/togomak/v1/pkg/orchestra.Orchestra
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/pkg/orchestra/orchestra.go:181 +0x10e5
WARN[0001] [stage=deploy] 
WARN[0001] [stage=deploy] goroutine 35 [chan receive]:
WARN[0001] [stage=deploy] github.com/srevinsaju/togomak/v1/pkg/orchestra.daemonKiller({0x14c2368, 0xc0010e2fc0}, 0x0?, 0xc000011188)
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/pkg/orchestra/orchestra.go:433 +0x1c5
WARN[0001] [stage=deploy] created by github.com/srevinsaju/togomak/v1/pkg/orchestra.Orchestra
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/pkg/orchestra/orchestra.go:182 +0x118d
WARN[0001] [stage=deploy] 
WARN[0001] [stage=deploy] goroutine 67 [runnable]:
WARN[0001] [stage=deploy] github.com/hashicorp/hcl/v2/hclsyntax.Variables.func1({0xc000520640?, 0x4?, 0x4?})
WARN[0001] [stage=deploy] 	/home/runner/go/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclsyntax/variables.go:18 +0xec
WARN[0001] [stage=deploy] github.com/hashicorp/hcl/v2/hclsyntax.(*variablesWalker).Enter(0xc0002c5200, {0x14bebb0?, 0xc0010c1200?})
WARN[0001] [stage=deploy] 	/home/runner/go/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclsyntax/variables.go:51 +0x277
WARN[0001] [stage=deploy] github.com/hashicorp/hcl/v2/hclsyntax.Walk({0x14bebb0, 0xc0010c1200}, {0x14bebd8, 0xc0002c5200})
WARN[0001] [stage=deploy] 	/home/runner/go/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclsyntax/walk.go:37 +0x5d
WARN[0001] [stage=deploy] github.com/hashicorp/hcl/v2/hclsyntax.Variables({0x14c38c0?, 0xc0010c1200})
WARN[0001] [stage=deploy] 	/home/runner/go/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclsyntax/variables.go:23 +0xcc
WARN[0001] [stage=deploy] github.com/hashicorp/hcl/v2/hclsyntax.(*ScopeTraversalExpr).Variables(0xc001103460?)
WARN[0001] [stage=deploy] 	/home/runner/go/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclsyntax/expression_vars.go:54 +0x25
WARN[0001] [stage=deploy] github.com/srevinsaju/togomak/v1/pkg/blocks/data.Variables({0x14c6280, 0xc0010ebb00}, {0x14c24b8, 0xc0010d8630})
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/pkg/blocks/data/provider.go:36 +0x17f
WARN[0001] [stage=deploy] github.com/srevinsaju/togomak/v1/pkg/ci.Data.Variables({{0xc00104c3f4, 0x3}, {0xc00104c3f8, 0x6}, {0x0, 0x0}, {0x0, 0x0}, {0x14c24b8, 0xc0010d8630}})
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/pkg/ci/data_hcl.go:16 +0xf9
WARN[0001] [stage=deploy] github.com/srevinsaju/togomak/v1/pkg/ci.Data.Run({{0xc00104c3f4, 0x3}, {0xc00104c3f8, 0x6}, {0x0, 0x0}, {0x0, 0x0}, {0x14c24b8, 0xc0010d8630}}, ...)
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/pkg/ci/data_run.go:90 +0xef8
WARN[0001] [stage=deploy] github.com/srevinsaju/togomak/v1/pkg/orchestra.Orchestra.func1({0xc0010df130, 0xf})
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/pkg/orchestra/orchestra.go:306 +0xce
WARN[0001] [stage=deploy] created by github.com/srevinsaju/togomak/v1/pkg/orchestra.Orchestra
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/pkg/orchestra/orchestra.go:305 +0x1f3e
WARN[0001] [stage=deploy] 
WARN[0001] [stage=deploy] goroutine 42 [select]:
WARN[0001] [stage=deploy] io.(*pipe).read(0xc00056a360, {0xc00119c000, 0x10000, 0x4215c5?})
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/io/pipe.go:57 +0xb1
WARN[0001] [stage=deploy] io.(*PipeReader).Read(0x2085b10?, {0xc00119c000?, 0xc000578000?, 0x1000000014c2368?})
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/io/pipe.go:136 +0x25
WARN[0001] [stage=deploy] bufio.(*Scanner).Scan(0xc00016cf28)
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/bufio/scan.go:214 +0x876
WARN[0001] [stage=deploy] github.com/sirupsen/logrus.(*Entry).writerScanner(0xdfe88e?, 0xc000570020, 0xc0005662d0)
WARN[0001] [stage=deploy] 	/home/runner/go/pkg/mod/github.com/srevinsaju/[email protected]/writer.go:79 +0x11f
WARN[0001] [stage=deploy] created by github.com/sirupsen/logrus.(*Entry).WriterLevel
WARN[0001] [stage=deploy] 	/home/runner/go/pkg/mod/github.com/srevinsaju/[email protected]/writer.go:52 +0x3d1
WARN[0001] [stage=deploy] 
WARN[0001] [stage=deploy] goroutine 43 [select]:
WARN[0001] [stage=deploy] io.(*pipe).read(0xc00056a3c0, {0xc00122e189, 0xfe77, 0x14bf650?})
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/io/pipe.go:57 +0xb1
WARN[0001] [stage=deploy] io.(*PipeReader).Read(0x0?, {0xc00122e189?, 0xc000552c90?, 0xc000578480?})
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/io/pipe.go:136 +0x25
WARN[0001] [stage=deploy] bufio.(*Scanner).Scan(0xc001101f28)
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/bufio/scan.go:214 +0x876
WARN[0001] [stage=deploy] github.com/sirupsen/logrus.(*Entry).writerScanner(0x0?, 0xc000570030, 0xc0005662e0)
WARN[0001] [stage=deploy] 	/home/runner/go/pkg/mod/github.com/srevinsaju/[email protected]/writer.go:79 +0x11f
WARN[0001] [stage=deploy] created by github.com/sirupsen/logrus.(*Entry).WriterLevel
WARN[0001] [stage=deploy] 	/home/runner/go/pkg/mod/github.com/srevinsaju/[email protected]/writer.go:52 +0x3d1
WARN[0001] [stage=deploy] 
WARN[0001] [stage=deploy] goroutine 68 [runnable]:
WARN[0001] [stage=deploy] syscall.Syscall6(0xc001224558?, 0x14?, 0x7fb60655ff08?, 0x7fb62db14f18?, 0xd0?, 0xc000076c00?, 0xc001213450?)
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/syscall/syscall_linux.go:91 +0x36
WARN[0001] [stage=deploy] syscall.fstatat(0x0?, {0xc001224558?, 0x14?}, 0xc001213488, 0x0?)
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/syscall/zsyscall_linux_amd64.go:1443 +0x9a
WARN[0001] [stage=deploy] syscall.Stat(...)
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/syscall/syscall_linux_amd64.go:66
WARN[0001] [stage=deploy] os.statNolog.func1(...)
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/os/stat_unix.go:32
WARN[0001] [stage=deploy] os.ignoringEINTR(...)
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/os/file_posix.go:245
WARN[0001] [stage=deploy] os.statNolog({0xc001224558, 0x14})
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/os/stat_unix.go:31 +0x58
WARN[0001] [stage=deploy] os.Stat({0xc001224558, 0x14})
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/os/stat.go:13 +0x34
WARN[0001] [stage=deploy] os/exec.findExecutable({0xc001224558, 0x14})
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/os/exec/lp_unix.go:23 +0x27
WARN[0001] [stage=deploy] os/exec.LookPath({0xfc1893, 0x4})
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/os/exec/lp_unix.go:71 +0x205
WARN[0001] [stage=deploy] os/exec.Command({0xfc1893, 0x4}, {0xc00120b380, 0x3, 0xc?})
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/os/exec/exec.go:419 +0x225
WARN[0001] [stage=deploy] os/exec.CommandContext({0x14c2368, 0xc0010e2fc0}, {0xfc1893?, 0x0?}, {0xc00120b380?, 0x419871?, 0x208d740?})
WARN[0001] [stage=deploy] 	/opt/hostedtoolcache/go/1.20.5/x64/src/os/exec/exec.go:446 +0x53
WARN[0001] [stage=deploy] github.com/srevinsaju/togomak/v1/pkg/ci.(*Stage).Run(0xc000578b40, {0x14c2368, 0xc0010e2fc0})
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/pkg/ci/stage_run.go:378 +0x1fa5
WARN[0001] [stage=deploy] github.com/srevinsaju/togomak/v1/pkg/orchestra.Orchestra.func1({0xc0010df1b0, 0xa})
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/pkg/orchestra/orchestra.go:306 +0xce
WARN[0001] [stage=deploy] created by github.com/srevinsaju/togomak/v1/pkg/orchestra.Orchestra
WARN[0001] [stage=deploy] 	/home/runner/work/togomak/togomak/pkg/orchestra/orchestra.go:305 +0x1f3e
ERRO[0001] <nil>: failed to run command (stage.deploy); exit status 2```

data env blocks without default throw an error

this doesn't work

data env version {
  key     = "CI_PIPELINE_VERSION"
}

INFO[0000] togomak (version=1.2.1)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x196f888]

goroutine 27 [running]:
github.com/srevinsaju/togomak/v1/pkg/blocks/data.(*EnvProvider).DecodeBody(0xc00134a180, {0x20d01f8, 0xc00148a0b0})
	/home/runner/work/togomak/togomak/pkg/blocks/data/env.go:98 +0x2e8
github.com/srevinsaju/togomak/v1/pkg/ci.Data.Run({{0xc00146bc04, 0x3}, {0xc00146bc07, 0x7}, {0x0, 0x0}, {0x0, 0x0}, {0x20d01f8, 0xc00148a0b0}}, ...)
	/home/runner/work/togomak/togomak/pkg/ci/data_run.go:43 +0x30a
github.com/srevinsaju/togomak/v1/pkg/orchestra.Orchestra.func1({0xc001485080, 0x10})
	/home/runner/work/togomak/togomak/pkg/orchestra/orchestra.go:306 +0xce
created by github.com/srevinsaju/togomak/v1/pkg/orchestra.Orchestra
	/home/runner/work/togomak/togomak/pkg/orchestra/orchestra.go:305 +0x1f3e

if i add

default = ""

it works

add ability to import data sources

it's not possible to access a file in a git repository that is comprised of other files in the repo.

  version = 1
}

## data

data env version {
  key     = "CI_PIPELINE_VERSION"
  default = ""
}

data git cicd {
  url   = "..."
  tag   = data.env.version.value
  files = [
    "python/togomak/build.hcl", "python/togomak/docs.hcl", "python/togomak/publish.hcl"
  ]
}

## macros

macro build {
  files = {
    "build.hcl" : data.git.cicd.files["python/togomak/build.hcl"]
  }
}

macro docs {
  files = {
    "docs.hcl" : data.git.cicd.files["python/togomak/docs.hcl"]
  }
}

macro deploy {
  files = {
    "publish.hcl" : data.git.cicd.files["python/togomak/publish.hcl"]
  }
}

## stages

stage build {
  use {
    macro = macro.build
  }
}

stage docs {
  use {
    macro = macro.docs
  }
}

stage deploy {
  use {
    macro = macro.deploy
  }
}

i'd like to condense most of this into a single hcl file store in a separate repository that can then be imported as a data source by the caller. there needs to be a way for the imported file to be able to access the other files it imports.

collapse macro/stage logging

it would be nice if logging could be collapsed into something like macro.stage instead of the current format that duplicates the level, etc.

allow git provider to pull via a ref

i'd like to be able to define a ref that could be used as part of check out in the provider. this would allow me to specify a branch or a tag in the environment and not have to distinguish what's being used in the hcl configuration.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.