Giter VIP home page Giter VIP logo

frep's Introduction

Build Status License

frep

Generate file using template from environment, arguments, json/yaml/toml config files.

NAME:
   frep - Generate file using template

USAGE:
   frep [options] input-file[:output-file] ...

VERSION:
   1.3.x

AUTHORS:
   Guoqiang Chen <[email protected]>

OPTIONS:
   -e, --env name=value    set variable name=value, can be passed multiple times
       --json jsonstring   load variables from json object string
       --load file         load variables from json/yaml/toml file
       --no-sys-env        exclude system environments, default false
       --overwrite         overwrite if destination file exists
       --dryrun            just output result to console instead of file
       --strict            exit on any error during template processing
       --delims value      template tag delimiters (default: {{:}})
       --missing           handling of missing vars, one of: default/invalid, zero, error (default: default)
       --help              print this usage
       --version           print version information

EXAMPLES:
   frep nginx.conf.in -e webroot=/usr/share/nginx/html -e port=8080
   frep nginx.conf.in:/etc/nginx.conf -e webroot=/usr/share/nginx/html -e port=8080
   frep nginx.conf.in --json '{"webroot": "/usr/share/nginx/html", "port": 8080}'
   frep nginx.conf.in --load config.json --overwrite
   echo "{{ .Env.PATH }}"  | frep -

Downloads

v1.3.13 Release: https://github.com/subchen/frep/releases/tag/v1.3.13

  • Linux

    curl -fSL https://github.com/subchen/frep/releases/download/v1.3.13/frep-1.3.13-linux-amd64 -o /usr/local/bin/frep
    chmod +x /usr/local/bin/frep
    
    # centos / redhat
    yum install https://github.com/subchen/frep/releases/download/v1.3.13/frep-1.3.13-204.x86_64.rpm
    
    # ubuntu
    curl -fSL https://github.com/subchen/frep/releases/download/v1.3.13/frep_1.3.13-204_amd64.deb -o frep_1.3.13-204_amd64.deb
    dpkg -i frep_1.3.13-204_amd64.deb
    
  • macOS

    brew install subchen/tap/frep
    
  • Windows

    wget https://github.com/subchen/frep/releases/download/v1.3.13/frep-1.3.13-windows-amd64.exe
    

Docker

You can run frep using docker container

docker run -it --rm subchen/frep --help

Examples

Load template variables

  • Load from environment

    export webroot=/usr/share/nginx/html
    export port=8080
    frep nginx.conf.in
    
  • Load from arguments

    frep nginx.conf.in -e webroot=/usr/share/nginx/html -e port=8080
    
  • Load from JSON String

    frep nginx.conf.in --json '{"webroot": "/usr/share/nginx/html", "port": 8080}'
    
  • Load from JSON file

    cat > config.json << EOF
    {
      "webroot": "/usr/share/nginx/html",
      "port": 8080,
      "servers": [
        "127.0.0.1:8081",
        "127.0.0.1:8082"
      ]
    }
    EOF
    
    frep nginx.conf.in --load config.json
    
  • Load from YAML file

    cat > config.yaml << EOF
    webroot: /usr/share/nginx/html
    port: 8080
    servers:
      - 127.0.0.1:8081
      - 127.0.0.1:8082
    EOF
    
    frep nginx.conf.in --load config.yaml
    
  • Load from TOML file

    cat > config.toml << EOF
    webroot = /usr/share/nginx/html
    port = 8080
    servers = [
       "127.0.0.1:8081",
       "127.0.0.1:8082"
    ]
    EOF
    
    frep nginx.conf.in --load config.toml
    

Input/Output

  • Input from file

    // input file: nginx.conf
    frep nginx.conf.in
    
  • Input from console(stdin)

    // input from stdin pipe
    echo "{{ .Env.PATH }}" | frep -
    
  • Output to default file (Removed last file ext)

    // output file: nginx.conf
    frep nginx.conf.in --overwrite
    
  • Output to the specified file

    // output file: /etc/nginx.conf
    frep nginx.conf.in:/etc/nginx.conf --overwrite -e port=8080
    
  • Output to console(stdout)

    frep nginx.conf.in --dryrun
    frep nginx.conf.in:-
    
  • Output multiple files

    frep nginx.conf.in redis.conf.in ...
    

Template

Templates use Golang text/template.

You can access environment variables within a template

Env.PATH = {{ .Env.PATH }}

If your template file uses {{ and }} as part of it's syntax, you can change the template escape characters using the --delims.

frep --delims "<%:%>" ...

There are some built-in functions as well: Masterminds/sprig v2.22.0

More funcs added:

  • toJson
  • toYaml
  • toToml
  • toBool
  • fileSize
  • fileLastModified
  • fileGetBytes
  • fileGetString
  • fileExists
  • include
  • countRune
  • pipeline compatible regex functions from sprig
    • reReplaceAll
    • reReplaceAllLiteral
    • reSplit
  • awsSecret
  • awsParameterStore

Sample of nginx.conf.in

server {
    listen {{ .port }} default_server;

    root {{ .webroot | default "/usr/share/nginx/html" }};
    index index.html index.htm;

    location /api {
        {{ include "shared/log.nginx" | indent 8 | trim }}
        proxy_pass http://backend;
    }
}

upstream backend {
    ip_hash;
{{- range .servers }}
    server {{.}};
{{- end }}
}

Sample using secrets, first of all take into account that in order to use the secret functionality you need to have a proper AWS configuration in place and permissions enough to read secrets from AWS Secrets Manager. More details of how to configure AWSCLI can be found at https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html

Once you have all the requirements just create a template like this one:

# application.conf
mysql_host: {{ .mysql_host }}
mysql_user: {{ .mysql_user }}
mysql_pass: {{ awsSecret "application/mysql/password" }}

In above example mysql_host and mysql_user will be filled as usual by using frep config file or environment variables but mysql_pass will be fetch straight from AWS Secrets Manager by looking at secret name application/mysql/password

If you have multiple items in a single secret you can retrieve an specific key by specifying the key you want in template, for example:

# application.conf
mysql_host: {{ .mysql_host }}
mysql_user: {{ .mysql_user }}
mysql_pass: {{ awsSecret "application/mysql/password" }}

external_api_client: {{ awsSecret "application/external_api" "client_id" }}
external_api_secret: {{ awsSecret "application/external_api" "secret_key" }}

Sample using AWS Parameter Store, first of all take into account that in order to use the ssm functionality you need to have a proper AWS configuration in place and permissions enough to read parameters from AWS Parameter Store. More details of how to configure AWSCLI can be found at https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html

Once you have all the requirements just create a template like this one:

# application.conf
mysql_host: {{ .mysql_host }}
mysql_user: {{ .mysql_user }}
mysql_pass: {{ awsSecret "application/mysql/password" }}
mysql_dns: {{ awsParameterStore "application/mysql/dns" }}

In above example mysql_dns will be filled as usual by using frep config file or environment variables but mysql_pass will be fetch straight from AWS Parameter Store by looking at application/mysql/dns

SSM Limitation: You can get parameter from ParameterStore just in textplain.

frep's People

Contributors

ismferd avatar jasonrm avatar kevpie avatar kiorky avatar leonardolang avatar overdrive3000 avatar pavelmxfox avatar seth-priya avatar subchen avatar yingzhuo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

frep's Issues

arm64 支持

arm aarch64 架构软件支持,期待能编译运行。

使用 toml 作为配置文件 异常

frep nginx.conf.in --load config.toml
fatal: bad toml format, caused:

Near line 1 (last key parsed 'webroot'): expected value but found '/' instead

Security Update

Hello,
in the meantime thank you very much for the great tool.
Our container security scanning tool (Trivy) shows few security issues caused by the frep tool.
Could you update the package below?
Thank you very much in advance
Francesco

<style> </style>
Vulnerability: CVE-2020-29652
Severity: High
nvd: 7.5
redhat: 7.5
Package: golang.org/x/crypto
Current Version: v0.0.0-20190308221718-c2843e01d9a2
Fixed in Version: 0.0.0-20201216223049-8b5274cf687f
   
Vulnerability: CVE-2020-7919
Severity: High
nvd: 7.5
redhat: 7.5
Package: golang.org/x/crypto
Current Version: v0.0.0-20190308221718-c2843e01d9a2
Fixed in Version: 0.0.0-20200124225646-8b5121be2f68
   
Vulnerability: CVE-2020-9283
Severity: High
nvd: 7.5
redhat: 7.5
Package: golang.org/x/crypto
Current Version: v0.0.0-20190308221718-c2843e01d9a2
Fixed in Version: 0.0.0-20200220183623-bac4c82f6975
   
Vulnerability: CVE-2021-43565
Severity: High
nvd: 7.5
redhat: 7.5
Package: golang.org/x/crypto
Current Version: v0.0.0-20190308221718-c2843e01d9a2
Fixed in Version: 0.0.0-20211202192323-5770296d904e
   
Vulnerability: CVE-2022-27191
Severity: High
nvd: 7.5
redhat: 7.5
Package: golang.org/x/crypto
Current Version: v0.0.0-20190308221718-c2843e01d9a2
Fixed in Version: 0.0.0-20220314234659-1baeb1ce4c0b
   
Vulnerability: CVE-2020-8911
Severity: Medium
nvd: 5.6
redhat: 5.6
Package: github.com/aws/aws-sdk-go
Current Version: v1.30.29
Fixed in Version:  
   
Vulnerability: CVE-2019-11254
Severity: Medium
nvd: 6.5
redhat: 6.5
Package: github.com/go-yaml/yaml
Current Version: v2.1.0+incompatible
Fixed in Version:  
   
Vulnerability: CVE-2019-11840
Severity: Medium
nvd: 5.9
redhat: 5.9
Package: golang.org/x/crypto
Current Version: v0.0.0-20190308221718-c2843e01d9a2
Fixed in Version: 0.0.0-20190320223903-b7391e95e576
   
Vulnerability: CVE-2021-4235
Severity:  
nvd:  
redhat: github.com/go-yaml/yaml
Package:  
Current Version: v2.1.0+incompatibleNo
Fixed in Version:  
   
Vulnerability: CVE-2022-2582
Severity:  
nvd:  
redhat:  
Package: github.com/aws/aws-sdk-go
Current Version: v1.30.29
Fixed in Version:  
   
Vulnerability: CVE-2020-8912
Severity: Low
nvd: 2.5
redhat: 2.5
Package: github.com/aws/aws-sdk-go
Current Version: v1.30.29
Fixed in Version:  

Vulnerability: CVE-2020-29652
Severity: High
nvd: 7.5
redhat: 7.5
Package: golang.org/x/crypto
Current Version: v0.0.0-20190308221718-c2843e01d9a2
Fixed in Version:  0.0.0-20201216223049-8b5274cf687f

Vulnerability: CVE-2020-7919
Severity: High
nvd: 7.5
redhat: 7.5
Package: golang.org/x/crypto
Current Version: v0.0.0-20190308221718-c2843e01d9a2
Fixed in Version:  0.0.0-20200124225646-8b5121be2f68

Vulnerability: CVE-2020-9283
Severity: High
nvd: 7.5
redhat: 7.5
Package: golang.org/x/crypto
Current Version: v0.0.0-20190308221718-c2843e01d9a2
Fixed in Version:  0.0.0-20200220183623-bac4c82f6975

Vulnerability: CVE-2021-43565
Severity: High
nvd: 7.5
redhat: 7.5
Package: golang.org/x/crypto
Current Version: v0.0.0-20190308221718-c2843e01d9a2
Fixed in Version:  0.0.0-20211202192323-5770296d904e

Vulnerability: CVE-2022-27191
Severity: High
nvd: 7.5
redhat: 7.5
Package: golang.org/x/crypto
Current Version: v0.0.0-20190308221718-c2843e01d9a2
Fixed in Version:  0.0.0-20220314234659-1baeb1ce4c0b

Vulnerability: CVE-2020-8911
Severity: Medium
nvd: 5.6
redhat: 5.6
Package: github.com/aws/aws-sdk-go
Current Version: v1.30.29
Fixed in Version:

Vulnerability: CVE-2019-11254
Severity: Medium
nvd: 6.5
redhat: 6.5
Package: github.com/go-yaml/yaml
Current Version: v2.1.0+incompatible
Fixed in Version:

Vulnerability: CVE-2019-11840
Severity: Medium
nvd: 5.9
redhat: 5.9
Package: golang.org/x/crypto
Current Version: v0.0.0-20190308221718-c2843e01d9a2
Fixed in Version:  0.0.0-20190320223903-b7391e95e576

Vulnerability: CVE-2021-4235
Severity:
nvd:
redhat: github.com/go-yaml/yaml
Package:
Current Version: v2.1.0+incompatibleNo
Fixed in Version:

Vulnerability: CVE-2022-2582
Severity:
nvd:
redhat:
Package: github.com/aws/aws-sdk-go
Current Version: v1.30.29
Fixed in Version:

Vulnerability: CVE-2020-8912
Severity: Low
nvd: 2.5
redhat: 2.5
Package: github.com/aws/aws-sdk-go
Current Version: v1.30.29
Fixed in Version:

func of count runes of string

I do know this is NOT a bug of frep itself.

template :

{{ len "中文" }}

if i run frep like this:

frep /path/to/template --dryrun

i got :

6

So, I think chines users need a new func.

How about:

func CountRune(s string) int {
    return len([]rune(s))
}

and register is:

funcMap := template.FuncMap() {
    "countRune": CountRune,
    // other funcs
}

How to access environment inside the range?

Hi there,

I want to access .ENV.Secret inside the range. How do I do that?

{{- range (splitList "," .Env.XMPP_JVBS) }}
Component "{{.}}"
    component_secret "{{.ENV.Secret}}"
{{- end }}

Thanks!

ARM builds?

Any chance we could get ARM (v7 and arm64) builds here? Love the package BTW, great work!

Please preserve file permissions

File perms (specifically +x ) are lost when a template file is run though frep. Please preserve originating file permissions when writing out the rendered template.

Do not use environment variables by default - security issue

Please exclude support for environment variables in the templates (the .Env.* insertions according to the documentation). This can expose a lot of unintentional information (just run set in your shell to see what is available).

Preferred behaviour:

  1. Change default value of --no-sys-env to true (or rename option).
  2. Remove all support for system environments and only accept definitions from --env, --json or --load.

Simple work around is to add --no-sys-env parameter.

Issue a new release ?

as #10 is merged, would you mind publishing another release to have shiny portable linux binaries ? 😄

配置文件中的数组类型怎么判断最后一个元素呢

比如
环境变量:
{
host_ip": {
"127.0.0.1:8081",
"127.0.0.1:8082"
}
}
配置文件中的配置项:
host={{range .host_ip}}{{.}},{{end}}

用frep替换之后的结果是:
host=127.0.0.1:8081,127.0.0.1:8082,

怎么去掉最后8082后面的,呢

latest binary does not work under alpine

Under a alpine:latest image, the released amd64 binary fails with:

/ # /foo/docker-images/helpers/add_frep.sh  # downloads & put the binary in $PATH
frep-1.3.4-linux-amd64: OK
'frep-1.3.4-linux-amd64' -> '/usr/bin/frep'
/ # frep --version
sh: frep: not found

Same glue against an ubuntu image just works.

Rebuilding the binary with latest golang:alpine image make the bin works under alpine, but then not on ubuntu (so glibc based).

Please use uptodated 'Masterminds/sprig'

version info: frep-1.3.10-linux-amd64

According to git.mod, frep now uses 'Masterminds/sprig' of V2.22.

github.com/Masterminds/sprig v2.22.0+incompatible

I hope to have a 'get' dict function which is available from V3 of 'Masterminds/sprig'.

I always use frep with '--missing error'.
With 'get', I can say somthing like

{{ if eq (get . "optional_value") "ON" }}

instead of

{{ if hasKey . "optional_value" }} {{ if eq .optional_value "ON" }}

`--missing error` does not work with 'include'

Missing vars in included file does not rise error with --missing error.

$ more a.in b.in | cat
::::::::::::::
a.in
::::::::::::::
{{- include "b.in" -}}
::::::::::::::
b.in
::::::::::::::
var1={{.var1}}
$ frep  --no-sys-env --missing error  --overwrite b.in:b.out 
fatal: render template error, caused:

   template: b.in:1:7: executing "b.in" at <.var1>: map has no entry for key "var1"

$ frep  --no-sys-env --missing error --overwrite a.in:a.out 
$ cat a.out 
var1=<no value>
$ 

Unable to install on macOS : Invalid formula

Error on installing frep using homebrew.

System setup: macOS Monterey Version 12.4

% brew install subchen/tap/frep
==> Tapping subchen/tap
Cloning into '/usr/local/Homebrew/Library/Taps/subchen/homebrew-tap'...
remote: Enumerating objects: 180, done.
remote: Counting objects: 100% (32/32), done.
remote: Compressing objects: 100% (25/25), done.
remote: Total 180 (delta 14), reused 19 (delta 7), pack-reused 148
Receiving objects: 100% (180/180), 32.90 KiB | 1.37 MiB/s, done.
Resolving deltas: 100% (65/65), done.
Error: Invalid formula: /usr/local/Homebrew/Library/Taps/subchen/homebrew-tap/Formula/gometalinter.rb
gometalinter: wrong number of arguments (given 1, expected 0)
Error: Cannot tap subchen/tap: invalid syntax in tap!

Build on ppc64le fails to generate correct artifacts

The build is generating Intel binaries

[user3@p006vm77 frep]$ make build
rm -rf frep ./_releases ./_build
go fmt ./...
CGO_ENABLED=0 GOOS=linux GOARCH=amd64
go build -a -installsuffix cgo -ldflags "-s -w -X 'main.BuildVersion=1.3.11' -X 'main.BuildGitBranch=heads/master' -X 'main.BuildGitRev=166' -X 'main.BuildGitCommit=e325a0c21a6db33f763aa8e1f7ea7bd65402661a' -X 'main.BuildDate=Tue, 08 Sep 2020 13:06:33 +0000'" -o _releases/frep-1.3.11-linux-amd64
[user3@p006vm77 frep]$ find . -name frep-1.3.11-linux-amd64
./_releases/frep-1.3.11-linux-amd64

[user3@p006vm77 frep]$ file ./_releases/frep-1.3.11-linux-amd64
./_releases/frep-1.3.11-linux-amd64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped

Is it a bug or what ?

Hi,

I am using a template as below:

作者:
    姓名: {{ .author.name }}
    邮件: {{ .author.email }}
frep /path/to/template -e "author.name=应卓" -e "[email protected]" --dryrun

output:

作者:
    姓名: <no value>
    邮件: <no value>

I was wondering maybe if the key contains . will cause golang's template rendering bug.

frep version:

Name:       frep
Version:    1.3.7
Patches:    68
Git branch: tags/v1.3.7
Git commit: 9eca7e8f14390b28f08255bb34d987dffd3f9b10
Built:      Tue, 19 Mar 2019 07:11:21 +0000
Go version: go1.11
OS/Arch:    darwin/amd64

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.