Giter VIP home page Giter VIP logo

envman's Introduction

envman

Manage your Environment Variable collections. Switch between Environment Variable sets quickly and easily, or run a single command with a pre-defined set of Environment Variables.

envman can also be used as a bridge between separate tools, one tool can register its outputs through envman and the next tool can access these as simple environment variables.

Public Beta: this repository is still under active development, frequent changes are expected, but we we don't plan to introduce breaking changes, unless really necessary. Feedback is greatly appreciated!

Part of the Bitrise Continuous Integration, Delivery and Automations Stack, with bitrise and stepman.

Who and for what?

  • connect tools with each other : one tool saves an ENV, the other uses it (input & output)
  • manage your environment-sets : use different ENVs for different projects
  • complex environment values : if you want to store a complex input as ENV (for example a change log text/summary), envman makes this easy, you don't have to encode the value so that when you call bash's export KEY=value it will store your value as you intended. You can just give envman the value as it is through a --valuefile option or as an input stream (or just edit the related .envstore.yml file manually), no encoding required.
  • switch between environment sets : if you work on multiple projects where each one requires different environments you can manage this with envman

Install

Check the latest release for instructions at: https://github.com/bitrise-io/envman/releases

How? - Use cases

  • multi PATH handling: you have packer in your $HOME dir, in a bin subdir and terraform in another
  • create an envman .envset to include these in your $PATH

Develop & Test in Docker

Build:

docker build -t envman .

Run:

docker run --rm -it -v `pwd`:/go/src/github.com/bitrise-io/envman --name envman-dev envman /bin/bash

How does it work?

envman will run the command you specify with the environments found in its environment store.

First, run envman init to create an empty .envstore.yml file in the current directory. Now when you add a new key-value pair with envman add it stores the key and value in an .envstore.yml file in the current directory (this can be changed), and when you call envman run the next time the environments in the .envstore.yml file will be loaded for the command, and the command will be executed with an environment which includes the specified environment variables.

This is the same as you would manually set all the environments, one by one with export KEY=value (in bash) before calling the command.

envman makes it easy to switch between environment sets and to isolate these environment sets from each other - you don't have to unset environments, the specified environment set will only be available for that single run of envman / the command and won't affect subsequent calls of the command or envman.

Usage example: Simple Bash example

Add/store an environment variable:

envman add --key MY_TEST_ENV_KEY --value 'test value for test key'

Echo it:

envman run bash -c 'echo "Environment test: $MY_TEST_ENV_KEY"'

Mark an Env Var as sensitive and redact it from the build logs

Add/store an environment variable:

envman add --key MY_SECRET_ENV_KEY --value 'this is a secret value' --sensitive

Why bash -c is required? Because echo in itself does not do any environment expansion, it simply prints its input. So if you want to see the value of an environment you have to run it through a tool/shell which actually performs the environment expansion (replaces the environment key with the value associated with it).

Usage example: Ruby

Add environment variable with --value flag

system( "envman add --key SOME_KEY --value 'some value' --expand false" )

Add environment variable from an input stream

IO.popen('envman add --key SOME_KEY', 'r+') {|f|
	f.write('some value')
	f.close_write
	f.read
}

Add environment variable with a value file

require 'tempfile'

file = Tempfile.new('SOME_FILE_NAME')
file.write('some value')
file.close

system( "envman add --key SOME_KEY --valuefile #{file.path}" )

file.unlink

Usage example: Go

Add environment variable with --value flag

import "os/exec"

c := exec.Command("envman", "add", "--key", "SOME_KEY", "--value", "some value")
err := c.Run()
if err != nil {
   // Handle error
}

Add environment variable from an input stream

import "os/exec"

func RunPipedEnvmanAdd(keyStr, valueStr string) error {
	envman := exec.Command("envman", "add", "--key", keyStr)
	envman.Stdin = strings.NewReader(valueStr)
	envman.Stdout = os.Stdout
	envman.Stderr = os.Stderr
	return envman.Run()
}

err := RunPipedEnvmanAdd("SOME_KEY", "some value")
if err != nil {
	// Handle error
}

Add environment variable with a value file

import (
	"os/exec"
	"fmt"
)

c := exec.Command("envman", "add", "--key", "SOME_KEY", "--valuefile", "/path/to/file/which/contains/the/value")
err := c.Run()
if err != nil {
	// Handle error
}

Usage example: Bash

Add environment variable with --value flag

envman add --key SOME_KEY --value 'some value'

Add environment variable from an input stream

echo "some value" | envman add --key SOME_KEY

Add environment variable with a value file

envman add --key SOME_KEY --valuefile /path/to/file/which/contains/the/value --expand false

envman's People

Contributors

bazscsa avatar birmacherakos avatar damienbitrise avatar godrei avatar hisaac avatar kdobmayer avatar lpusok avatar lszucs avatar mrs1669 avatar ofalvai avatar renovate[bot] avatar tothszabi avatar trapacska avatar viktorbenei avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

envman's Issues

environment list size (100.048828125 KB) - max allowed size: 100 KB

`+------------------------------------------------------------------------------+

| (9) [email protected] |
+------------------------------------------------------------------------------+
| id: deploy-to-itunesconnect-application-loader |
| version: 0.10.1 |
| collection: https://github.com/bitrise-io/bitrise-steplib.git |
| toolkit: go |
| time: 2020-03-20T17:05:41Z |
+------------------------------------------------------------------------------+
| |
WARN[17:05:43] environment list too large
WARN[17:05:43] environment list size (100.048828125 KB) - max allowed size: 100 KB
FATA[17:05:43] [ENVMAN] environment list too large
ERRO[17:05:43] Step ([email protected]) failed: Failed to export environment list for the Step, error: exit status 1
| |
+---+---------------------------------------------------------------+----------+
| x | deploy-to-itunesconnect-application-loader@0... (exit code: 1)| 5.37 sec |
+---+---------------------------------------------------------------+----------+
| Issue tracker: ...ib/steps-deploy-to-itunesconnect-application-loader/issues |
| Source: .../bitrise-steplib/steps-deploy-to-itunesconnect-application-loader |
+---+---------------------------------------------------------------+----------+`

I set up the size limit with

mkdir -p ~/.envman && echo -e '{"env_bytes_limit_in_kb": 0}' > ~/.envman/configs.json

also changed it to be more specific with:

set -ex mkdir -p ~/.envman && echo -e '{"env_bytes_limit_in_kb": 300}' > ~/.envman/configs.json

still get the same 100kb limit error, any idea why?

need info loglevel and/or --verbose switch

To address the common repetitious pattern: envman add --key k --value v ; echo "added k:v"
It would be nice if there was a --verbose switch that echoed to STDOUT.
Or more messages that supported --loglevel info would be ok too.

Secret env variables

Hi,

I'd like support for secret variables. By this I mean that the value isn't shown when you run envman print.
Use case for this is to print the env in a CI pipeline so it's easier to see why your build is failing but I'd like to hide credentials from developers.

This commit seems to add support for what I'm asking but I'm not sure what is does. I tried to set this field but the value was still printed:

envs:
  - SECRET: password
    opts:
      is_sensitive: true

Workaround could be to use something like envman print | grep -v SECRET_VAR but a native solution would be nice.

Import path should be "github.com/sirupsen/logrus", not "github.com/Sirupsen/logrus"

Background

As README of logrus said, github.com/Sirupsen/logrus is the old path, anything using it should be updated to import and require github.com/sirupsen/logrus.

Some environments experienced problems with the upper-case variant, so the lower-case was decided. Everything using logrus will need to use the lower-case: github.com/sirupsen/logrus. Any package that isn't, should be changed.

Solution

Replace all the import paths, change "github.com/Sirupsen/logrus" to "github.com/sirupsen/logrus".
Where did you import it: https://github.com/bitrise-io/envman/search?q=github.com%2FSirupsen%2Flogrus&unscoped_q=github.com%2FSirupsen%2Flogrus

How to make env vars available to JVMs?

I am working on a new step that would let people execute arbitrary Groovy scripts (https://github.com/hiyainc/bitrise-step-groovy-script-runner). During testing in my own pipeline, though, I am unable to access environment variables I expect to be present. I use the standard JVM approach of calling System.getenv('BITRISE_BUILD_NUMBER') (for example). When I test that in a bash script (with $BITRISE_BUILD_NUMBER), I get the expected value. When I test it in my Groovy script, it's always a blank string (not even null). I have tested this same Groovy script locally, and System.getenv(...) works as expected.

How does envman -- and indeed, Bitrise -- expose environment variables to steps?

To provide further examples, I have a step that has this output:

Screen Shot 2020-07-17 at 10 48 24 AM

and I cannot access that env var in my groovy step which runs immediately afterwards. Just for the hell of it, I inserted a bash step between the two that does this:

#!/usr/bin/env bash
set -ex
echo -n $BITRISE_APK_PATH | envman add --key PROD_RELEASE_APK_PATH

and I also could not access PROD_RELEASE_APK_PATH. This works when consumed from a bash script.

Thanks for your time.

Command not finishing

In one of my workflows I have a "Script"-Task that runs a NodeJS script (via the node interpreter that comes with my Stack).

This is the code I use to set environment variables from the script:

const cp = require("child_process");

const setEnvironment = (key, value) => {
    const command = `envman add --key ${key} --value "${value}"`;
    console.log("Executing:", command);
    cp.exec(command, {timeout: 1000}, (err, stdout, stderr) => {
        if (err) {
            if (err.code === null) {
                console.log("Terminated after timeout:", command);
            } else {
                console.log(`Error setting env (code ${err.code}):`, stdout, stderr);
            }
        } else {
            console.log("Success setting", stdout, stderr);
        }
    })
}

// Call
setEnvironment("SONAR_QUBE_QUALITY_GOAL_STATUS", status);

As far as I can tell, the cp.exec-function does basically the same as the Ruby or Go examples from the documentation.

However, I have noticed that the envman command does not terminate if I call it in this way. The environment variables are set correctly, but this code always prints Terminated after timeout.

If I don't set a timeout for the child-processes, the script never terminates (waiting for it's children to terminate) and the build never concludes.

I'm running this on the "Android & Docker, on Ubuntu 16.04" Stack wich currently comes with Node 8.11.1. What am I overlooking?

Issue with special characters

Even seems to save it correctly, I'm having issues accessing later to the variable stored, some characters are missing when I retrieve it.

Let's say I want to save Rick'"$@Morty

I execute:
envman add --key MY_TEST_ENV_KEY --value 'Rick'\''"$@Morty'

Doing envman print I get correctly:
MY_TEST_ENV_KEY: Rick'"$@Morty

While doing:
envman run bash -c 'echo "$MY_TEST_ENV_KEY"'

I do get:
Rick'"Morty

So $ and @ got missing.

What I'm doing wrong?

Allow `envman add` to accept empty value

This would be super useful in case I want to create an environment list template file, which defines all the required environments but the values will be specified by someone else later, before use.

Maybe also add some option to the stored env item which defines whether the given env should be specified before it could be used with an envman run - similar to expand.

This way I could create a template envstore but I would mark all the required inputs, so that while those are empty/not filled in someone else can't run it with envman run.

Proposal: Adding aliases type for env variables

If PR #221 is accepted

It could be great to make envman source of truth for both environment variables and aliases.

With envlist format it will be possible to have something like

export ENV_A=VAL_A
....
alias ls_alias='ls -lsoptions'

What do you think about adding aliases to envman ?

Loading environment variable set?

Looking through the documentation, it's not very clear if/how you should load a variable set. I see .envset mentioned in passing in the README, but it doesn't appear in the source code, as far as I can tell.

If this is indeed a feature, would you be able to add how it should be used, including file structure and command line usage?

Variables passed from envman not working correctly

I have a build that is failing iOS provisioning due to envman:

project (ios/WellOne.xcworkspace) does not contain scheme: WellOneProductionRelease

It is failing because of the following shell script command that's run on step 2

echo "WellOneProductionRelease" | envman add --key IOS_P_SCHEME

This script is basically setting IOS_P_SCHEME to be WellOneProductionRelease which is the same exact value as what it was set to in the "Env Vars" section and whenever this line is commented the build works flawslessly although the variable should have the same value
Please let me know how to resolve this, thanks!

Env Vars set using envman are not available in current step

In a single script step do the following

envman add --key SOME_KEY --value 'some value'
echo $SOME_KEY

Note that the Env Var is not set

Ideally Envman would also set the env var using export so it is available in the current step

export SOME_KEY="some value"

How to increase the per-env-var limit of envman

envman by default has a 20KB per-environment (key) value size limit, and a 120KB total envs size limit. If you try to add a value that's larger than 20KB you'll get an error like:

$ ruby -e 'puts "a"*40_000' | bitrise envman add --key TEST2
WARN[18:22:12] environment value too large
WARN[18:22:12] environment value size (39.0634765625 KB) - max allowed size: 20 KB
FATA[18:22:12] [ENVMAN] environment value too large - rejected
FATA[18:22:12] Command failed, error: exit status 1

This limit is in place because lots of tools, including e.g. bash itself, can't work with really large environment variables. If the environment size is too large bash for example will fail to run any command with an error argument list too long.

So, in general, you should find an alternative solution to store larger data, e.g. in files instead of in env vars, but if you can't solve it any other way you can increase these limits via envman's config file (should be stored at ~/.envman/configs.json).

To increase the per-key value size limit to e.g. 30KB you can use (e.g. in a Script step):

#!/usr/bin/env bash
set -ex
mkdir -p ~/.envman && echo -e '{"env_bytes_limit_in_kb": 30}' > ~/.envman/configs.json

[Feature] Install script

I made a simple shell script thats download a release of envman
If a version is passed with -v arg, this is downloaded, if not then latest version is downloaded
And after run chmod in executable

Then, can be used like this:
curl -fL https://github.com/bitrise-io/envman/blob/master/install.sh | sh

Code:

#!/bin/sh
while getopts v: flag
do
    case "${flag}" in
        v) version=${OPTARG};;
    esac
done

if [ -z "$version" ];
then
  echo -e "\033[0;32mNo version provided, downloading latest version.\033[0m"
  curl -fL https://github.com/bitrise-io/envman/releases/latest/download/envman-"$(uname -s)"-"$(uname -m)" > /usr/local/bin/envman
else
  echo -e "\033[0;32mDownloading envman v$version\033[0m"
  curl -fL https://github.com/bitrise-io/envman/releases/download/$version/envman-"$(uname -s)"-"$(uname -m)" > /usr/local/bin/envman
fi

echo -e "\033[0;33mMaking envman binary executable...\033[0m"
chmod +x /usr/local/bin/envman
echo -e "\033[0;33mDone!\033[0m"
echo -e "\033[0;32menvman succesfully installed!\033[0m"

Run's help fails on first run

Just after installing envman, I can't get help for the run command.

$ envman run --help
FATA[10:48:57] [ENVMAN] - Failed to load EnvStore:No file found at path: [...]/.envstore.yml 

All the other commands' help works without an envstore file.

How to debug envman?

Somewhat similar to this issue #175

My build is failing at the github-release stage, and it seems like it's happening because an env var I add on a prior step is not being picked up at this step. I'd like to be able to debug somehow to see what the value is at the step it's created and the step it's consumed by.

I tried using echo "${MY_VAR}", but it didn't do anything. Which I guess either means MY_VAR is empty or something else is wrong.

Set `env_bytes_limit_in_kb` and `env_list_bytes_limit_in_kb` to `0` make other scripts fail

In our workflow, before envman got updated to 2.0.0, we were doing in a Script step:

mkdir -p ~/.envman
echo -e "{\n\t\"env_bytes_limit_in_kb\": 0,\n\t\"env_list_bytes_limit_in_kb\": 0\n}" > ~/.envman/configs.json

And the rest of the workflow executed as expected, and we could set environment variable of arbitrary size via envman (which is a requirement of our workflow, as some of our generated environment variables can get quite big).
After the envman update to 2.0.0, the same Script still succeed, but then any other Script step that executes afterwards fail with the following error message:

+------------------------------------------------------------------------------+
| (X) <Script Step>                                                            |
+------------------------------------------------------------------------------+
| id: script                                                                   |
| version: 1.1.5                                                               |
| collection: https://github.com/bitrise-io/bitrise-steplib.git                |
| toolkit: bash                                                                |
| time: 2018-09-17T06:11:24-07:00                                              |
+------------------------------------------------------------------------------+
|                                                                              |
 [!] => Failed: No script (content) defined for execution!
|                                                                              |

@viktorbenei has a reproducing example from one of our build, that I won't link here.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.