Giter VIP home page Giter VIP logo

livecycle / preevy Goto Github PK

View Code? Open in Web Editor NEW
2.0K 15.0 74.0 23.98 MB

Quickly deploy preview environments to the cloud!

Home Page: https://preevy.dev

License: Apache License 2.0

JavaScript 3.27% TypeScript 93.42% Batchfile 0.03% Shell 1.08% Dockerfile 0.10% Starlark 0.05% CSS 0.43% MDX 1.32% Nunjucks 0.31%
devops docker docker-compose ephemeral-environments pr-environments preview-environment aws cli awesome awesome-list

preevy's Introduction


Preevy

Provision preview environments with minimal configuration โ€ข made by Livecycle

GitHub GitHub last commit Version


We recently launched the Livecycle Docker Extension. Now you can share local environments instantly. Get feedback while your code is still in flight. Check it out here

Preevy-demo.mp4

Preevy

Preevy is a Command Line Interface (CLI) tool designed to simplify the process of creating ephemeral preview environments from Dockerized applications. Integrate Preevy into your CI flow to deploy Pull Requests as preview environments, using your existing cloud provider or Kubernetes cluster.

Preevy makes use of affordable VMs from AWS Lightsail, Google Cloud, Microsoft Azure, or any Kubernetes cluster.

Preevy can deploy your app with public or protected access, on the public internet or inside your existing private network.

Deploying a preview environment per Pull Request offers a range of benefits:

  • ๐ŸŒ Universal Access: Just by sharing a URL, you can allow anyone to try your product revision on any device.

  • ๐Ÿ“ฉ Effortless Asynchronous Updates: Keep non-technical stakeholders in the loop without coordinating synchronous meetings.

  • ๐ŸŽจ Hassle-free Design Reviews: Designers can verify implementation independently, minimizing interruptions.

  • ๐Ÿš€ Parallel E2E Tests: Use external test agents against preview environments to expedite the testing process.

  • ๐Ÿ’ก Streamlined Feedback Cycle: Preview environments let your team engage with and feedback on new features early in the pipeline.

  • ๐Ÿงช Non-production Experimentation: Develop and share unique versions of your product for presentations, examples, or tests.

  • ๐Ÿ” Secure Collaboration: Generate private sandboxes to share with external stakeholders, ensuring secure collaborative efforts.


Visit The full documentation here: https://preevy.dev/

Contents

What

Preevy can take any Docker-Compose application definition and with a single up command perform the following:

  • Provision and configure a virtual machine (VM) on your cloud, or a Pod on your Kubernetes cluster.
  • Build your application on the VM/Pod or on any BuildKit builder (optional)
  • Deploy your application on the VM/Pod.
  • Expose each service of your application to the web with a user-friendly public HTTPS URL without any DNS/Certificate configuration.

These environments can be managed using the Preevy command-line interface (CLI) and can be easily updated or destroyed when necessary. While Preevy can be used for sharing local environments with your team, its primary goal is to implement preview environments for pull requests. Therefore, it's designed to be easily integrated into CI/CD flows.

Why

At Livecycle, we believe that preview environments are an integral part of any development flow, in any engineering team. These non-production, ephemeral environments, created for every Pull Request, can significantly improve PR workflows. In recent years, preview environments have become increasingly popular, with some PaaS providers even offering deeply integrated preview environments. However, setting up preview environments can be a complex and costly task, which is why many teams have been hesitant to implement them. Preevy is designed to simplify this task and provide a framework for provisioning and utilizing preview environments to optimize the PR flow. You can read more about the story and philosophy behind Preevy here.

Getting started

Prerequisites

Required software

Your Docker Compose app

If you don't have an existing Docker Compose app, check out Awesome Compose - a curated list of Compose samples, from React to Minecraft.

Your cloud provider or Kubernetes cluster

Preevy deploys your app to one of the supported deploy runtimes.

Choose the cloud provider or Kubernetes cluster you're going to use and configure access credentials for it:

Note Preevy only uses your credentials when you run the Preevy CLI to set up and connect to your environments. Your credentials are not sent or stored anywhere outside of your computer.

Local Kubernetes server using Docker Desktop

If you don't have an existing cloud account or prefer to try Preevy first locally, you can use the Docker Desktop Kubernetes server. Go to: Docker Settings -> Kubernetes -> Enable Kubernetes.

Install the Preevy CLI

npm install -g preevy

Or use npx to run the CLI without installing it:

npx preevy <command>

Set up a profile

preevy init

Preevy will ask you to select a deploy target and a storage location for your profile. You can start by storing the profile locally.

Deploy your first environment

From the same directory where your docker-compose.yml or compose.yml file is located, run the command below:

preevy up

Note: Preevy uses the git repo at the current directory to calculate a stable environment ID for your project. Make sure a git repo is initialized (with at least one commit), or add the --id flag to explicitly specify the environment ID.

Access and share your new preview environment at the *.livecycle.run links provided in the command output.

Update your environment

Code changed? Re-run preevy up to quickly sync the preview environment with your changes on the existing VM or Kubernetes Pod.

List and remove your environment

Run preevy ls to show all environments in your deploy target which are linked to your profile.

Run preevy down to remove your environment. Preevy will delete the VM or Kubernetes Pod.

Service URLs

Every Compose service is exposed individually with a generated URL in the following format: https://{service}-{[port]}-{env-id}-{client-id}.{tunnel-server-domain}. If the service exposes a single port, the port part is omitted. See here for a more detailed explanation.

  • env-id is automatically generated from the Compose project and Git branch, or can be explicitly specified using the --id flag of the preevy up command.
  • client-id is a random identifier based on the profile's public tunneling SSH key (generated in preevy init).
  • tunnel-service-domain is where the tunnel service is hosted. It can be specified using the --tunnel-url flag of the preevy up command, and defaults to Livecycle's hosted service at *.livecycle.run.

Under the hood

Preevy has two main components:

The CLI is a Node.js program responsible for:

  • Provisioning and tearing down VMs.
  • Exposing environments' state and URLs to the end user.
  • Storing and accessing profile data (settings, keys, etc).
  • Setting up a VM with Docker tooling.
  • Syncing Compose source code and local volumes.
  • Running the Compose app, augmented with a daemon for connecting to the tunneling service.

The tunnel server is a Node.js base server responsible for exposing friendly HTTPS URLs for the Compose services.

A free public instance is hosted by Livecycle on livecycle.run, and it can be self-hosted as well.

A public Docker/OCI image is available: ghcr.io/livecycle/preevy/tunnel-server

To host your own Tunnel Server instance, see the deployment guide.

CI Integration

Preevy is designed to work seamlessly with your CI, by importing a shared preview profile from AWS S3 Google Cloud Storage (GCS) and Azure Blob Storage (AZBlob).

Profiles are created using preevy init. Choose a S3/GCS/AZBlob URL for storing the profile - Preevy will create a bucket if one doesn't exist.

If you already have a locally stored Preevy Profile, it can be migrated to remote storage using preevy profile cp

Once the profile is created, it can be imported to the CI runtime using preevy init --from <profile-url>

Examples

Faster builds in CI

Check out our documentation to find out how to speed up your builds and reduce the costs of your preview environments by running Preevy with BuildKit Builders in CI.

Example repos

GitHub Actions

Shortcut for setting up a cost-efficient Kubernetes cluster on AWS EKS

Don't have a Kubernetes cluster? See an example repo for setting up AWS EKS using Terraform. The example includes Karpenter which can reduce the cost of running Preview Environments by automatically expanding and shrinking your cluster using EC2 Spot Instances

Security

In case you find a security issue or have something you would like to discuss, refer to our security policy.

Private environments

Preevy can add an authentication layer to your provisioned environments. When you configure your service as private the Tunnel Server restricts access based on a pre-shared secret or a Livecycle login (SSO via Google/Microsoft/GitHub).

Exposure of preview environments

Services on provisioned environments are not exposed directly, but rather via a tunnel created by the tunneling server.

Livecycle access to data

When you use Preevy, Livecycle does not get access to your credentials or code. Preevy only uses your cloud provider or Kubernetes credentials to provision and connect to environments - it does not send or store the credentials.

Encrypted traffic to and from your environments goes through Preevy's Tunnel Server. Livecycle hosts the default Tunnel Server at livecycle.run which is available as part of Livecycle's SaaS offering. Like most SaaS providers, we keep logs for monitoring and troubleshooting purposes which include metadata of the requests. The Tunnel Server code is part of the Preevy OSS project; you can run it on your own infrastructure and specify the its address via the --tunnel-url flag.

Network isolation

The Tunnel Server can be deployed on your private network (e.g. VPC), which access to your environments at the network level.

Configuration

Preevy loads its configuration from the following sources, in order:

Preevy Profile

The Preevy profile is created by the init command and can be stored locally or remotely on your cloud provider. A profile is required to create environments. The profile includes the following:

  • A tunneling key, that is used to identify your services when connecting to the Tunnel Server.
  • The default driver to use for provisioning environments.
  • Default driver options to use per driver (e.g. AWS region, K8s namespace).
  • Driver state whose contents depend on the specific driver.

Profiles can be migrated to a different storage location using preevy profile cp.

The default profile can be overridden using the global command line argument --profile.

Note: The profile currently combines context and state, and some changes are planned.

Compose files

Preevy extracts its runtime settings from the Compose file.

Project Compose files

Just like with the docker compose CLI, you can use the global --file | -f command line argument to specify the path(s) for the Compose file. If not specified, the default loading order is used. Multiple files are supported.

Preevy-specific Compose file

In addition to the project compose files, an optional Preevy-specific Compose file can be used. Preevy attempts to load files named compose.preevy.yaml, compose.preevy.yml, docker-compose.preevy.yaml or docker-compose.preevy.yml. If one of these exists, it is loaded BEFORE the project composes file(s). The name of the Preevy-specific compose file can be overridden by specifying the argument --system-compose-file.

x-preevy: Preevy-specific configuration in the Compose file(s)

A x-preevy top-level element can be added to the Compose file(s).

services:
  ...
x-preevy:
  driver: lightsail
  drivers:
    lightsail:
      region: eu-central-1
    kube-pod:
      context: dev-cluster
  plugins:
    ...

The following properties are supported, all of them optional:

driver

Override the default driver to use for this Compose project. Available values: lightsail, gce, azure, kube-pod.

This value can be overridden per command execution using the --driver CLI flag.

drivers

Override the default options per driver for this Compose project. See the specific driver documentation.

These values can be overridden per command execution using the specific driver CLI flags, e.g. --lightsail-bundle-id=2xlarge_2_0

Example:

x-preevy:
  drivers:
    lightsail:
      bundle-id: large_2_0
    kube-pod:
      context: dev-cluster

plugins

See Plugins below.

Plugins

Plugins are a way to extend Preevy's functionality via externally-published NPM packages.

A plugin can add hooks that execute code in response to events. It can also define new commands, and add flags to existing commands to customize their behavior.

Default plugins

The GitHub integration plugin packaged as @preevy/plugin-github is bundled with Preevy and enabled by default.

Enabling or disabling plugins

From the Docker Compose file

Plugins can be configured in the Preevy configuration section of your Compose file. Add a plugins section to the x-preevy top-level element:

services:
  ...
x-preevy:
  plugins:
    - module: '@preevy/plugin-github'
      disabled: false # optional, set to true to disable plugin
      # ...additional plugin-specific configuration goes here

See the included GitHub integration plugin for a detailed example.

From the environment

Plugins can be enabled or disabled by setting the PREEVY_ENABLE_PLUGINS and PREEVY_DISABLE_PLUGINS environment variables to a comma-separated list of packages.

Example: To disable the default GitHub integration plugin, set PREEVY_DISABLE_PLUGINS=@preevy/plugin-github.

From the CLI flags

Specify the global --enable-plugin=<module> and --disable-plugin=<module> flags to enable or disable plugins per command execution. CLI flags take priority over the Docker Compose and environment configuration.

Docs and support

Read about Preevy's components and learn how to use them in our documentation.

Ask a question or join our Livecycle Community to get support.

Telemetry

The Preevy CLI collects telemetry data to help us understand product usage and direct future development.

The data collected is anonymous and cannot be used to uniquely identify a user. Access to the data is limited to Livecycle's employees and not shared with 3rd parties.

To see the collected data, set the environment variable PREEVY_TELEMETRY_FILE to a filename.

We appreciate the usage data sent to us as - it's the most basic and raw type of feedback we get from our users. However, if you are concerned about sending out data, you may choose to disable telemetry.

Telemetry collection can be disabled by setting the environment variable PREEVY_DISABLE_TELEMETRY to 1 or true.

preevy's People

Contributors

assafkr avatar ayaan49 avatar chinmay7016 avatar codebutler avatar dependabot[bot] avatar deva2580 avatar dromzeh avatar eliyabar avatar github-actions[bot] avatar hassansuhaib avatar kburakozdemir avatar naorzr avatar panquesito7 avatar pradumnasaraf avatar priyankarpal avatar roy9495 avatar royra avatar yaara-hendel avatar yshayy avatar zevireinitz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

preevy's Issues

[Docs]: Disable blank issues and provide a misc. template

what's wrong with the documentation?

By disabling blank issues, we will prevent issues without enough information or spammy issues.
We can add a miscellaneous template that will be used for anything else that does not fit the current templates.

If you like this idea, I'd like to work on this. ๐Ÿ™‚

Add screenshots

I don't think this applies here.

Record

  • I agree to follow this project's Code of Conduct

Implement initial support for Kubernetes as machine provider

This feature will allow users to use their existing k8s infrastructure as backbone for Preevy environments.
Initial implementation can create a pod with support for nested Docker (probably need to be privileged or require additional CRI runtime), port-forwarding can be done with kubectl.
Persistent volume might also be interesting to add in the future to support "suspending" an environment.

Tasks

Basic Authn/Authz support

Initial implementation will be based on http basic authentication.

  • Support for declaring "private" services
  • Support for authentication using tunneling-key based authentication

[BUG]: tunnel server can produce URLs that include invalid subdomain with >63 length

Bug description?

Subdomain is limited to 63 characters, we should trim to the characters to support valid URLs.
Implementation can be done in tunnel server or Preevy/compose tunnel agent.
In both cases, we should pay attention to the CLI functionality of predicting the URL for injecting env variables:
https://preevy.dev/recipes/service-discovery

We should also think if we want to truncate the entire subdomain url, or set a size limit for each fragment:
https://{service}-{[port]}-{env-id}-{client-id}

Additional context

Current workaround: use the --id flag in up to create a shorter env identifier or use shorter branch names.

Record

  • I agree to follow this project's Code of Conduct

Add a "troubleshooting" section

For example, use ssh to go into the VM and use the internal docker commands to find out what's wrong

  • Cover debug with preevy ssh
  • Cover authorization scenarios for frontend apps (domain origin in such)

Implement store based on Azure blob storage for profile store

Implement a profile storage for azure blob storage.

Should be similar to other implementations:

Registration is done here:
https://github.com/livecycle/preevy/blob/main/packages/cli/src/fs.ts

Url format should follow conventions of golang cloud sdk:
https://gocloud.dev/howto/blob/#azure

[Improvement]:Update Package Scripts for Cross-Platform Compatibility

What feature?

Currently, some of the package scripts in our package.json files are not compatible with all operating systems. To ensure seamless development across different platforms, we need to update the scripts to be cross-platform compatible.

Affected script:
clean The current implementation uses rm -rf to remove directories, which is not compatible with Windows systems.

Describe the solution you'd like

We should switch to a cross-platform solution like rimraf.

Describe alternatives you've considered

No response

Additional context

No response

Add screenshots

No response

Record

  • I agree to follow this project's Code of Conduct

Support authentication session in tunnel server

We should be able to support the session for:

  • Support flows different from basic authentication
  • Improving the experience of the login between different environments/services (no need to re-login)
  • Allow to create a session without passing the authorization header the can conflict with internal app authorization header and doesn't work properly when using websockets

Reduce stack trace noise in errors

We should take into consideration the level of error verbosity and the type of error to remove noise from the CLI output.
For example, error in mismatching or invalid args/flags shouldn't print out a stack trace.

Package/Component renaming

Current package names are a bit confusing, suggestion:
standalone-proxy -> tunneling, or tunneling-service is more fitting to the context.
docker-proxy -> compose-tunneling-agent or docker-tunneling-agent as it basically a client daemon that connect to a tunneling service based on compse/docker state

bug showing logs

using the CLI to show logs show no prompt.
after executing up command, trying to show logs with a wrong service name does nothing.

Steps to reproduce:

  • preevy up
  • preevy logs any-text-you-want
  • nothing happens, no error, no logs.

using debug level shows that socket is closed.

image

  • if there is no env up or after 'preevy down' command, the output is ok ( Error: No machine found for envId wekan-master)

rename 'envId' to 'environment id'

when using preevy up, the following text appears:
'Using envId: xxx'
replace it with something more readable like
'Using environment id: xxx'

Add environment variables for tunnel-server urls for usage in docker-compose

When doing service-service communication, Preevy can work well due to compose service discovery.
However, when using a frontend that utilize the exposed port for connecting to backend (can be common in dev environments), it's a bit tricky since we need to replace the url with tunneling url which we don't know about yet.

One way to solve it, is by injecting/resolving env variables of tunneling url in compose normalization. (so it can affect both build args, and environment)

For example:

services:
     api:
           ....
           ports:
           - 9005:3000
     fronend:
          environment:
          - API_URL=http://localhost:9005

This could be a common setup.

With Preevy, it could be something like:

services:
     api:
           ....
           ports:
           - 9005:3000
     fronend:
          environment:
          - API_URL={PREEVY_BASE_URI_API_3000:-http://localhost:9005}

In the future, we can enhance this by supporting inference of this configuration automatically and replacing all static http://localhost:xx variables' values based on public ports definitions.

[Improvement]: Add messaging for initial CLI cloud interactions

What feature?

When using preevy up or other commands, it can take several seconds until we see the output due to connectivity with cloud providers. (for example, downloading profile)
We can improve this experience by adding console messages that log these interactions.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

Add screenshots

No response

Record

  • I agree to follow this project's Code of Conduct

feat: add new issue templates

currently, we have only two issue templates we can add a few templates such as style, docs etc.

  • we need to use the .yml format issue template like these.
    image

features if we use the .yml format

  • we can add multiple textboxes, checkboxes, tags etc.
    image

I want to work on this issue

Support adding & querying metadata on environments

Can be exposed by 'compose-tunnel-agent' which might worth changing to simply Preevy agent.
Such metadata can be:

  • Time of creation
  • Provider flags (size, region)
  • Git Context
  • Current services/tunnels (already implemented partly)
  • Plugins metadata

This data can be used by Preevy widget/dashboard

[BUG] - Preevy should print meaningful error if docker executable not found

Currently, preevy print this error:

Error: spawn docker ENOENT
    at Process.ChildProcess._handle.onexit (node:internal/child_process:283:19)
    at onErrorNT (node:internal/child_process:476:16)
    at processTicksAndRejections (node:internal/process/task_queues:82:2

It's very aggressive and we should print a friendlier message

Suggestion: Create Action to ease the use of Preevy in GH actions

Is your feature request related to a problem? Please describe.
An easy way to set up Preevy up/down flows in GH actions.
Currently, it requires installing Preevy with npm -g or using npx.

Describe the solution you'd like
A GH action that wrap the up/down parameters.
Profile is specified as parameter and importing is done automatically.

uses: preevy/action@v1
with:
     profile-url: "s3://aabcd"
     command: up
     args: 

Describe alternatives you've considered
It's possible to remove NPM dependency/installation by packing Preevy CLI as a Docker image, and support environment variable for profile URL, but there's a drawback which it's not trivial to get the cloud provider context when running this way.

Preevy deployment with Google cloud

Currently, if you'll try to deploy with the GCP driver while your branch name contains Uppercase characters, you'll get the following error

Run echo "/home/runner/work/_actions/livecycle/preevy-up-action/v1.2.0" >> $GITHUB_PATH
Run # Convert the docker-compose-yaml-paths input to an env var
Docker file paths docker-compose.yml
Running Preevy with files: -f docker-compose.yml -f docker-compose.dev.yml 
Using environment ID: preevy-saas-eliyaenvironmentsUI
- Creating Google Cloud machine
โœ– Creating Google Cloud machine: Creating from existing snapshot
    Error: Invalid value for field 'resource.labels': ''. Label value 
    'preevy-saas-eliyaenvironmentsUI' violates format constraints. The value 
    can only contain lowercase letters, numeric characters, underscores and 
    dashes. The value can be at most 63 characters long. International 
    characters are allowed.
    Code: 3

Possible solution: lowercase the resource label value.

README - aws login

The main README says:
In AWS, it could be by using aws login or aws configure

the AWS cli doesn't contain any login command.

what am i missing?

Add first class support for TLS in tunnel server

At present, Preevy tunnel only forwards HTTP traffic, and HTTPS is handled by an external service that performs TLS termination. The tunnel server uses the HTTP host header to route traffic to the appropriate service.

If TLS support is added to the tunnel server, Preevy will be able to route non-HTTP traffic to the correct host using Server Name Indication (SNI). Specific protocol manipulation, such as rewriting headers, can be accomplished by explicitly specifying the channel type or using Application-Layer Protocol Negotiation (ALPN).

This new feature will enable users to directly connect to their non-HTTP services, such as databases, provided that their client supports TLS and uses SNI.

Preevy widget

Initial functionality:

  • Show all services and links
  • Show current git context (branch/pr)

Add Preevy link in Pull Requests

Add a simple way to get to the latest Preevy environment of a PR, ideally using a Preevy bot adding the link as a comment in the PR

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.