Giter VIP home page Giter VIP logo

piston's Introduction

engineer-man piston Piston

A high performance general purpose code execution engine.


GitHub last commit GitHub issues GitHub pull requests




About

Piston is a high performance general purpose code execution engine. It excels at running untrusted and possibly malicious code without fear from any harmful effects.


It's used in numerous places including:


Official Extensions

The following are approved and endorsed extensions/utilities to the core Piston offering.

  • I Run Code, a Discord bot used in 4100+ servers to handle arbitrary code evaluation in Discord. To get this bot in your own server, go here: https://emkc.org/run.
  • Piston CLI, a universal shell supporting code highlighting, files, and interpretation without the need to download a language.
  • Node Piston Client, a Node.js wrapper for accessing the Piston API.
  • Piston4J, a Java wrapper for accessing the Piston API.
  • Pyston, a Python wrapper for accessing the Piston API.
  • Go-Piston, a Golang wrapper for accessing the Piston API.
  • piston_rs, a Rust wrapper for accessing the Piston API.
  • piston_rspy, Python bindings for accessing the Piston API via piston_rs.

Public API

  • Requires no installation and you can use it immediately.
  • Reference the Runtimes/Execute sections below to learn about the request and response formats.

When using the public Piston API, use the following two URLs:

GET  https://emkc.org/api/v2/piston/runtimes
POST https://emkc.org/api/v2/piston/execute

Important Note: The Piston API is rate limited to 5 requests per second. If you have a need for more requests than that and it's for a good cause, please reach out to me (EngineerMan#0001) on Discord so we can discuss potentially getting you an unlimited key. What is and isn't a good cause is up to me, but, in general if your project is a) open source, b) helping people at no cost to them, and c) not likely to use tons of resources thereby impairing another's ability to enjoy Piston, you'll likely be granted a key.


Getting Started

All In One

Host System Package Dependencies

  • Docker
  • Docker Compose
  • Node JS (>= 13, preferably >= 15)

After system dependencies are installed, clone this repository:

# clone and enter repo
git clone https://github.com/engineer-man/piston

Installation

# Start the API container
docker-compose up -d api

# Install all the dependencies for the cli
cd cli && npm i && cd -

The API will now be online with no language runtimes installed. To install runtimes, use the CLI.

Just Piston (no CLI)

Host System Package Dependencies

  • Docker

Installation

docker run \
    -v $PWD:'/piston' \
    --tmpfs /piston/jobs \
    -dit \
    -p 2000:2000 \
    --name piston_api \
    ghcr.io/engineer-man/piston

Piston for testing packages locally

Host System Package Dependencies

Installation

# Build the Docker containers
./piston start

# For more help
./piston help

Usage

CLI

The CLI is the main tool used for installing packages within piston, but also supports running code.

You can execute the cli with cli/index.js.

# List all available packages
cli/index.js ppman list

# Install latest python
cli/index.js ppman install python

# Install specific version of python
cli/index.js ppman install python=3.9.4

# Run a python script using the latest version
echo 'print("Hello world!")' > test.py
cli/index.js run python test.py

# Run a python script using a specific version
echo 'print("Hello world!")' > test.py
cli/index.js run python test.py -l 3.9.4
cli/index.js run python test.py -l 3.x
cli/index.js run python test.py -l 3

If you are operating on a remote machine, add the -u flag like so:

cli/index.js -u http://piston.server:2000 ppman list

API

The container exposes an API on port 2000 by default. This is used by the CLI to carry out running jobs and package management.

Runtimes Endpoint

GET /api/v2/runtimes This endpoint will return the supported languages along with the current version and aliases. To execute code for a particular language using the /api/v2/execute endpoint, either the name or one of the aliases must be provided, along with the version. Multiple versions of the same language may be present at the same time, and may be selected when running a job.

HTTP/1.1 200 OK
Content-Type: application/json

[
    {
        "language": "bash",
        "version": "5.1.0",
        "aliases": [
            "sh"
        ]
    },
    {
        "language": "brainfuck",
        "version": "2.7.3",
        "aliases": [
            "bf"
        ]
    },
    ...
]

Execute Endpoint

POST /api/v2/execute This endpoint requests execution of some arbitrary code.

  • language (required) The language to use for execution, must be a string and must be installed.
  • version (required) The version of the language to use for execution, must be a string containing a SemVer selector for the version or the specific version number to use.
  • files (required) An array of files containing code or other data that should be used for execution. The first file in this array is considered the main file.
  • files[].name (optional) The name of the file to upload, must be a string containing no path or left out.
  • files[].content (required) The content of the files to upload, must be a string containing text to write.
  • files[].encoding (optional) The encoding scheme used for the file content. One of base64, hex or utf8. Defaults to utf8.
  • stdin (optional) The text to pass as stdin to the program. Must be a string or left out. Defaults to blank string.
  • args (optional) The arguments to pass to the program. Must be an array or left out. Defaults to [].
  • compile_timeout (optional) The maximum time allowed for the compile stage to finish before bailing out in milliseconds. Must be a number or left out. Defaults to 10000 (10 seconds).
  • run_timeout (optional) The maximum time allowed for the run stage to finish before bailing out in milliseconds. Must be a number or left out. Defaults to 3000 (3 seconds).
  • compile_memory_limit (optional) The maximum amount of memory the compile stage is allowed to use in bytes. Must be a number or left out. Defaults to -1 (no limit)
  • run_memory_limit (optional) The maximum amount of memory the run stage is allowed to use in bytes. Must be a number or left out. Defaults to -1 (no limit)
{
    "language": "js",
    "version": "15.10.0",
    "files": [
        {
            "name": "my_cool_code.js",
            "content": "console.log(process.argv)"
        }
    ],
    "stdin": "",
    "args": ["1", "2", "3"],
    "compile_timeout": 10000,
    "run_timeout": 3000,
    "compile_memory_limit": -1,
    "run_memory_limit": -1
}

A typical response upon successful execution will contain 1 or 2 keys run and compile. compile will only be present if the language requested requires a compile stage.

Each of these keys has an identical structure, containing both a stdout and stderr key, which is a string containing the text outputted during the stage into each buffer. It also contains the code and signal which was returned from each process.

HTTP/1.1 200 OK
Content-Type: application/json

{
    "language": "js",
    "version": "15.10.0",
    "run": {
        "stdout": "[\n  '/piston/packages/node/15.10.0/bin/node',\n  '/piston/jobs/9501b09d-0105-496b-b61a-e5148cf66384/my_cool_code.js',\n  '1',\n  '2',\n  '3'\n]\n",
        "stderr": "",
        "output": "[\n  '/piston/packages/node/15.10.0/bin/node',\n  '/piston/jobs/9501b09d-0105-496b-b61a-e5148cf66384/my_cool_code.js',\n  '1',\n  '2',\n  '3'\n]\n",
        "code": 0,
        "signal": null
    }
}

If a problem exists with the request, a 400 status code is returned and the reason in the message key.

HTTP/1.1 400 Bad Request
Content-Type: application/json

{
    "message": "html-5.0.0 runtime is unknown"
}

Supported Languages

awk, bash, befunge93, brachylog, brainfuck, bqn, c, c++, cjam, clojure, cobol, coffeescript, cow, crystal, csharp, csharp.net, d, dart, dash, dragon, elixir, emacs, emojicode, erlang, file, forte, forth, fortran, freebasic, fsharp.net, fsi, go, golfscript, groovy, haskell, husk, iverilog, japt, java, javascript, jelly, julia, kotlin, lisp, llvm_ir, lolcode, lua, matl, nasm, nasm64, nim, ocaml, octave, osabie, paradoc, pascal, perl, php, ponylang, powershell, prolog, pure, pyth, python, python2, racket, raku, retina, rockstar, rscript, ruby, rust, samarium, scala, smalltalk, sqlite3, swift, typescript, basic, basic.net, vlang, vyxal, yeethon, zig,


Principle of Operation

Piston uses Docker as the primary mechanism for sandboxing. There is an API within the container written in Node which takes in execution requests and executees them within the container safely. High level, the API writes any source code to a temporary directory in /piston/jobs. The source file is either ran or compiled and ran (in the case of languages like c, c++, c#, go, etc.).


Security

Docker provides a great deal of security out of the box in that it's separate from the system. Piston takes additional steps to make it resistant to various privilege escalation, denial-of-service, and resource saturation threats. These steps include:

  • Disabling outgoing network interaction
  • Capping max processes at 256 by default (resists :(){ :|: &}:;, while True: os.fork(), etc.)
  • Capping max files at 2048 (resists various file based attacks)
  • Cleaning up all temp space after each execution (resists out of drive space attacks)
  • Running as a variety of unprivileged users
  • Capping runtime execution at 3 seconds
  • Capping stdout to 65536 characters (resists yes/no bombs and runaway output)
  • SIGKILLing misbehaving code

License

Piston is licensed under the MIT license.

piston's People

Contributors

brikaa avatar ccall48 avatar dependabot[bot] avatar dev-null-undefined avatar dthree avatar dvargas46 avatar enderchief avatar ethancedwards8 avatar frostydan avatar hexf avatar isakal avatar jonxslays avatar kevrocks67 avatar lorypelli avatar lvfrazao avatar neysanfoo avatar niki4tap avatar pan93412 avatar pizzalord22 avatar pyroseza avatar realtux avatar scpchicken avatar shhivv avatar shivansh-007 avatar sirius902 avatar ssahai avatar titouant avatar tommyxr avatar vladexa avatar vrganj avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

piston's Issues

scala executor

See packages/ for existing examples
Please merge to v3 branch!

prolog executor

See packages/ for existing examples
Please merge to v3 branch!

Bash executor

See packages/ for existing examples

Please merge to v3 branch!

Piston API

Piston API is the core API which manages the lifecycle of languages and code execution.

The Piston API entails both the package manager and execution manager.

[Request] Add support for POSIX shell/dash

Since dash (an implementation of POSIX sh) is already installed, it would be nice to be able to directly use it (instead of having to write to a file then execute dash with it).

Different Field for Std Err

For example:

{
	"input": "there",
	"source": "print('hello' + 1)",
	"language": "python"
}

should probably respond with

{
  "ran": true,
  "language": "python",
  "version": "3.6.9",
  "output": "",
  "error": "Traceback (most recent call last):\n  File \"code.code\", line 1, in <module>\n    print('hello' + 1)\nTypeError: can only concatenate str (not \"int\") to str"
}

AWK executor

See packages/ for existing examples

Please merge to v3 branch!

Output whitespace is trimmed

When running the below code

print(' ')

and

#include <iostream>
#include <iomanip>
std::cout << std::setw(8) << "hello" << '\n';

I would expect the outputs to be a single space for the first and

   hello

for the second but instead the bot says "Your code ran without output" on the first and outputs

hello

for the second.

Edit: the same thing happens with tabs.

crystal executor

See packages/ for existing examples

Please merge to v3 branch!

rust executor

See packages/ for existing examples
Please merge to v3 branch!

Hosted version appears vulnerable to CVE-2021-3156

Recent exploit was published on sudo: https://www.qualys.com/2021/01/26/cve-2021-3156/baron-samedit-heap-based-overflow-sudo.txt

The hosted version of piston on emkc.org though I'm not sure if this is serious given your security model.

POC:

$ curl -sX POST https://emkc.org/api/v1/piston/execute -d '{"language":"bash","source":"sudoedit -s '"'"'\\'"'"' `perl -e '"'"'print \"A\" x 65536'"'"'`","stdin":"","args":[""]}' -H "Content-Type: application/json" | jq
{
  "ran": true,
  "language": "bash",
  "version": "4.4.20",
  "output": "code.code: line 1:  2264 Segmentation fault      sudoedit -s '\\' `perl -e 'print \"A\" x 65536'`",
  "stdout": "",
  "stderr": "code.code: line 1:  2264 Segmentation fault      sudoedit -s '\\' `perl -e 'print \"A\" x 65536'`"
}

Accept Standard Input

For example:

{
	"input": "there",
	"source": "print('hello ' + input())",
	"language": "python"
}

should respond with

{
  "ran": true,
  "language": "python",
  "version": "3.6.9",
  "output": "hello there"
}

Ability to install PIP packages for Python code execution

Most of the time users want a sandbox environment to test how a function from a package works; it would be very useful to have an option to provide a list of packages that should be installed on the container before code execution.

Something like this would be my suggestion to keep backward compatibility:

{
    "language": "python",
    "source": "import bcrypt\npassword = b\"super secret password\"\nhashed = bcrypt.hashpw(password, bcrypt.gensalt())\nif bcrypt.checkpw(password, hashed):\n    print('It Matches!')\nelse:\n    print('It Does not Match :(')",
    "stdin": "",
    "args": [
    ],
    "packages": [
        "bcrypt"
    ]
}

node executor

See packages/ for existing examples
Please merge to v3 branch!

php executor

See packages/ for existing examples
Please merge to v3 branch!

nim executor

See packages/ for existing examples
Please merge to v3 branch!

perl executor

See packages/ for existing examples
Please merge to v3 branch!

python3 executor

See packages/ for existing examples
Please merge to v3 branch!

zig executor

See packages/ for existing examples

Please merge to v3 branch!

Update bash

You should update bash, the current one is ~4 years old.

c executor

See packages/ for existing examples

Please merge to v3 branch!

API returns 200 on invalid language

POST to https://emkc.org/api/v1/piston/execute with a language that isn't whitelisted will result in a 200 with the response

{"output":""}

Here's some code to reproduce it:

fetch('https://emkc.org/api/v1/piston/execute', {
	method: 'POST',
	headers: {
		'Content-Type': 'application/json'
	},
	body: JSON.stringify({ language: 'randomstuff123', source: 'potatoes', args: [] }),
}).then(res => res.json()).then(console.log);

python2 executor

See packages/ for existing examples
Please merge to v3 branch!

cpp executor

See packages/ for existing examples

Please merge to v3 branch!

Run Rust Test snippents.

engineer-man/piston-bot#5

I would like to implement this functionality in piston.
There are two ways I think I can do this.

  • Add rust_test as a new language which would require the least changes.
  • Add test as an arg to the rustc.
    • This might produce some changes in the current rust language behaviour.

Which path would be preferable.

Update rust

Update rust to a newer version, the one used currently is from 2019

Update C++ Standard to C++20

Since the C++20 standard has been published in December 2020 piston should use it.

This could be as easy as changing -std=c++17 to -std++20 in

timeout -s KILL 10 g++ -std=c++17 -o binary -x c++ code.code

but I'm not sure if the installed g++ compiler supports C++20

Preserve newlines in stdin

Stdin is passed as argv, which is probably why it turns newlines into spaces. To get around this, the stdin file should be written to directly, the same way the code is.

By the way, why does the API write to a temporary file and let the executor copy it instead of just writing to the final path? Is there a reason for that? I just copied what the previous API did when rewriting it.

osabie executor

See packages/ for existing examples
Please merge to v3 branch!

No response when running certain programs

Programs like:

bash

yes

and

cpp

#include <iostream>

int main()
{
    while (true)
    {
        &printf == 0x000;
        printf("test");
    }
}

cause the API to "hang" and not send any reply.

paradoc executor

See packages/ for existing examples
Please merge to v3 branch!

Failed to fetch when installing

When i try to do the following step:

install all necessary piston dependencies

The following error comes up: (when running apt-get update)

root@piston:/# echo 'source /opt/.profile' >> /opt/.bashrc
root@piston:/# echo 'export HOME=/opt' >> /opt/.profile
root@piston:/# echo 'export TERM=linux' >> /opt/.profile
root@piston:/# echo 'export PATH=$PATH:/opt/.local/bin' >> /opt/.profile
root@piston:/# export HOME=/opt
root@piston:/# export TERM=linux
root@piston:/# sed -i 's//root//opt/' /etc/passwd
root@piston:/# sed -i \

's/http:\/\/archive.ubuntu.com\/ubuntu/http:\/\/mirror.math.princeton.edu\/pub\/ubuntu/' \
/etc/apt/sources.list

root@piston:/# apt-get update
Err:1 http://security.ubuntu.com/ubuntu bionic-security InRelease
Temporary failure resolving 'security.ubuntu.com'
Err:2 http://mirror.math.princeton.edu/pub/ubuntu bionic InRelease
Temporary failure resolving 'mirror.math.princeton.edu'
Err:3 http://mirror.math.princeton.edu/pub/ubuntu bionic-updates InRelease
Temporary failure resolving 'mirror.math.princeton.edu'
Reading package lists... Done
W: Failed to fetch http://mirror.math.princeton.edu/pub/ubuntu/dists/bionic/InRelease Temporary failure resolving 'mirror.math.princeton.edu'
W: Failed to fetch http://mirror.math.princeton.edu/pub/ubuntu/dists/bionic-updates/InRelease Temporary failure resolving 'mirror.math.princeton.edu'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease Temporary failure resolving 'security.ubuntu.com'

Im on a debian VPS, can someone help me out?

ruby executor

See packages/ for existing examples
Please merge to v3 branch!

Deno cannot import packages

Deno allows you to import from remote URLs in your code when building, however piston disables networking.
Some libraries are required (e.g. std), which you pull from a remote url

import { printf } from "https://deno.land/std/fmt/printf.ts";
printf("Hello world!")

Minor nit- Platforms

https://github.com/engineer-man/piston/blob/master/var/install.txt#L19 mentions debian/arch/mac os

You have Arch 😂 debian should be analogous to Ubuntu. Ubuntu is based from Debian. I am 99% certain the packages you install have the same names. OSx would run in a VM, which could be any of the above three (that is how docker on osx works).

Suggestion

Remove at least arch. Maybe debian. Maybe make a docker image using alpine so Mac users can do something almost as easy as brew install (docker run ...)

crystal executor

See packages/ for existing examples

Please merge to v3 branch!

swift executor

See packages/ for existing examples
Please merge to v3 branch!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.