Giter VIP home page Giter VIP logo

angreal's Introduction

Angreal

image PyPI - Downloads image Angreal Tests Angreal Docs Deploy Angreal Release

Docs are available here.

Angreal is meant to:

  • allow the consistent creation of projects
  • provide consistent methods for interacting with projects

Quick Start

  1. Install via pip
  2. Initialize a project from a template
  3. Use the template
$: pip install 'angreal>=2' #pip install angreal will also work
$: angreal init https://github.com/angreal/python

What is it?

Angreal is an attempt to solve two problems that I was running into in both my personal and professional life as a data scientist and software developer. I do things often enough that they needed automation, I don't do things so often that I remember all of the steps/commands I might need to get them done. Angreal solves this problem by allowing me to remember by forgetting : I only have to remember the command to do something not the actual steps to complete the task.

How does it solve these challenges ?

Angreal provides a way to template the structure of projects and a way of executing methods for interacting with that project in a consistent manner. These methods (called tasks) travel with the project so while templated initially, they're customizable to the project - allowing some level of flexibility in how a task functions between projects.

Why 2.0 ?

The original angreal was built on top of a number of python modules that were under active development and used by a number of other projects. The nature of the application itself meant that core application found itself in dependency hell regularly - and became rather annoying to use. The 2.0.0 release is a complete rewrite that uses Rust to provide a compiled binary with the goal that it will require no external python dependencies.

angreal's People

Contributors

ashb avatar dependabot[bot] avatar dylanbstorey avatar rodrigogiraoserrao avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

angreal's Issues

airflow template

As a developer I would like a template to bootstrap an airflow deployment for prototyping.

Positive exit on Init

As a user I would like to know that the init command succeeded in a noticable way, regardless of what happens (or doesn't happen) in the init.py script.

compound command structure

I would like to be able to provide a sub command structure so that someone can group commands into logical groups.

i.e.

angreal dev start
angreal dev stop
angreal dev clean

vs.

angreal dev-start
angreal dev-stop
angreal dev-clean

grouped commands aren't accepting arguments (correctly or at all ?)

@angreal.group(name="test_",about="commands for testing")
@angreal.command(name="echo", about="")
@angreal.argument(name="phrase", help="the phrase to echo", required=True)
def build_hugo(phrase):
    print(phrase)

invoked with

angreal test_ docs weee

should output

wee

does output

functools.partial(<function group at 0x102e6b060>, name='test', about='commands for testing the application and library')

init behavior

init behavior doesn't appear to be 100% correct, not checking /angreal prefix for projects in the .angrealrc folder.

airflow provider template

As a developer i would like to have a template to bootstrap an airflow provider.

Features:

  • code testing
  • integration testing
  • example dag deployment

render templated file structure from python

As a developer I would like to be able to render a templated folder/file structure from the python api so that i can add structured folders to my project during development.

We should be able to support a context handed or a repl style interface for getting variable values (i.e. angreal.toml, or command generated values)

Example:

  • in a python mono repo i would define a python service template and task so that when i execute angreal new service , a new templated structure would appear in the appropriate folder.
angreal.render_templated_director("/path/to/top_dir" , "/path/to/dest/", context = {variable_name: value})


def render_templated_directory(source_path, destination_path, context, remove_on_failure=True):

    try:
         render 
    except:
        if remove_on_failure:
             os.unlink(destination)
       raise
  1. refactor init::render_template, pull out the context generator as a seperate function f(path)->context
  2. create a context->toml serde function f(context)->map
  3. create a render_dir() rust function f(dir,context)->ok()
  4. re-implement render-template
  5. expose/wrap render_dir() via pyo3, incoming context will need to get serde to context object

Code Refactors

Code smells :

  • Got a fair amount of copy / pasta and the init::init function doesn't feel very good as a result of hacking.
  • Functions aren't returning Result<>, we should use that with unwrap()
  • Some internal methods probably aren't namespaced intelligently.

Deprecated click method get_os_args

Hi Dylan, we found and issue with angreal, where the method get_os_args is used from click, which has been deprecated. We've created a PR to remove this method from being imported. Thank you

More dev ex

  • Document the development process more fully. (how to test, how to release, etc.)
  • Update/Expand the tasks for angreal development more fully.
  • Add how to / discussion articles on :
  • command groups discussion on how chains are build / nested commands can overlap in weird ways if you're not careful
  • virtual environments, how they're built / executed on
  • docker-py / integration

Scheme for dealing init some "default" templates

As a user I'd like to be able to use angreal init <template name> instead of knowing a full repo path.

Options :

  • assume the repo is in github to begin with so angreal init test would be angreal init github.com/test.git , angreal init my/test would be angreal init github.com/my/test.git

  • create some "official" repos so angreal init test would be angreal init github.com/angreal/test.git

  • some mix of the 2

CI/CD should include windows platform tests.

Got a report from Rodrigo that initialization is failing with an OS level error on windows. Add testing on all platforms to the CI/CD stack. (angreal run-tests on mac/linux/windows should be sufficient for now)

templating api exposed from angreal

As an author
I would like to be able to easily render templates, without additional dependencies, within a task
So that I can provide a template task to my users.

angreal template

As a developer I would like an angreal template for bootsrapping angreal templates.

It should include :

  • github action for testing template rendering

Have CLI check for new releases on startup.

Users should get a friendly notice of new releases if one exists.

import requests
response = requests.get(f'https://pypi.org/pypi/angreal/json')
latest_version = response.json()['info']['version']

new template flow

As a developer,

I would like place a template into {$HOME}/.angrealrc/my-template

So that I can run angreal init my-template

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.