Giter VIP home page Giter VIP logo

green-cost-explorer's Introduction

Green Web Foundation API

In this repo you can find the source code for the API and checking code that the Green Web Foundation servers use to check the power a domain uses.

Build Status

Overview

Following Simon Brown's C4 model this repo includes the API server code, along with the green check worker code in packages/greencheck.

API

Apps - API Server at api.thegreenwebfoundation.org

This repository contains the code served to you when you visit http://api.thegreenwebfoundation.org.

When requests come in, symfony accepts and validates the request, and creates a job for enqeueue to service with a worker.

API

The greenweb api application running on https://api.thegreenwebfoundation.org

This provides a backend for the browser extensions and the website on https://www.thegreenwebfoundation.org

This needs:

  • an enqueue adapter, like fs for development, amqp for production
  • php 7.3
  • nginx
  • redis for greencheck library
  • ansible and ssh access to server for deploys

Currently runs on symfony 5.x

To start development:

  • Clone the monorepo git clone [email protected]:thegreenwebfoundation/thegreenwebfoundation.git
  • Configure .env.local (copy from .env) for a local mysql database
  • composer install
  • bin/console server:run
  • check the fixtures in packages/greencheck/src/TGWF/Fixtures to setup a fixture database

To deploy:

  • bin/deploy

To test locally:

Packages - Greencheck

In packages/greencheck is the library used for carrying out checks against the Green Web Foundation Database. Workers take jobs in a RabbitMQ queue, and call the greencheck code to return the result quickly, before passing the result, RPC-style to the original calling code in symfony API server.

API

Packages - public suffix

In packages/publicsuffix is a library provides helpers for retrieving the public suffix of a domain name based on the Mozilla Public Suffix list. Used by the API Server.

green-cost-explorer's People

Contributors

allcontributors[bot] avatar arendjantetteroo avatar dependabot[bot] avatar edsu avatar jell avatar mrchrisadams avatar sjwarner-bp avatar vsmart avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

green-cost-explorer's Issues

Include Costs Paid for by Credits

I just ran this script on our AWS account. This is the result I got:

│ Total Green Cost             │ Total Grey Cost              │
├──────────────────────────────┼──────────────────────────────┤
│ 100.0% ($0.08)               │ 0.0% ($0.00)                 │

Now, that's good, because we run everything in Ireland anyway, but it's bad because we definitely spent more than 8 cents last month.

Our AWS bill is fully covered by credits we got from them. It seems that the script is excluding these costs in the calculation.

I believe that a lot of companies, especially startups, have vast amounts of credits from AWS via their various accelerator programs. I think that these "costs" should be included in the calculation.


After some googling, it seems that CostExplorer.getCostAndUsage supports this via something like:

    Dimensions: {
      Key: 'RECORD_TYPE',
      Values: [
        'Credit',
      ]
    },

I'll try to get a PR up for this.

Add nicer CLI flags and output

Right now, we just dump a bunch of javascript objects to the command line.

This isn't very helpful.

What would be a nice start, would be to get a lighthouse ci style breakdown like so:

lighthouse-ci-cli-ui

What would be really nice would be to have a flag to output json in a format that works nicely with vega-lite.

This would make it straightforward to talk about analysis using observable, or if you understandably aren't crazy about sharing AWS API keys onto thirty platforms, run the checks locally.

https://beta.observablehq.com/@mbostock/exploring-data-with-vega-lite

Set up a nice deployment/release pipeline

We'd like a nice way to get releases onto npm easily.

This would include

  • setting up with travis
  • setting up something like np gem
  • something like greenkeeper or similar to keep deps safe and er... dependable

Refactor into smaller, testable modules

Now we have this running, and we have an idea of what we want to support with this, we spoke about refactoring this, to make it easier to change, as the single file is doing a bunch of work:

A proposed structure

We discussed breaking this into three smaller components, like this picture here:

IMG_20190815_154914

  • CLI parser - this takes care of setting attributes based on the flags we pass in on the command line
  • filter/grouper - this takes the data we care get back from cost explorer, and reshapes it so it's easy to represent in json, CLI output, on a page, and so on
  • formatter - this takes the datastructure we generate with the component above, and takes care of formatting it in the chosen output format (i.e. as a json file, as html, or a terminal table)

Not published in npm?

npx greencost does not work...
npm ERR! 404 'greencost@latest' is not in the npm registry.

Is this project still maintained?

I just want to know if this project is still maintained. Dont want to come off as rude, but it doesnt look like that atm (no commits since 2020).

When running I'll get greeted by:

NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.

Create output that can easily be rendered with Vega Lite

If we make this output in the format vegalite expects, we can then rely on VegaLite to give us all kind of nice viz, aggregations and rollups, which then makes it easy to embed in dashboards or anywhere you might use vega embed:

https://github.com/vega/vega-embed

This gives us nice PNG and PDF outputs to make it easy for people to drop into reports, emails, slack channels, etc and make the case for getting off fossil fuels.

For this, I think the key thing we need is some notion of a timestamp in each json object for the region.

Warn if the AWS keys in use have more privileges than you need for cost explorer

Because you can npm install js in loads of places, and because you could plausibly put this on glitch or other such services, it makes sense to have some safeguard to either warn you, or not run if a set of credentials with more privileges than are strictly needed have been detected.

I don't know how to do this yet, but I'm hoping some kind soul might.

C

Add breakdown by service

If you have a target of getting off fossil fuels, it helps to know which services you are spending on, as some things are easier to move across regions than others.

Also, if we had a breakdown by service, then it would be possible to show charts like this, but grouped by if they can in green vs grey regions as a way to track progress over time

cost-breakdown

This is from the post below. Worth a read.

https://blog.mapbox.com/tracking-aws-infrastructure-cost-on-a-team-level-5b84c3fc8467?gi=2e6b996a0823

Add better error messages when you don't have credentials

When we were pairing we came across a few things that weren't obvious to debug.

Depending on the kinds of errors, we should probably:

  • Make list of the common errors we've seen so far
  • Make list of the common errors you see elsewhere
  • Link to the most relevant pages on the AWS SDK help docs when we know them

Project Hygiene

This project needs a bunch of stuff like:

  • CoC
  • proper coding and contribution guidelines
  • setting up with CI
  • a roadmap

And more. Leaving this ticket here as a reminder (and invitation to any product people…)

Handle "NoRegion"/"Global" charges

Some charges have no region info attached to them or are global.

How should these be handled? Grey by default, a third "neutral" category, or disregarded entirely?

Load in indicative figures for to convert to CO2 from money spent.

Until cloud providers provide more transparency on how much power, and therefore carbon any of the services emit, we can get some ballpark figures by either:

  1. using Amazon's own emissions intensity figures from their own sustainability reporting. This is for all of Amazon, not AWS. So we'll have innaccuracies as running exclusively datacentres have a different energy use profile to all of e-commerce.

or

  1. adapt this spreadsheet, and in particular the Platform part, I built when working with another agency to get some figures. For this we used the UK figures for emissions intensity of IT from the ONS, adjusted for what we knew about likely energy use, and straight up cost of energy.

Show cost data over time

As outlined in the readme, show cost data split over time using the API's given time series chunks:

|          | Grey   | Green  |
-----------|--------|--------|
|Aug  2018 |  40%   |  60%   |
|Sept 2018 |  40%   |  60%   |
|Oct  2018 |  40%   |  60%   |
|Nov  2018 |  40%   |  60%   |
|Dec  2018 |  40%   |  60%   |

rather than only

|          | Grey   | Green  |
-----------|--------|--------|
|Total     |  40%   |  60%   |

Changes are needed in two places to do this:

  1. note timestamp when aggregating data by region here
  2. respect timestamp when summing up total here

cc @mrchrisadams

Accept command line flags for setting dates, and ideally tags to filter by

Right now the we have hard coded dates for running checks.

There's a normal 1 year rolling window I think for cost explorer, but it would be nice to set the time bounds, like you can with the python cli.

For example, when you have this:

aws ce get-cost-and-usage --time-period Start=2018-08-01,End=2019-05-01 --granularity MONTHLY --metrics "BlendedCost" "UsageQuantity" --group-by Type=DIMENSION,Key=AZ

I would be good to do something to either use the same kinds of flags:

greencost --time-period Start=2018-08-01,End=2019-05-01

Or ideally use flags like so, which

I'm not sure what the convention is for node js, but if we know we only ever have maximum of today plus one year back, we might be able to have some sensible defaults to show the last year of spend analysis, or be able to set arbitrary figures like so:

greencost --time-period-start 2018-08-01 --time-period-end 2019-05-01

I'm not sure what the most command CLI tool for node is, but commander looks well documented, and is used by the Vue CLI, so there's code examples we can look to for seeing how different things are implemented

https://www.npmjs.com/package/commander

### Bonus bit

If filtering by tags is simple, then it gives us a nice way to restrict a query to a single project team.

https://docs.aws.amazon.com/aws-cost-management/latest/APIReference/API_GetCostAndUsage.html#API_GetCostAndUsage_RequestSyntax

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.