Hello,

I was testing greenframe-cli with the following testing script :

import { test, expect } from '@playwright/test';

test('has title', async ({ page }) => {
  await page.goto('https://playwright.dev/');
  await expect(page).toHaveTitle(/Playwright/);
});

test('get started link', async ({ page }) => {
  await page.goto('https://playwright.dev/');
  await page.getByRole('link', { name: 'Get started' }).click();
  await expect(page.getByRole('heading', { name: 'Installation' })).toBeVisible();
});

test('Playwright Test', async ({ page }) => {
  await page.goto('https://playwright.dev/');
  await expect(page).toHaveTitle(/Playwright/);
  const getStarted = page.getByText('Get started');
  await expect(getStarted).toHaveAttribute('href', '/docs/intro');
  await getStarted.click();
  await expect(page).toHaveURL(/.*intro/);
});

but it raised an error, since it does not recognize the test and expect keywords, even in the package.json of the greenframe-cli container the @playwright/test library.

Howdy 👋

Problem

Greenframe CLI analyzes never reaches an end.

Reproduction steps

The following command remains pending and never produces a result.

$ cd ~
$ greenframe analyze https://marmelab.com 
✔ Check configuration file
✔ The folder is not a git repository
❯ Analysis is in progress locally
  ✔ Docker version 20.10.23, build 7155243
  ⠼ Running 1 scenario(s)... # this step never stops

Information

➜  ~ node -v
v16.20.1
➜  ~ greenframe -v                          
greenframe-cli/1.6.8 darwin-arm64 node-v16.20.1

Am I missing something? Any idea what's going on?

Playwright is wonderful but many organizations including my own have written all our tests using Cyprus. It would be ideal to at a flip of the switch run green frame using Cyprus where migration to Playwright is unrealistic and/or political. It is difficult for me and I would suppose others to sell the idea of measuring carbon emissions to our teams let alone requiring migration to Playwright. Framework agnostic or at least support of more popular systems like Cyprus would make for a much easier sell and transition to greening software.

Side Note: TestCafe is also a very well known test ecosystem. I'm sure they would love to have something like this on their cloud and part of their enterprise studio solution.

Great tool! Discovered this during a Kubecon EU talk.

Is it possible to implement functionality so greenframe can export metrics in OpenTelemetry format so that other system can consume the output?

I've been reading through your scientific model and I came across some irritating syntax. So the following paragraph

• cpuWh = pue _ avg(cpu.totalUsageInUserMode + cpu.totalUsageInKernelMode) _ config.CPU / 3600
• memWh = pue _ avg(memory.usage) _ config.MEM * totalTime / 3600
• diskWh = pue _ avg(io.totalByte) _ config.DISK
• networkWh = avg(network.totalReceived + network.totalTransmitted) * config.NETWORK / 2
• screenWh = avg(userTime) * config.SCREEN / 3600

contains this specific expressions:

pue _ ...

I'm unsure what this expression should be saying. E.g. is it a multiplication or division? However, this might as well be missing knowledge on my side for scientific maths. I would appreciate some clarification here!
Thanks

Problem

No playwright configuration

Can't pass options to browser or context, like ignoreHTTPErrors, locale, timezone. Every time we want to support a playwright feature, we need to implement a new flag and publish a new version

Can't reuse existing playwright tests

If I'm already using playwright as my e2e test framework and have some actual e2e tests, I would like to be able to use the same files and not have to maintain the same test for playwright, and for greenframe

Can't use page objects

Since I can't import local files in my greenframe tests, I can't implement the page object pattern on my scenarios. It's really a pity because this pattern is very common and makes tests so much more readable.

VM2 isolation is not needed in the CLI

We don't need VM2 isolation when executing tests locally, which is what greenframe-cli is doing.
VM2 makes it impossible to use imports and such.

Distant mode still needs to work

For now, it only sends the scenario files set in the configuration file. But if we want to be able to plug greenframe to any pre existing test suite, we need to be able to have the full code, imported modules too.

Solution for local open and analyze

• Remove VM2
• Execute playwright programmatically directly from the CLI code, with all the context of the project we're in. (playwright can't be executed programmatically, we'll have to do a bash exec with npx run playwright)
• Add a greenframe init that proxies playwright init ?

This will break the functions we added to playwright page object.
We should probably have a @greenframe/playwright that overrides the page object of playwright to add waitForNetworkIdle, scrollToElement, scrollByDistance, scrollToEnd, addMilestone, getMilestones.

Solution for distant mode

In order for us to execute distant playwright tests, we need to...

• resolve all local imports
• allow people to import "@playwright/test"
• allow people to use pure playwright test

My guess is that, if we transpile the playwright test locally before sending to greenframe servers, it should include local imports (and maybe even playwright and all other libs or playwright extensions used)
We could even support typescript if we do that.

The analyzes running on greenframe servers should still use VM2 and disallow all external or node-builtin(fs,fetch...) module access.

We can probably do a cheaper solution by disallowing any import other than @playwright/test and @greenframe/playwright in the tests files. It should be good enough for people wanting distant analyzes.

Breaking changes

All previously written scenarios won't work anymore.
They will need a few changes:

• import @playwright/test
• import @greenframe/playwright if any GF functions were used
• wrap the function in test()

This can probably be codmodded

Just trying to follow the usage model https://github.com/marmelab/greenframe-cli/tree/main#usage

in greenframe-cli/1.6.8 darwin-arm64 node-v16.20.1

I'm getting this error with % greenframe open https://marmelab.com ./my-scenario.js

Running 1 scenarios...
❌ Error : main scenario
Unexpected token 'const'

I'm just using the basic scenario provided % cat my-scenario.js                            

// in my-scenario.js
const scenario = async (page) => {
    await page.goto('', { waitUntil: 'networkidle' }); // Go to the baseUrl
    await page.waitForTimeout(3000); // Wait for 3 seconds
    await page.scrollToElement('footer'); // Scroll to the footer (if present)
    await page.waitForNetworkIdle(); // Wait every request has been answered as a normal user.
};

module.exports = scenario;

I am analyzing the footprint of a Wordpress stack. I get significantly different scores for the browser device depending on the presence or absence of containers (ie, with or without the --containers option).

I used the Docker Compose setup described here.

Full stack estimation (browser + Apache + DB):

greenframe analyze http://localhost:8080 --containers="wordpress-benchmarks-wordpress-1" --databaseContainers="wordpress-benchmarks-db-1"
[…]
The estimated footprint is 10.177 mg eq. co2 ± 1.3% (23.025 mWh).

Removing the Apache server (estimating browser + DB):

reenframe analyze http://localhost:8080 --databaseContainers="wordpress-benchmarks-db-1"
[…]
The estimated footprint is 5.953 mg eq. co2 ± 1.1% (13.468 mWh).

Removing the DB (estimating browser + Apache):

greenframe analyze http://localhost:8080 --containers="wordpress-benchmarks-wordpress-1"
[…]
The estimated footprint is 8.828 mg eq. co2 ± 1.9% (19.973 mWh).

From the executions above, we can deduce the following break down:

Note that we deduced the energy consumption of the browser by subtracting the Apache and DB consumption from the total. Now, if we run the analysis again but for the browser only (without any --containers options), we would expect to get something close to 10.413 mWh. However, this is not what we get:

greenframe analyze http://localhost:8080
[…]
The estimated footprint is 8.343 mg eq. co2 ± 1.6% (18.875 mWh).

We get 18.9 instead of 10.4.

I also customized the output of greenframe-cli to show a per container breakdown (so that I don’t need to do the subtraction manually), and here is what I got.

Full-stack analysis (browser + Apache + DB):

bin/run analyze http://localhost:8080 --containers="wordpress-benchmarks-wordpress-1" --databaseContainers="wordpress-benchmarks-db-1"
[…]
The estimated footprint is 8.615 mg eq. co2 ± 0.7% (19.49 mWh).
  For container greenframe-runner (DEVICE), the footprint is 5.693 mg eq. co2 (12.88 mWh) ;
  For container wordpress-benchmarks-wordpress-1 (SERVER), the footprint is 2.78 mg eq. co2 (6.291 mWh) ;
  For container wordpress-benchmarks-db-1 (DATABASE), the footprint is 0.141 mg eq. co2 (0.319 mWh) ;

Browser only:

bin/run analyze http://localhost:8080
[…]
The estimated footprint is 8.38 mg eq. co2 ± 1.6% (18.959 mWh).
  For container greenframe-runner (DEVICE), the footprint is 8.38 mg eq. co2 (18.959 mWh) ;

I find a similar difference in the estimation of the browser energy consumption (12.88 vs 18.96) depending on the options passed to greenframe-cli.

I don’t think those differences are caused by the expected variability we should get when running those benchmarks on a laptop, where my OS runs many background tasks at the same time because when I try to re-run greenframe-cli I consistently get the same differences.

My goal is to be able to compare different stacks, which means that I will have to compare the output of greenframe-cli run with different --containers options (e.g. to compare a static site to a Wordpress stack), but I am not sure this way of using greenframe-cli is reliable. Maybe it has been designed to compare the results of identical invocations only?

I performed another test to compare a static site served by Nginx with a dynamically generated site served by Apache + PHP and I got very confusing results where the energy consumption of the browser only (greenframe analyse http://localhost) was higher than the energy consumption of the browser + Apache (greenframe analyse http://localhost --containers=php).

As it is intended to be used in CI, it would be nice to dockerize greenframe-cli, no?

I just followed the installation procedure:

➜  greenframe-cli git:(main) ./bin/run analyze https://greenframe.io
✔ Check configuration file
✔ Retrieving Git information
❯ Analysis is in progress locally
  ✔ Docker version 24.0.2, build cb74dfc
  ✖ Could not connect to Docker daemon on undefined:undefined

❌ Failed!
ConfigurationError
Could not connect to Docker daemon on undefined:undefined

Contact us for support: [email protected]

Currently it is not possible to override kubeContainers and kubeDatabaseContainers parameters with a command line flag

$ greenframe analyze --help
Create an analysis on GreenFrame server.

USAGE
  $ greenframe analyze [BASEURL] [SCENARIO] [-C <value>] [-K <value>] [-t <value>] [-p <value>] [-c <value>] [--commitId <value>] [-b
    <value>] [-s <value>] [-d] [-a] [-i] [--dockerdHost <value>] [--dockerdPort <value>] [--containers <value>] [--databaseContainers <value>]

ARGUMENTS
  BASEURL   Your baseURL website
  SCENARIO  Path to your GreenFrame scenario

FLAGS
  -C, --configFile=<value>      Path to config file
  -K, --kubeConfig=<value>      Path to kubernetes client config file
  -a, --useAdblock              Use an adblocker during analysis
  -b, --branchName=<value>      Pass branch name manually
  -c, --commitMessage=<value>   Pass commit message manually
  -d, --distant                 Run a distant analysis on GreenFrame Server instead of locally
  -i, --ignoreHTTPSErrors       Ignore HTTPS errors during analysis
  -p, --projectName=<value>     Project name
  -s, --samples=<value>         Number of runs done for the score computation
  -t, --threshold=<value>       Consumption threshold
  --commitId=<value>            Pass commit id manually
  --containers=<value>          Pass containers manually
  --databaseContainers=<value>  Pass database containers manually

<-- Missing kubeContainers and kubeDatabaseContainers flags -->

  --dockerdHost=<value>         Docker daemon host
  --dockerdPort=<value>         Docker daemon port

DESCRIPTION
  Create an analysis on GreenFrame server.

When I try to add these parameters anyway, I get the following error:

Unexpected argument: --kubeContainers=....

Hi all !

I'm trying to run an analysis with Greenframe CLI in a Gitlab CI. It's a fullstack analysis on a Kubernetes cluster. I built a custom Docker image to run it, from Node 16 Alpine, with docker and kubectl installed.

But this analysis failed when try to run the getHostIP.sh script. Grep version is not the same in Alpine Linux and doesn't have the -P option.

Here is the complete output :

$ greenframe --version
greenframe-cli/1.6.8 linux-x64 node-v16.20.2
$ greenframe kube-config
[STARTED] Check configuration file
[SUCCESS] Check configuration file
[STARTED] Intializing kubernetes client
[SUCCESS] Intializing kubernetes client
[STARTED] Creating greenframe namespace
[TITLE] Greenframe namespace already exists
[SUCCESS] Greenframe namespace already exists
[STARTED] Creating greenframe daemonset
[TITLE] Greenframe daemonset already exists
[SUCCESS] Greenframe daemonset already exists
Kubernetes configuration complete !
$ greenframe analyze
[STARTED] Check configuration file
[SUCCESS] Check configuration file
[STARTED] Retrieving Git information
[SUCCESS] Retrieving Git information
[STARTED] Analysis is in progress locally
[STARTED] Detect docker version
[TITLE] Docker version 24.0.6, build ed223bc
[SUCCESS] Docker version 24.0.6, build ed223bc
[STARTED] Initialize Kubernetes client
[SUCCESS] Initialize Kubernetes client
[STARTED] Detect kubernetes version
[TITLE] Kubernetes version 1.24, build v1.24.14-gke.2700, platform linux/amd64
[SUCCESS] Kubernetes version 1.24, build v1.24.14-gke.2700, platform linux/amd64
[STARTED] Running 1 scenario(s)...
[SUCCESS] Running 1 scenario(s)...
[SUCCESS] Analysis is in progress locally
Analysis complete !
Result summary:
❌ main scenario failed
This scenario fail during the execution:
Command failed: /root/.local/lib/greenframe/dist/bash/getHostIP.sh
grep: unrecognized option: P
BusyBox v1.36.1 (2023-07-27 17:12:24 UTC) multi-call binary.
Usage: grep [-HhnlLoqvsrRiwFE] [-m N] [-A|B|C N] { PATTERN | -e PATTERN... | -f FILE... } [FILE]...
Search for PATTERN in FILEs (or stdin)
	-H	Add 'filename:' prefix
	-h	Do not add 'filename:' prefix
	-n	Add 'line_no:' prefix
	-l	Show only names of files that match
	-L	Show only names of files that don't match
	-c	Show only count of matching lines
	-o	Show only the matching part of line
	-q	Quiet. Return 0 if PATTERN is found, 1 otherwise
	-v	Select non-matching lines
	-s	Suppress open and read errors
	-r	Recurse
	-R	Recurse and dereference symlinks
	-i	Ignore case
	-w	Match whole words only
	-x	Match whole lines only
	-F	PATTERN is a literal (not regexp)
	-E	PATTERN is an extended regexp
	-m N	Match up to N times per file
	-A N	Print N lines of trailing context
	-B N	Print N lines of leading context
	-C N	Same as '-A N -B N'
	-e PTRN	Pattern to match
	-f FILE	Read pattern from file
ash: usage: printf FORMAT [ARGUMENT...]
Use greenframe open command to run your scenario in debug mode.

Since we are able to calculate the estimated Co2 emmissions on the stack, it could be a cool idea for a service that let's the users opt in to be charged the cost of the Co2 offset of their stack.

Then each month, said company gets an invoice stating X amount of Co2 is used by their stack and it has been offset for a cost of Y.

This could also be a part of the website embed telling visitors that the emissions are offset to good projects.

I have an issue with the greenframe analyze cli in a kubernetes execution context.
All the analyse I run fails with the error :

Cannot find network container for pod pod-name-746fd5878bvvmnd

I have not made any changes to my kubernetes cluster.

greenframe-cli/1.6.6 linux-x64 node-v16.19.1
Kubernetes version 1.23, build v1.23.14-gke.1800, platform linux/amd64

@fzaninotto @guilbill What’s going on?

Hi there,

I was just wondering if currently there is a way to retrieve the details provided by the GreenFrame online report at https://app.greenframe.io/YOUR_ANALYSIS_ID, directly from the results of the following command:

GREENFRAME_SECRET_TOKEN=your-token-here greenframe analyze https://marmelab.com

Can anyone of the dev just confirm that in this moment this kind of functionality is not available?

With advanced metrics/details, I'm referring to this data:

Thank you in advance.

Hi all,

I have installed the cli on linux vm. When i try to call analyze for any website it returns the error: greenframe-runner container is not running
Docker is up and running. What might be a problem?