Giter VIP home page Giter VIP logo

monocart-reporter's Introduction

Monocart Reporter

Preview

https://cenfun.github.io/monocart-reporter

Install

npm i -D monocart-reporter

Playwright Config

Note: Most examples use CommonJS by default, please move to ESM according to your needs.

// playwright.config.js
module.exports = {
    reporter: [
        ['list'],
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './test-results/report.html'
        }]
    ]
};

Playwright Docs https://playwright.dev/docs/test-reporters

Examples

Output

  • path-to/your-filename.html
    Single HTML file (data compressed), easy to transfer/deploy or open directly anywhere

Note: Test attachments (screenshots images/videos) are not included but linked with relative path in report. All attachments will be found in playwrightConfig.outputDir

// playwright.config.js
// attachments outputDir and report outputFile used same folder
const date = new Date().toISOString().slice(0, 10); //2022-10-10
const outputDir = `./test-results/${date}`;
module.exports = {
    outputDir: outputDir,
    reporter: [
        ['monocart-reporter', {  
            name: `My Test Report ${date}`,
            outputFile: `${outputDir}/index.html`
        }]
    ]
};
// deploy the folder to your report site and easy checking online
// http://your.report.site/test-results/2022-10-10
  • path-to/your-filename.json
    Separated metadata file (Already included in the above HTML and compressed, it can be deleted). Can be used for debugging or custom data collection.

Reporter Options

{
    // the report name
    name: '',

    // the output file path (relative process.cwd)
    outputFile: './test-results/report.html',

    // attachment path handler
    attachmentPath: null,
    // attachmentPath: (currentPath, extras) => `https://another-path/${currentPath}`,

    traceViewerUrl: 'https://trace.playwright.dev/?trace={traceUrl}',

    // logging levels: off, error, info, debug
    logging: 'info',

    // timezone offset in minutes, GMT+0800 = -480
    timezoneOffset: 0,

    // global coverage settings for addCoverageReport API
    coverage: null,
    // coverage: {
    //     entryFilter: (entry) => true,
    //     sourceFilter: (sourcePath) => sourcePath.search(/src\/.+/) !== -1,
    // },

    // trend data handler
    trend: null,
    // trend: () => './test-results/report.json',

    // custom tags style
    tags: null,
    // tags: {
    //     smoke: {
    //         'background': '#6F9913'
    //     },
    //     sanity: {
    //         'background': '#178F43'
    //     }
    // },

    // columns data handler
    columns: null,
    // columns: (defaultColumns) => {},

    // rows data handler (suite, case and step)
    visitor: null,
    // visitor: (data, metadata) => {},

    // enable/disable custom fields in comments. Defaults to true.
    customFieldsInComments: true,

    // onEnd hook
    onEnd: null
    // onEnd: async (reportData, helper) => {}
}

View Trace Online

The Trace Viewer requires that the trace file must be loaded over the http:// or https:// protocols without CORS issue.

  • Start a local web server with following CLI:
# serve and open report
npx monocart show-report <path-to-report>

# serve report
npx monocart serve-report <path-to-report>

The server add the http header Access-Control-Allow-Origin: * to allow requesting from any origin, it works with http://localhost:port/ or http://127.0.0.1:port/

  • To successfully work with other IP or domain, you can start web server with https:
npx monocart show-report <path-to-report> --ssl <path-to-key,path-to-cert>

For example: npx monocart show-report test-results/index.html --ssl ssl/key.pem,ssl/cert.pem

You can create and install local CA with mkcert

  • Using your own trace viewer url with option traceViewerUrl:
// reporter options
{
    name: "My Test Report",
    outputFile: './test-results/report.html',
    // defaults to 'https://trace.playwright.dev/?trace={traceUrl}'
    traceViewerUrl: 'https://your-own-trace-viewer-url/?trace={traceUrl}'
}

Custom Fields Report

You can add custom fields to the report. for example: Owner, JIRA Key etc.

Custom Columns

The report will be displayed in a Tree Grid. The columns function is used to customize the grid columns. The column properties following:

  • id (String) Column id (required)
  • name (String) Column name, shows in grid header
  • align (String) left (default), center, right
  • width (Number) Column width
  • minWidth, maxWidth (Number) Default to 81 ~ 300
  • styleMap (Object, String) Column style (css)
  • formatter (Function) column formatter. Arguments: value, rowItem, columnItem, cellNode
  • sortable (Boolean) Column sortable when click column header name
  • resizable (Boolean) Column width resizable
  • searchable (Boolean) Specifies whether the column is searchable
  • markdown (Boolean) Specifies whether the column needs to use markdown conversion
  • detailed (Boolean) Specifies whether the column needs to display the layout in detail (horizontal)
  • more properties columnProps
// playwright.config.js
module.exports = {
    reporter: [
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './test-results/report.html',

            // custom columns
            columns: (defaultColumns) => {

                // insert custom column(s) before a default column
                const index = defaultColumns.findIndex((column) => column.id === 'duration');
                defaultColumns.splice(index, 0, {
                    // define the column in reporter
                    id: 'owner',
                    name: 'Owner',
                    align: 'center',
                    searchable: true,
                    styleMap: {
                        'font-weight': 'normal'
                    }
                }, {
                    // another column for JIRA link
                    id: 'jira',
                    name: 'JIRA Key',
                    width: 100,
                    searchable: true,
                    styleMap: 'font-weight:normal;',
                    formatter: (v, rowItem, columnItem) => {
                        const key = rowItem[columnItem.id];
                        return `<a href="https://your-jira-url/${key}" target="_blank">${v}</a>`;
                    }
                });

            }
        }]
    ]
};

Column Formatter

Note: The formatter function will be serialized into string via JSON, so closures, contexts, etc. will not work!

// playwright.config.js
module.exports = {
     reporter: [
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './test-results/report.html',
            columns: (defaultColumns) => {

                // duration formatter
                const durationColumn = defaultColumns.find((column) => column.id === 'duration');
                durationColumn.formatter = function(value, rowItem, columnItem) {
                    if (typeof value === 'number' && value) {
                        return `<i>${value.toLocaleString()} ms</i>`;
                    }
                    return value;
                };

                // title formatter
                // Note: The title shows the tree style, it is a complicated HTML structure
                // it is recommended to format title base on previous.
                const titleColumn = defaultColumns.find((column) => column.id === 'title');
                titleColumn.formatter = function(value, rowItem, columnItem, cellNode) {
                    const perviousFormatter = this.getFormatter('tree');
                    const v = perviousFormatter(value, rowItem, columnItem, cellNode);
                    if (rowItem.type === 'step') {
                        return `${v}<div style="position:absolute;top:0;right:5px;">โœ…</div>`;
                    }
                    return v;
                };

            }
        }]
    ]
};

Searchable Fields

// playwright.config.js
module.exports = {
     reporter: [
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './test-results/report.html',
            columns: (defaultColumns) => {
                const locationColumn = defaultColumns.find((column) => column.id === 'location');
                locationColumn.searchable = true;
            }
        }]
    ]
};

Custom Fields in Comments

The code comments are good enough to provide extra information without breaking existing code, and no dependencies, clean, easy to read, etc.

  • First, enable option customFieldsInComments to true
// playwright.config.js
module.exports = {
    reporter: [
        ['monocart-reporter', {  
            // enable/disable custom fields in comments. Defaults to true.
            customFieldsInComments: true
        }]
    ]
};
  • Then, add comments for the tests

Note: Each comment item must start with @ which is similar to JSDoc.

For example, adding owner and jira to the cases, steps, and suites. or updating the value if the field exists like title

/**
 * for file (comment file in the first line)
 * @owner FO
 */
const { test, expect } = require('@playwright/test');

/**
 * for case
 * @owner Kevin
 * @jira MCR-16888
 */
test('case title', () => { 

});

/**
 * @description multiple lines text description
multiple lines text description
multiple lines text description
 * @jira MCR-16888
*/
test('case description', () => {

});

/**
 * for describe suite
 * @owner Mark
 * @jira MCR-16900
 */
test.describe('suite title', () => {

    test('case title', ({ browserName }, testInfo) => {

        /**
         * rewrite assert step title "expect.toBe" to
         * @title my custom assert step title
         * @annotations important
         */
        expect(testInfo).toBe(test.info());

        // @owner Steve
        await test.step('step title', () => {
        
        });

    });

});

/**
 * rewrite "beforeAll hook" title to
 * @title do something before all
 */
test.beforeAll(() => { 

});

/**
 * rewrite "beforeEach hook" title to
 * @title do something before each
 */
test.beforeEach(() => { 
    
});

Create Diagrams and Visualizations with Mermaid

  • Enable Mermaid
// playwright.config.js
module.exports = {
     reporter: [
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './test-results/report.html',
            mermaid: {
                // mermaid script url, using mermaid CDN: https://www.jsdelivr.com/package/npm/mermaid
                scriptSrc: 'https://cdn.jsdelivr.net/npm/mermaid@latest/dist/mermaid.min.js',
                // mermaid config: https://mermaid.js.org/config/schema-docs/config.html
                config: {
                    startOnLoad: false
                }
            }
        }]
    ]
};
  • Write Mermaid code in markdown:
/**
 * @description Sequence diagram for Monocart Reporter
```mermaid
flowchart LR

A[Hard] -->|Text| B(Round)
B --> C{Decision}
C -->|One| D[Result 1]
C -->|Two| E[Result 2]
```
*/
test('case description', () => {

});

see Mermaid doc

Custom Data Visitor

The visitor function will be executed for each row item (suite, case and step). Arguments:

  • data data item (suite/case/step) for reporter, you can rewrite some of its properties or add more
  • metadata original data object from Playwright test, could be one of Suite, TestCase or TestStep

Collect Data from the Title

For example, we want to parse out the jira key from the title:

test('[MCR-123] collect data from the title', () => {

});

You can simply use regular expressions to parse and get jira key:

// playwright.config.js
module.exports = {
    reporter: [
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './test-results/report.html',
            visitor: (data, metadata) => {
                // [MCR-123] collect data from the title
                const matchResult = metadata.title.match(/\[(.+)\]/);
                if (matchResult && matchResult[1]) {
                    data.jira = matchResult[1];
                }
            }
        }]
    ]
};

multiple matches example: collect-data

Collect Data from the Annotations

It should be easier than getting from title. see custom annotations via test.info().annotations

test('collect data from the annotations', () => {
    test.info().annotations.push({
        type: "jira",
        description: "MCR-123"
    })
});
// playwright.config.js
module.exports = {
    reporter: [
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './test-results/report.html',
            visitor: (data, metadata) => {
                // collect data from the annotations
                if (metadata.annotations) {
                    const jiraItem = metadata.annotations.find((item) => item.type === 'jira');
                    if (jiraItem && jiraItem.description) {
                        data.jira = jiraItem.description;
                    }
                }
            }
        }]
    ]
};

Remove Secrets and Sensitive Data

The report may hosted outside of the organizationโ€™s internal boundaries, security becomes a big issue. Any secrets or sensitive data, such as usernames, passwords, tokens and API keys, should be handled with extreme care. The following example is removing the password and token from the report data with the string replacement in visitor function.

// playwright.config.js
module.exports = {
    reporter: [
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './test-results/report.html',
            visitor: (data, metadata) => {
                const mySecrets = [process.env.PASSWORD, process.env.TOKEN];
                mySecrets.forEach((secret) => {
                    // remove from title
                    data.title = data.title.replace(secret, '***');
                    // remove from logs
                    if (data.logs) {
                        data.logs = data.logs.map((item) => item.replace(secret, '***'));
                    }
                    // remove from errors
                    if (data.errors) {
                        data.errors = data.errors.map((item) => item.replace(secret, '***'));
                    }
                });
            }
        }]
    ]
};

see example: remove-secrets

Style Tags

  • Add tag to test/describe title ( starts with @ )
test('test title @smoke @critical', () => { });
test.describe('describe title @smoke @critical', () => { });
  • Custom tag style
// playwright.config.js
module.exports = {
    reporter: [
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './test-results/report.html',
            tags: {
                smoke: {
                    style: {
                        background: '#6F9913'
                    },
                    description: 'This is Smoke Test'
                },
                critical: {
                    background: '#c00'
                }
            }
        }]
    ]
};

Metadata

All metadata will be listed in the report in a key/value format.

  • Global level metadata
// playwright.config.js
module.exports = {
    globalSetup: require.resolve('./common/global-setup.js'),
    metadata: {
        product: 'Monocart',
        env: 'STG',
        type: 'Regression',
        executor: 'Mono',
        
        // test home page object model
        url: 'https://www.npmjs.org/package/monocart-reporter'
    },
     reporter: [
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './test-results/report.html'
        }]
    ]
};
  • Project level metadata
// playwright.config.js
module.exports = {
    projects: [
        {
            name: 'Desktop Chromium',
            use: {
                browserName: 'chromium'
            },
            metadata: {
                projectData: 'project level metadata',
                owner: 'PO',
                link: 'https://github.com/cenfun/monocart-reporter'
            }
        }
    ]
}
// ./common/global-setup.js
import { chromium } from '@playwright/test';
export default async (config) => {
    const metadata = config.metadata;
    // collect data and save to global metadata
    const browser = await chromium.launch();
    const chromiumVersion = await browser.version();
    metadata.chromiumVersion = chromiumVersion;
};
  • Collect metadata in a test

    Playwright Test runs tests in parallel with isolate test data by default, so we need to utilize global state management to collect metadata in a test.

    const { test } = require('@playwright/test');
    const { useState } = require('monocart-reporter');
    
    const state = useState({
        // port: 8130
    });
    
    test('test metadata url', async ({ page }) => {
        const url = await page.evaluate(() => window.location.href);
        await state.set('url', url);
    });

Trend Chart

Note: The trend chart requires historical data generally stored in the server database. There is a serverless solution which is connecting and collecting historical trend data from previous report data before test every time.

  • If a report is generated in the same place every time, you can simply connect the data with the report JSON path (the data is not 100% safe if there is any runtime error, the previous output dir will be empty by Playwright but the reporter processing not finish)
// playwright.config.js
module.exports = {
    reporter: [
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './test-results/report.html',
            // connect previous report data for trend chart
            trend: './test-results/report.json'
        }]
    ]
};
  • Recommended: resolve the data by yourself (could be requested from the server), required data fields:
    • date (Number) the previous test date in milliseconds
    • duration (Number) the previous test duration
    • summary (Object) the previous test summary
    • trends (Array) historical data list, but except the previous self
// playwright.config.js
module.exports = {
    reporter: [
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './test-results/report.html',
            // connect previous report data for trend chart
            trend: async () => {
                const previousReportData = await readDataFromSomeWhere("path-to/report.json");
                // do some data filtering to previous trends
                previousReportData.trends = previousReportData.trends.filter((item) => {
                    // remove data a week ago
                    return item.date > (Date.now() - 7 * 24 * 60 * 60 * 1000)
                });
                return previousReportData;
            }
        }]
    ]
};

Code Coverage Report

The reporter integrates monocart-coverage-reports for coverage reports, there are two APIs:

  • addCoverageReport(data, testInfo) Add coverage to global coverage report from a test. see Global Coverage Report
    • data There are two supported data inputs: Istanbul (Object) or V8 (Array)
    • testInfo see TestInfo
  • attachCoverageReport(data, testInfo, options) Attach a coverage report to a test. Arguments:
    • data same as above
    • testInfo same as above
    • options (Object) see Coverage Options

Global Coverage Report

The global coverage report will merge all coverages into one global report after all the tests are finished.

// playwright.config.js
module.exports = {
    reporter: [
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './test-results/report.html',
            // global coverage report options
            coverage: {
                entryFilter: (entry) => true,
                sourceFilter: (sourcePath) => sourcePath.search(/src\/.+/) !== -1,
            }
        }]
    ]
};
// fixtures.js for v8 coverage
import { test as testBase, expect } from '@playwright/test';
import { addCoverageReport } from 'monocart-reporter';

const test = testBase.extend({
    autoTestFixture: [async ({ page }, use) => {

        // NOTE: it depends on your project name
        const isChromium = test.info().project.name === 'Desktop Chromium';

        // console.log('autoTestFixture setup...');
        // coverage API is chromium only
        if (isChromium) {
            await Promise.all([
                page.coverage.startJSCoverage({
                    resetOnNavigation: false
                }),
                page.coverage.startCSSCoverage({
                    resetOnNavigation: false
                })
            ]);
        }

        await use('autoTestFixture');

        // console.log('autoTestFixture teardown...');
        if (isChromium) {
            const [jsCoverage, cssCoverage] = await Promise.all([
                page.coverage.stopJSCoverage(),
                page.coverage.stopCSSCoverage()
            ]);
            const coverageList = [... jsCoverage, ... cssCoverage];
            // console.log(coverageList.map((item) => item.url));
            await addCoverageReport(coverageList, test.info());
        }

    }, {
        scope: 'test',
        auto: true
    }]
});
export { test, expect };
  • Adding server side coverage on global teardown

For example, a Node.js web server start at the beginning of the test with the env NODE_V8_COVERAGE=dir, the V8 coverage data will be saved to dir with calling API v8.takeCoverage() manually or terminating server gracefully. On global teardown, reading all from dir and adding them to global coverage report. For Node.js, the V8 coverage data requires appending source manually.

// global-teardown.js
import fs from 'fs';
import path from 'path';
import { fileURLToPath } from 'url';
import { addCoverageReport } from 'monocart-reporter';

export default async (config) => {

    const dir = "your-v8-coverage-data-dir";

    const files = fs.readdirSync(dir);
    for (const filename of files) {
        const content = fs.readFileSync(path.resolve(dir, filename)).toString('utf-8');
        const json = JSON.parse(content);
        let coverageList = json.result;
        coverageList = coverageList.filter((entry) => entry.url && entry.url.startsWith('file:'));

        // appending source
        coverageList.forEach((entry) => {
            entry.source = fs.readFileSync(fileURLToPath(entry.url)).toString('utf8');
        });

        // there is no test info on teardown, just mock one with required config
        const mockTestInfo = {
            config
        };
        await addCoverageReport(coverageList, mockTestInfo);
    }
}

see Collecting V8 Coverage Data from Node.js

Coverage Options

Coverage Examples

Attach Lighthouse Audit Report

Attach an audit report with API attachAuditReport(runnerResult, testInfo). Arguments:

const { test, chromium } = require('@playwright/test');
const { attachAuditReport } = require('monocart-reporter');
const lighthouse = require('lighthouse/core/index.cjs');
test('attach lighthouse audit report', async () => {
    const port = 9222;
    const browser = await chromium.launch({
        args: [`--remote-debugging-port=${port}`]
    });
    const options = {
        // logLevel: 'info',
        // onlyCategories: ['performance', 'best-practices', 'seo'],
        output: 'html',
        port
    };
    const url = 'https://github.com/cenfun/monocart-reporter';
    const runnerResult = await lighthouse(url, options);
    await browser.close();
    await attachAuditReport(runnerResult, test.info());
});

Attach Network Report

Attach a network report with API attachNetworkReport(har, testInfo). Arguments:

Generate HAR with recordHar option in browser.newContext() (see example: report-network.spec.js preview report)

const fs = require('fs');
const path = require('path');
const { test } = require('@playwright/test');
const { attachNetworkReport } = require('monocart-reporter');
let context;
test.describe('attach network report 1', () => {

    const harPath = path.resolve('.temp/network-report1.har');
    if (fs.existsSync(harPath)) {
        // remove previous
        fs.rmSync(harPath);
    }

    test('first, open page', async ({ browser }) => {
        context = await browser.newContext({
            recordHar: {
                path: harPath
            }
        });
        const page = await context.newPage();
        await page.goto('https://github.com/cenfun/monocart-reporter');
    });

    test('next, run test cases', async () => {
        
    });

    test('finally, attach HAR', async () => {
        // Close context to ensure HAR is saved to disk.
        await context.close();
        await attachNetworkReport(harPath, test.info());
    });
});

Generate HAR with playwright-har

import { test } from '@playwright/test';
import { attachNetworkReport } from 'monocart-reporter';
import { PlaywrightHar } from 'playwright-har';

const harPath = path.resolve('.temp/network-report2.har');
if (fs.existsSync(harPath)) {
    // remove previous
    fs.rmSync(harPath);
}

test('first, open page', async ({ browser }) => {
    const context = await browser.newContext();
    const page = await context.newPage();
    playwrightHar = new PlaywrightHar(page);
    await playwrightHar.start();
    await page.goto('https://github.com/cenfun/monocart-reporter');
});

test('next, run test cases', async () => {
    
});

test('finally, attach HAR', async () => {
    await playwrightHar.stop(harPath);
    await attachNetworkReport(harPath, test.info());
});

Preview Network HTML Report

Global State Management

When tests are executed in isolation mode, the reporter and each test may run in a different process, they cannot share data with each other. we can start a local WebSocket server to serve the global data, and read/write the global data with useState API from a test.

Setup Global State

module.exports = {
    reporter: [
        ['list'],
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './test-results/report.html',
            state: {
                data: {
                    count: 0
                },
                server: {
                    // port: 8130
                },
                onClose: (data, config) => {
                    // save state data to global metadata
                    Object.assign(config.metadata, data);
                }
            }
        }]
    ]
};

Get, Set, and Remove Global Data

const { test } = require('@playwright/test');
const { useState } = require('monocart-reporter');
test('state test', async ({ browserName }) => {
    const state = useState({
        // port: 8130
    });

    const count = await state.get('count');
    console.log('count', count);

    await state.set('count', count + 1);

    await state.set({
        browser: browserName,
        someKey: 'some value'
    });

    const [browser, someKey] = await state.get('browser', 'someKey');
    console.log(browser, someKey);

    await state.remove('someKey');

    const all = await state.get();
    console.log(all);
});

Send and Receive Messages between Processes

  • send message and receive response from a test (child process)
const { test } = require('@playwright/test');
const { useState } = require('monocart-reporter');
test('state test send message', async () => {
    const state = useState({
        // port: 8130
    });
    const response = await state.send({
        testId: test.info().testId,
        data: 'my test data'
    });
    console.log('receive response on client', response);
});
  • receive message and send response from global state (main process)
module.exports = {
    reporter: [
        ['list'],
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './test-results/report.html',
            state: {
                onReceive: function(message) {
                    const test = this.getTest(message.testId);
                    if (test) {
                        // current test
                    }
                    console.log('receive message on server', message);
                    return {
                        data: 'my response data'
                    };
                }
            }
        }]
    ]
};

see example: Allow specified test cases to run in sequence mode with lock/unlock state

Merge Shard Reports

There will be multiple reports to be generated if Playwright test executes in sharding mode. for example:

npx playwright test --shard=1/3
npx playwright test --shard=2/3
npx playwright test --shard=3/3

There are 3 reports will be generated. Using merge(reportDataList, options) API to merge all reports into one.

Note: One more suite level "shard" will be added, its title will be the machine hostname, and the summary will be restated. You may need to transfer the attachments by yourself and update the path of the attachments with attachmentPath API.

import { merge } from 'monocart-reporter';

const reportDataList = [
    // json file path
    'path-to/shard1/index.json',
    'path-to/shard2/index.json',
    // or JSON data
    JSON.parse(fs.readFileSync(path.resolve('path-to/shard3/index.json')))
];

await merge(reportDataList, {
    name: 'My Merged Report',
    outputFile: 'merged-report/index.html',
    attachmentPath: (currentPath, extras) => {
       // return `https://cenfun.github.io/monocart-reporter/${currentPath}`;
    },
    onEnd: async (reportData, helper) => {
        // send email or third party integration
    }
});

see example merge.js

onEnd Hook

The onEnd function will be executed after report generated. Arguments:

  • reportData all report data, properties:
    • name (String) report name
    • date (Number) start date in milliseconds
    • dateH (String) human-readable date with Date.toLocaleString()
    • duration (Number) test duration in milliseconds
    • durationH (String) human-readable duration
    • summary (Object) test summary, includes tests, suites, steps, etc.
    • rows and columns (Array) all rows and columns data, both are tree structure, see detail
    • tags (Object) tag collection
    • metadata (Object) metadata collection
    • system (Object) system information
    • trends (Array) historical trend data
    • caseTypes and suiteTypes (Array)
    • cwd, outputDir and outputFile (String)
    • htmlPath, jsonPath and summaryTable (String)
    • ...
  • helper APIs:
    • helper.find(callback) Find item like array find function.
    • helper.filter(callback) Filter list like array filter function.
    • helper.forEach(callback) Iterate all rows of data (suites/cases/steps), return break will break the iteration.
    • helper.sendEmail(emailOptions)
// playwright.config.js
module.exports = {
    reporter: [
        ['monocart-reporter', {  
            name: "My Test Report",
            outputFile: './test-results/report.html',
            // async hook after report data generated
            onEnd: async (reportData, helper) => {
                // console.log(reportData.summary);

                // find a test by title
                const myCase = helper.find((item, parent) => item.type === 'case' && item.title.includes('inline tag'));
                console.log(myCase && myCase.title);

                // find a suite by title
                const mySuite = helper.find((item, parent) => item.type === 'suite' && item.title.includes('new syntax'));
                console.log(mySuite && mySuite.title);

                // filter failed cases
                const failedCases = helper.filter((item, parent) => item.type === 'case' && item.caseType === 'failed');
                console.log(failedCases.map((it) => it.title));

                // Iterate all items
                helper.forEach((item, parent) => {
                    // do something
                });


            }
        }]
    ]
};

Send Email

Testrail Integration

Jira + Zephyr Scale Integration

Jira + Xray Integration

Slack Integration

Discord Integration

Teams Integration

BrowserStack Integration

Dingtalk/Weixin/Feishu Integration

Contributing

# Node.js 20+
npm install starfall-cli -g
npm install

npm run build
npm run test

npm run dev

Dependencies

monocart-reporter's People

Contributors

alex-vungle avatar anishkny avatar cenfun avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

monocart-reporter's Issues

Lots of "route.continue" generated in the Monocart report

** Test Environment: **

  • Playwright version: 1.33.0
  • monocart-reporter version: 1.6.16

How to reproduce:

In my test framework, we have a route handler that is used to block 3rd-party plugins in order to accelerate page loading speed:

  page.route('**', route => {
    const url = route.request().url();
    // console.log('request url: ', request.url());
 const blackList = [
    "sentry.io",
    "segment.io",
    "segment.com",
    "fullstory.com",
    "hs-analytics.net",
    "hs-banner.com",
    "zdassets.com",
    "zendesk.com"
  ];

  if (blackList.some((kw) => url.includes(kw))) {
    return route.abort();
  }
    return route.continue();
  });

Current issue is the monocart report has many route.continue() generated in the steps, pls refer to below screenshot, it leads the report hard to read.

image

Is it possible to refer to playwright HTML report solution to group those continuous route.continue() together?

image

[Improve] Consider adding types for the package

Perhaps the title of this issue is not the best, I'm not the most experienced front-end developer. Let me explain what I'm seeing:

  1. I have a TypeScript project with eslint. The eslint configuration has lots of rules enabled.
  2. I want to use the attachAuditReport function so I do:
// near the top of my file, near the imports I have
const { attachAuditReport } = require("monocart-reporter");

// and in some tests I'll have
await attachAuditReport(<some params>);

When I do the above my eslint configuration warns me about the following rule:

@typescript-eslint/no-require-imports: Disallow invocation of require(). Prefer the newer ES6-style imports over require().

For now what I've done was change the usage of require to import such as import { attachAuditReport } from "monocart-reporter"; and then I created a monocart-reporter.d.ts file which only contains declare module "monocart-reporter";.

This does get rid of the warning but it's only a workaround.

Suggestion: I believe you could consider updating your package to support ESM format? Or is this the wrong suggestion and perhaps there is another way to solve this issue I'm getting?

If it helps, I found this link Migrating an NPM package to use ES Modules.

[Improve] Code coverage summary columns

Hi,

When using V8+monocart-reporter for collecting code coverage, the report summary table looks like so:

image

Note that there is no column to show lines total/covered/uncovered. What it's shown is bytes total/covered/uncovered.

Suggestion: I don't think most people look at bytes as a measure of code coverage. Could you consider replacing the bytes columns with lines columns? If you feel there's value in having those columns in the summary page then I think that's fine but the lines columns should be there as well.

Note that you already track that information. If I open the details for a covered file I get:

image

Could we get these 3 columns, Total lines, Lines covered and Lines uncovered showing on the code coverage summary ? What do you think?

[QUESTION] Is it possible to group everything by describe blocks rather than by file?

We run a lot of our tests in parallel and we really miss a feature of being able to aggregate tests by describe block. I had a quick look at the docs and can't find anything related to that. See below screenshot

image

Here I want the grouping to be by ABC articles and in general I want all common describe blocks on the same level grouped. Is this something you have planned in the works or are you aware of other reporters giving this functionality?

[BUG] Attachments not found

Hi,

It seems attachments are not being fetched properly depending on how you setup your playwright outputDir and monocart-reporter outputFile.

If I setup my playwright.config.ts as such:

const _testsDir = "./tests";
const _testsOutputBaseDir = path.join(_testsDir, "test-results");
const _testReportersOutputBaseDir = path.join(_testsOutputBaseDir, "reporters");
const _htmlReporterDir = path.join(_testReportersOutputBaseDir, "html");
const _monoCartReporterDir = path.join(_testReportersOutputBaseDir, "monocart");
const _monoCartReporterOutputFile = path.join(_monoCartReporterDir, "index.html");

export default defineConfig({
    ...
    outputDir: `${_testsOutputBaseDir}`,
    reporter: [
        [
            "monocart-reporter",
            {
                name: "Playwright Monocart Report",
                outputFile: _monoCartReporterOutputFile,
                coverage: {
                    sourceFilter: (sourceName: any) => sourceName.search(/src\//) !== -1,
                },
            },
        ],
    ],
    ...
});

And have an automatic fixture as such:

const { addCoverageReport, attachCoverageReport } = require("monocart-reporter");

export { expect } from "@playwright/test";

interface CodeCoverageFixture {
    autoTestFixture: void;
}

export const test = baseTest.extend<CodeCoverageFixture>({
    autoTestFixture: [
        async ({ page }, use): Promise<void> => {
            // coverage API is chromium only
            if (test.info().project.name === "chromium") {
                await Promise.all([
                    page.coverage.startJSCoverage(),
                    page.coverage.startCSSCoverage(),
                ]);
            }

            await use();

            if (test.info().project.name === "chromium") {
                const [jsCoverage, cssCoverage] = await Promise.all([
                    page.coverage.stopJSCoverage(),
                    page.coverage.stopCSSCoverage(),
                ]);
                const coverageList = [...jsCoverage, ...cssCoverage];
                await addCoverageReport(coverageList, test.info());
                const report = await attachCoverageReport(coverageList, test.info());
            }
        },
        {
            auto: true,
        },
    ],
});

Then I run a test that produces a snapshot and I have this output:

[MCR] coverage: tests/test-results/coverage-8aefe567157b61ce7446/index.html Coverage Report - basic test
[MCR] coverage: tests/test-results/reporters/monocart/coverage/index.html Coverage Report - Playwright Monocart Report (global)
[MCR] json report: tests/test-results/reporters/monocart/index.json
[MCR] html report: tests/test-results/reporters/monocart/index.html
[MCR] view report: npx monocart show-report tests/test-results/reporters/monocart/index.html

Notice that both the HTML report and the global coverage report share the same base dir tests/test-results/reporters/monocart/ and as such I can navigate from the index.html to the global code coverage report:

  • I can go from http://localhost:8090/index.html to http://localhost:8090/coverage/index.html

However the screenshot attachments and the code coverage attachments display as not found. When I try to access the test code coverage it tries to go to http://localhost:8090/coverage-8aefe567157b61ce7446/index.html and because the path coverage-8aefe567157b61ce7446/index.html does not exist at tests/test-results/reporters/monocart it can't find the file. That path is at tests/test-results/.

The same happens for the screenshots. If I try to go to a screenshot it tries to go to http://localhost:8090/find-institution-example-basic-test-chromium/basic-test-1-expected.png and it cant find the path /find-institution-example-basic-test-chromium/basic-test-1-expected.png at tests/test-results/reporters/monocart so it fails. The path is at tests/test-results/.

It seems to me that the attachments path uses playwrights outputDirto store data instead of the directory relative to where the monocart html report is at which is indicated byoutputFile` and then there's a mismatch in where files should be ? Do you think this is a bug or am I misusing the configuration?

Let me know if you need more info.

[Question] No test results for testrail even when the report is being generated with reportData

Hey there! I'm completely new with this reporter and gotta say that it is amazing, nonetheless when trying to integrate with Testrail I'm kinda lost since every time I run my tests I'm getting the error message saying that no test results for testrail are available, but my report-file is still being generated without any issues (both .html & .json).

Here's a look at my playwright.config.ts:

  reporter: [
    ['list'],
    ['monocart-reporter', {
        name: "My Test Report - Playwright Automated",
        outputFile: "./test-results/monocart_report.html",
        onEnd: async (reportData, capability) => {
          console.log('onEnd hook start');
          const testrail = require('./utils/testrail.js');
          await testrail(reportData, capability);
        }
    }]
  ],

And what I currently have in my testrail.js:

require('dotenv').config()
const Testrail = require('testrail-api');
const EC = require('eight-colors');
module.exports = async (reportData, capability) => {

    const options = {
        host: 'https://myInstance.testrail.io',
        user: process.env.TESTRAIL_USERNAME,
        password: process.env.TESTRAIL_PASSWORD
    };

    if (!options.password) {
        EC.logRed('[testrail] require a password');
        return;
    }

    const client = createClient(options);

    EC.logCyan('[testrail] collect test results ...');
    const results = [];

    // add testrail case id to comments
    /**
     * @testrail 2126
     */
    // test('Test case', async () => { });

    capability.forEach((item) => {
        if (item.type === 'case' && item.testrail) {
            // 1 Passed
            // 5 Failed
            const status_id = item.ok ? 1 : 5;
            results.push({
                case_id: item.testrail,
                status_id: status_id,
                comment: ''
            });
        }
    });

    if (!results.length) {
        EC.logRed('[testrail] no test results for testrail');
        return;
    }
    console.log(`[testrail] got test results: ${results.length}`);


I've tried installing the testrail-api dependency directly, logging the output (still getting reportData) and even creating a Test Suite with the same name into Testrail prior to executing it, but got no luck, is there something else I'm missing? Do I have to actually "mark" each and every test case that I want to be collected?

Here's an screenshot of my results if it serves for some purpose

Screen Shot 2023-04-04 at 14 06 51

Sorry for the silly question in advance and once again, amazing job with this reporter !! ๐Ÿš€

[BUG] test.step failures dont fail the test , appears as passed on the html report

please execute the following test and then view the report ... the test case passes on the report instead of failing.. i think when you use test.step and if the failure occurs inside the test.step it does not fail the test

const { test, expect } = require("@playwright/test");

const config = require("../../config/environment_manager");

//run command

//ENV=dev npx playwright test tests/training/test.spec.js

var number_one = 3;

var number_two = 4;

var total = "";

test("most basic test: add two numbers", async () => {

test.step("add two numbers", async () => {

    //adding two numbers

    total= number_one + number_two;

    console.log( "the total is: ",total);

    });



test.step("verify total", async () => {

//Assertion

expect(total).toBe(6);



});

});

Attachments are not displayed if the following conditions are met: the report is launched by double-click + the length of the path to the attachments is greater than 259 characters.

configuration:
"OS": Windows 10,
"@playwright/test": "1.35.0",
"monocart-reporter": "1.6.36"

How to reproduce:

  1. Extrat to the root folder (4example: d:) attached ag-test.zip file. Root folder because of in this case all attachments will have file-path length that reproduce the issue.
    image

  2. To install all packages go to the e2e folder and exec:
    --- npm run start

  3. Exec:
    --- To get report where all atachments are not shown
    --- npm run nth
    --- or exec 01-show-nothing-length-32xx.spec.ts

  4. Open ./test-results/index.html:
    --- by double click.
    --- Do not use CLI. In CLI everything is fine.
    --- Diff file-path length is greter than 259 = 260
    --- Actual file-path length is greter than 259 = 262
    --- Expected file-path length is greter than 259 = 264
    D:/ag-test/vsts-test/pw-dxxx-txxxxxxx-tests-dev/e2e/test-results/multiple-html/tests-d-x-01-show-nothing-length-32xx-Dxxx-Txxxxxxx-BASE-1-Dxxx-Cxxxxxx-EU-page-should-be-opened--Chrome-Stable/Dxxx-Txxxxxxx-BASE-1-Dxxx-Cxxxxxx-EU-page-should-be-opened-1-diff.png

image

  1. Repeat steps 3-4 for npm run diff:
    --- To get report where Diff is shown + Actual and Expected atachments are not shown.
    --- Diff file-path length = 259
    --- Actual file-path length is greter than 259 = 261
    --- Expected file-path length is greter than 259 = 263
    image

  2. When I try to copy ag-test folder to another location by using Total Commander:
    image

Use attached:

Thanks.

Report file name cannot be specified

Given for the examples on both playwright.dev and monocart-reporter github page input file name cannot be interfered, it creates index.html file for result like playwright's own html reporter

outputFile: 'playwright-report/grid/indexTest1.html'

Worker 0 sometimes shows in timeline graph

This issue is not always reproducible, I specify 10 workers for parallel execution, but sometimes, I saw a worker process named Worker 0 shows in the execution timeline graph.

npx playwright test --workers 10 --trace on

image

image

[BUG] Incorrect code coverage when converting from V8 to istanbul

Hi @cenfun,

Bug description

I mainly use the V8 code coverage when working in my dev machine and all works great. However, I have a requirement to upload code coverage to SonarQube and it requires an lcov file. When I enable the monocart-reporter option to get the lcov file as such:

coverage: {
    toIstanbul: true,
    lcov: true,
    ...
},

The code coverage that I get is incorrect. Please see screenshots below for more information. The code coverage screenshots were produced by running with:

  • babel + istanbul: this shows correct code coverage.
  • v8 + monocart reporter without the toIstanbul property set which defaults to false: this shows correct code coverage.
  • v8 + monocart reporter with the toIstanbul property set to true: this shows INCORRECT code coverage.
babel+istanbul

Note
The code coverage below is correct

app/core/ui-services/ui-helper-functions.ts
image

app/features/find-institution/institutions-list/institutions-list.component.ts
image

v8 + monocart reporter without the toIstanbul property set which defaults to false

Note
The code coverage below is correct

src/app/core/ui-services/ui-helper-functions.ts
image

src/app/features/find-institution/institutions-list/institutions-list.component.ts
image

v8 + monocart reporter with the toIstanbul property set to true

Warning
The code coverage below is INCORRECT. Some lines are showed as covered when they should not be.

app/core/ui-services/ui-helper-functions.ts
image

app/features/find-institution/institutions-list/institutions-list.component.ts
image

How to reproduce

  1. Checkout the temp branch at edumserrano/monocart-code-coverage-demo/tree/temp.
  2. run npm install to install all the required packages

To get the babel+istanbul code coverage:

  1. run npm run test-istanbul. The first time tests will fail because of missing snapshots, you can ignore this.
  2. run npm run nyc-report.
  3. run npm run open-istanbul-coverage.

To get the v8 + monocart reporter without the toIstanbul property set which defaults to false code coverage:

  1. run npm run test-monocart. The first time tests will fail because of missing snapshots, you can ignore this.
  2. run npm run open-monocart-coverage.

To get the v8 + monocart reporter with the toIstanbul property set to true code coverage:

  1. go to the playwright.confi.ts for the monocart setup and uncomment the lines:
// toIstanbul: true,
// lcov: true,
  1. run npm run test-monocart. The first time tests will fail because of missing snapshots, you can ignore this.
  2. run npm run open-monocart-coverage.

More notes

To clarify, the problem is that when using toIstanbul set to true the code coverage conversion marks some lines as covered when they should not be.

Let me know if you need more information.

[Question] Unexpected token error when exporting to TestRail

Hey cenfun!, me again (sorry ๐Ÿฅฒ )

I'm currently implementing the testrail integration across all my tests, but every time a test case fails OR is missing a step id idenfitication I'm getting error messages like this:

1 failed
    [chromium] โ€บ tests/assets/creating-assets.spec.ts:93:8 โ€บ Static Assets creation โ€บ Creating Static Asset - Other 
[MCR] generating report data ...
Unexpected token (5:13)
Unable to parse file for comments: /Users/clundstedt/Documents/zf-playwright/src/e2e/pages-objects/login.page.ts
Try use different options: https://babeljs.io/docs/babel-parser
Unexpected token (5:13)
Unable to parse file for comments: /Users/clundstedt/Documents/zf-playwright/src/e2e/pages-objects/menu-top-nav.component.ts
Try use different options: https://babeljs.io/docs/babel-parser
Unexpected token (5:13)
Unable to parse file for comments: /Users/clundstedt/Documents/zf-playwright/src/e2e/pages-objects/assets.page.ts
Try use different options: https://babeljs.io/docs/babel-parser
[MCR] generating test report ...

Although is not "super" critical as the scripts are running effectively, this can be misleading when integrating it on my CI/CD pipe, so I was wondering if this is something that could be related to some kind of config that I'm using.

Considerations:

  • I'm not strictly using babel at any point since it's not needed so far, should I absolutely need to install it and configure it (somehow?).

Thanks for your input and sorry for pestering you again

[Improve] Improvements to the search/filter functionality

Suggestion: if possible I think the following would enhance the search/filter functionality:

  1. move search box and toggles on the top right closer to the top left of the page

image

Rationale: all these options are filtering options, I might want to filter by Failed tests and then some keyword. It doesn't make much sense to me to have as part of the filter on the top left and another on the top right. You could even consider bring these options closer together and centre aligning them instead of left aligning.

  1. Make the search box larger. I think a larger input would look better. Currently, you can only enter a very small amount of text and quick enough you won't be able to see your entire query on the search box. The width on the html reporter feels more reasonable.

image

  1. Similar to what html reporter has on their search box, could we have a clear text button?

image

  1. When tags exist on the test suites, then add an extra filter option which would be a dropdown of the available tags and let the user select which tags he wants from that. This would be a shortcut to the user having to manually type all the tags he wants to filter by into the search box.

[Improve] Better `type` values on the code coverage report

When collecting code coverage with V8+monocart-reporter the code coverage report has a type column which will tell you if the coverage was done for a CSS file or a JS file.

Technically I guess those are the only 2 possible values of coverage that you can get but in the scenario where you can properly identify the source file then I think you could consider changing the type column to be the extension of the source file.

For instance, given this image:

image

You could say that the type of coverage could be html. Although in this example, technically speaking, the code coverage all comes from the main.js dist file, you do know it maps to a .html file.

The same with a TypeScript file:

image

In the above image, you could consider showing the type as ts instead of js.

Suggestion: my proposal would be to look at the mapped file extension and if possible use that as the value for the type column. If not possible fall-back to current behaviour.

What do you think?

[Improve] Metadata only works at global level

In Playwright you can configure metadata for all projects by adding:

export default defineConfig({
  ...
  metadata: {
    someKey: "someValue"
  },
  projects: [
  ...
  ]
  ...
})

Or you can add individual metadata per project by doing:

export default defineConfig({
  ...
  projects: [
    {
      name: "chromium",
      metadata: {
        someKey: "someValue"
      },
      ...
    }
  ...
})

If metadata is added per project monocart-reporter will not display anything.

Suggestion: if possible, add per project metadata by showing it under all tests that apply to that project. So when you open the test details you can see the metadata similar to how you can see the annotations.

I mocked something up using paint xD

image

What do you think? This would allow users to define different metadata per project and it would show up on the tests for that project.

[Issue] Comments are not collected

Have the next code in config (the same as in documentation):

visitor: (data, metadata, collect) => {

			const parserOptions = {
				// https://babeljs.io/docs/babel-parser
				sourceType: "module"
			}
			const comments = collect.comments(parserOptions);
			if (comments) {
				Object.assign(data, comments);
			}
		},
		trend: 'playwright-report/grid/index.json',
		columns: (defaultColumns) => {

			// insert custom column(s) before a default column
			const durationColumnIndex = defaultColumns.findIndex((column) => column.id === 'duration');
			defaultColumns.splice(durationColumnIndex, 0, 
			{
				// define the column in reporter
				id: 'owner',
				name: 'Owner',
				align: 'center',
				searchable: true,
				styleMap: {
					'font-weight': 'normal'
				}
            },{
				// JIRA column
				id: 'jira',
				name: 'JIRA Key',
				width: 100,
				searchable: true,
				styleMap: 'font-weight:normal;',
				formatter: (valueFormatted, rowItem, columnItem) => {
					const key = rowItem[columnItem.id];
					return `<a href="https://fotoware.atlassian.net/browse/${key}" target="_blank">${valueFormatted}</a>`;
				}
			});
	 	},

And the next code in the test itself

test.describe("test desc", () => {

    /** 
     * @jira xxx-11440
     * @owner I_N
     * @description bla-bal
     * */
    test("test"...

And as result I have nothing in the report. Is it something wrong in visitor or formatter? Thanks for help

Collapse continuous "route.continue()" message

Hi @cenfun ,

This is a follow-up ticket to #22

Just noticed that there're still many route.continue() message when I expand the test case to review the test steps. Is it possible to collapse those messages like the test case detail page?

Thanks,
Alex

image

[BUG] - Incorrect time in report

Hello.
I noticed that the time in the report is not correct.
image
97,367 ms = 1 min 37 sec. But report show 1m 53s

Proposal
Display test duration in min and second, not in ms. Or add key in config

[Improve] Default width for the title column

Consider changing how the default size for the title column is calculated. When I run my report I'm getting something like this:

image

With the default width for the title column, the text and tag is already cropped.

Suggestion: if possible, make it so that the default width for the title column takes as much space as possible in attempt to display all titles without cropping. So in my test case it should look like this by default:

image

If the user has a very long title in some test than it would start looking like this:

image

And if the user has such a long title that it can't fit on the screen because we also need space to display the other columns than we display as much of the title as possible.

image

[BUG] Wrong data in test run summary

I wrote and run(debug) 1 test.
In console I see test run summary
image
and I see 2 bugs:

  1. Suites โ”‚ 2 - Why run 2 suites when 1 run only 1 test from 1 test file?
  2. Duration โ”‚ 2m 156s - Strange duration. Total time was 156sec = 2 min 36 sec. Or display in sec, or correctly convert to min.

[Question] Is it possible to use custom test step names?

Hello, my name is Andres, I'm quite new to this reporter.

Is it possible to use custom test step names?
What I mean is if it is possible to change the names in the title column, e.g: "page.goto", "locator.fill" "waitforloadstate"
I would like to have other titles like "accessing the login page" "completing phone number"

Thanks in advance.
Cheers

[Bug] Incorrect monocart-reporter version shown on report details panel

Hi @cenfun,

I've just upgraded to 1.6.34 and the version of the monocart-reporter shown on the report details panel shows incorrectly as 1.6.33.

Here's how my package.json looks like:

...
"monocart-reporter": "^1.6.34",
...

Here's what is displayed on the report details panel:

  1. select the report details panel
    image

  2. scroll to the bottom to see the monocart-reporter version
    image

Note
I've tried previous versions of the monocart-reporter, versions before 1.6.34 and it seems the version shown on the reporter is always off by 1.

Disable the logging of the test results table in the console

Hello cenfun,

My name is Phil. And first of all, I wanted to say thanks for your monocart-reporter tool! It's super useful.
I have a question. Is it possible to disable the logging of the test results table in the console? And if not, are there plans to add this option in the future?

It's not really a problem, but rather a matter of personal convenience. I'm used to running tests without expanding the console too much. So I have to scroll up every time to see if the test passed or failed. The table takes up quite a lot of space, while being useless to me personally.

[Improve] Possibility colapse test step in test result panel

Hello.
I propose to make it possible to collapse the test steps in the test results pane, as it is done in the main panel.
image

The second improvement is to scroll the test result on the fallen step and, if possible, show a screenshot to the fallen step, and not (not only) in the header of the test resilt.

[Improve] Mouse over on test line styling

Currently when you mouse over on a test with tags the underline style will carry on under the tags as can be seen by:

image

Suggestion: if possible, consider only putting the underline style on the title of the text and not the tags

[BUG] - Incorrect Playwright version displayed in report

Hello.
It seems I found a problem with incorrect displaying Playwright version is report.
I have a branch with alpha version 'Playwright v1.33.0-alpha-apr-3-2023'. I work in this branch and run tests. After that I change branch with version '"@playwright/test": "^1.32.1",'
image
But report display alpha version from first branch
image

[Improve] Trace Viewer

This reporter is much better than the original one, but we have concerns about the current "Trace Viewer" feature state to use only this reporter.

Our current workflow supposes to run tests in GitLab and upload tests result to the GitLab pages. The original html reporter has his own trace.html that he also uploads to the GitLab pages, so, no CORS error, we store traces and this small html under the same domain.

And here are the question and possible improvements:

  1. Is it possible to add the same html in the monocart-reporter? So we can open traces without CORS problem
  2. We can use trace.html from the original playwright reporter, but we need the option to rewrite the generator of the links for the trace viewer block. So we can target it something like: https://tests.pages.gitlab.com/tests/trace.html?trace=${link to the trace}

[BUG] Regression: missing snapshot attachment

Hi @cenfun,

Something has happened and the attachments aren't displaying properly, at least for snapshots. From version 1.6.31 until latest what I get when a test fails is the following images:

image

image

When I downgrade to version 1.6.30 the attachments start showing:

image

I've re-used a demo playwright project I had to show the bug. You can go to edumserrano/playwright-bug-vrt-diff/tree/monocart-reporter and checkout the monocart-reporter branch.

Then from the root of the repo just do:

  • npm install
  • npm run test

This will give you the result above where no attachments are shown. Any ideas what might be going wrong? I don't understand how this was working and suddenly stopped.

[Improve] Improve trend to include test trend

Hey!
What about the idea of enhancing the trend feature by extending it to the test level? Currently, the trend feature is implemented based on launches.
I suggest storing additional data for each test and, for example, displaying the test history with results in the test information and constructing a trend graph.

Additionally, I propose grouping the test retries, for example, in a separate tab. Currently, they are simply displayed one after another.

[Question] trace viewer question

Hey!
I have question about trace viewer (https://github.com/cenfun/monocart-reporter/issues/26).

System info:
Playwright: 1.34.2
Monocart: 1.6.27
Config:
image

When I run my test and it fails, I attempt to open the report file and click 'View trace'. However, the trace does not open properly. The URL that opens is as follows: https://trace.playwright.dev/?trace=file%3A%2F%2FD%3A%2FCoding%2FHHub%2FHUB2%2FAutomationTesting%2FPlaywright%2Ftest-results%2FCatalogues-CatalogueRules-Delivery-options-rul-27844-se-41716-Validation-when-saving-contact-details-Chrome%2Ftrace.zip
image
And I see next page
image

In order to manually upload and open the file, I must use the following URL: https://trace.playwright.dev/?trace=blob%3Ahttps%3A%2F%2Ftrace.playwright.dev%2F48a36d89-4b61-480a-b0a6-b3ea104fa705&traceFileName=trace.zip.

How to fix that?

[Improve] Rendering multiline data collected from comments

I've followed the guide for https://github.com/cenfun/monocart-reporter#collect-data-from-the-comments and I added the following comment to a test:

/**
 * @description multiple lines text description 1
multiple lines text description 2
multiple lines text description 3
 */
test("basic test 3 @critical", async ({ baseURL, page }) => {
...
});

The report produced was as follows:

image

The description column in the main report page is fine I think. There's not much I think you can do there, it's only a peak into the value anyway.

However, when we open the test details we can see the description and in there there's enough space to respect multiline formats.

Suggestion: if possible, respect multiline format from columns when on the test details panel. This might mean that certain columns would start on their own line or whatever UX you feel better. For instance, the description column would start below the tags in the test details panel.

[Improve] Column resize on main report page

Currently there is no visual indication to the user that he can resize the columns in the main page of the report:

image

If the user tries to mouse over along the column line he will eventually find that the mouse cursor changes to indicate he can resize the column:

image

If possible consider two things:

  1. Add some visual indication, like a simple vertical bar is probably enough, for the columns on the top row so the user knows where he can click and hold to resize the column.
  2. Allow auto-resizing when using double click. Double clicking the delimiter for a column, the place we can click and hold to resize the column, should try to auto resize that column to fit the content. For instance, in the first image, if I double clicked the delimiter for the title column it should auto-resize the column to enough width to fit the text from all titles. If if can't fit the full text of the column being auto resized then it should expand to the max possible value.

[BUG] Incorrect test time

Hey,
I have launch my tests and saw the problem when incorrect test time was showed in the main grid.
image
But time on the diagram was corrected.
image

]Question] - How to correct configure custom tags?

Hello.
I create custom tags in my playwright.config file
image
After executing the tests, I opened the report file only to find that it contained tags that were completely different from the ones I had specified in the configuration. Instead of displaying only the test names that had the '@' symbol in their tags, the report included all code chunks that had the '@' symbol.
image

Failed in generating report data when test code throws exception

Monocart version: 1.6.20

I noticed if the test code throws unhandled exceptions directly:

......

test("foo", async () => {
    throw "Invalid error";
})

......

The monocart report will not able to generate report data with below error:

[MCR] generating report data ...
Error in reporter TypeError: err.split is not a function
    at /Users/ali/workspace/WebQA-Playwright/node_modules/monocart-reporter/lib/visitor.js:648:31
    at Array.map (<anonymous>)
    at Visitor.errorsToSnippets (/Users/ali/workspace/WebQA-Playwright/node_modules/monocart-reporter/lib/visitor.js:647:23)
    at /Users/ali/workspace/WebQA-Playwright/node_modules/monocart-reporter/lib/visitor.js:617:36
    at forList (/Users/ali/workspace/WebQA-Playwright/node_modules/monocart-reporter/lib/platform/share.js:116:32)
    at forList (/Users/ali/workspace/WebQA-Playwright/node_modules/monocart-reporter/lib/platform/share.js:120:35)
    at forList (/Users/ali/workspace/WebQA-Playwright/node_modules/monocart-reporter/lib/platform/share.js:120:35)
    at Object.forEach (/Users/ali/workspace/WebQA-Playwright/node_modules/monocart-reporter/lib/platform/share.js:126:9)
    at Visitor.duplicatedErrorsHandler (/Users/ali/workspace/WebQA-Playwright/node_modules/monocart-reporter/lib/visitor.js:597:14)
    at Visitor.start (/Users/ali/workspace/WebQA-Playwright/node_modules/monocart-reporter/lib/visitor.js:52:14)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at generateData (/Users/ali/workspace/WebQA-Playwright/node_modules/monocart-reporter/lib/generate-data.js:44:5)
    at Reporter.onEnd (/Users/ali/workspace/WebQA-Playwright/node_modules/monocart-reporter/lib/index.js:127:28)
    at Multiplexer.onExit (/Users/ali/workspace/WebQA-Playwright/node_modules/@playwright/test/lib/reporters/multiplexer.js:102:45)
    at Runner.runAllTests (/Users/ali/workspace/WebQA-Playwright/node_modules/@playwright/test/lib/runner/runner.js:75:5)
    at runTests (/Users/ali/workspace/WebQA-Playwright/node_modules/@playwright/test/lib/cli.js:135:138)
    at ji.<anonymous> (/Users/ali/workspace/WebQA-Playwright/node_modules/@playwright/test/lib/cli.js:48:7)

The Playwright HTML report is able to handle this scenario gracefully:

image

[BUG] Coverage report missing some files

Hi. I'm trying to set up Vue CT coverage based on your example here: https://github.com/cenfun/playwright-ct-vue
The problem is, almost half of the files are missing from coverage report (5 out of 11). Below is some info.

Report screenshots image
image
playwright-ct.config.ts
import { resolve } from 'path'
import { defineConfig, devices } from '@playwright/experimental-ct-vue'

/**
 * See https://playwright.dev/docs/test-configuration.
 */
export default defineConfig({
  testDir: './tests/',

  outputDir: './test-results',

  /* The base directory, relative to the config file, for snapshot files created with toMatchSnapshot and toHaveScreenshot. */
  snapshotDir: './playwright/__snapshots__',
  /* Maximum time one test can run for. */
  timeout: 10 * 1000,
  expect: {
    /**
     * Maximum time expect() should wait for the condition to be met.
     * For example in `await expect(locator).toHaveText();`
     */
    timeout: 2000,
  },
  /* Run tests in files in parallel */
  fullyParallel: true,
  /* Fail the build on CI if you accidentally left test.only in the source code. */
  forbidOnly: !!process.env.CI,
  /* Retry on CI only */
  retries: process.env.CI ? 2 : 0,
  /* Opt out of parallel tests on CI. */
  workers: process.env.CI ? 1 : undefined,
  /* Reporter to use. See https://playwright.dev/docs/test-reporters */
  reporter: [
    ['list'],
    [
      'monocart-reporter',
      {
        name: 'My Test Report',
        outputFile: './test-results/index.html',
        coverage: {
          // toIstanbul: true,
          // lcov: true,
          unpackSourceMap: true,
          excludeDistFile: true,
          sourceFilter: (sourceName: string) => {
            return sourceName.search(/src\//) !== -1
          },
          entryFilter: (entry: { url: string }) =>
            !entry.url.includes('tailwind'),
        },
      },
    ],
  ],
  /* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
  use: {
    /* Collect trace when retrying the failed test. See https://playwright.dev/docs/trace-viewer */
    trace: 'on-first-retry',

    /* Port to use for Playwright component endpoint. */
    ctPort: 3100,

    ctViteConfig: {
      // build: {
      //   sourcemap: true,
      // },

      resolve: {
        alias: {
          '@': resolve(__dirname, './src'),
        },
      },
    },
  },

  /* Configure projects for major browsers */
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    },
    /*
    {
      name: 'firefox',
      use: { ...devices['Desktop Firefox'] },
    },
    {
      name: 'webkit',
      use: { ...devices['Desktop Safari'] },
    },
    */
  ],
})
One of the missing files in report

VerticalNavDesktop.vue

<script setup lang="ts">
import type { VerticalNavItem } from './types'

defineProps<{
  navigationItems: VerticalNavItem[]
}>()
</script>

<template>
  <nav class="flex-col" aria-label="Sidebar">
    <ul role="list" class="-mx-2 space-y-1">
      <li v-for="item in navigationItems" :key="item.title">
        <RouterLink
          :to="item.route"
          :class="[
            item.current
              ? 'bg-gray-50 text-emerald-600'
              : 'text-gray-700 hover:text-emerald-600 hover:bg-gray-50',
            'group flex gap-x-3 rounded-md p-2 text-sm leading-6 font-medium',
          ]"
        >
          <component
            :is="item.icon"
            :class="[
              item.current
                ? 'text-emerald-600'
                : 'text-gray-400 group-hover:text-emerald-600',
              'h-6 w-6 shrink-0',
            ]"
            aria-hidden="true"
          />
          {{ item.title }}
        </RouterLink>
      </li>
    </ul>
  </nav>
</template>

VerticalNavDesktop.spec.ts

import { test, expect } from './fixture'
import VerticalNavDesktop from '@/VerticalNavDesktop.vue'

test('VerticalNavDesktop', async ({ mount }) => {
  const component = await mount(VerticalNavDesktop, {
    props: {
      navigationItems: [
        {
          title: 'Test',
          route: { name: 'test' },
          icon: 'icon',
          current: true,
        },
        {
          title: 'Test2',
          route: { name: 'test2' },
          icon: 'icon',
          current: false,
        },
      ],
    },
  })

  expect(await component.getByRole('listitem').count()).toBe(2)

  await expect(
    component.getByRole('listitem').first().getByRole('link')
  ).toHaveClass(/text-emerald/)

  await expect(
    component.getByRole('listitem').first().getByRole('link').locator('icon')
  ).toBeVisible()
})
fixture.ts
import { test as ctBase, expect } from '@playwright/experimental-ct-vue'
import { addCoverageReport } from 'monocart-reporter'

const test = ctBase.extend<{ autoTestFixture: string }>({
  autoTestFixture: [
    async ({ page }, use) => {
      // console.log('autoTestFixture setup...');
      // coverage API is chromium only
      if (test.info().project.name === 'chromium') {
        await Promise.all([
          page.coverage.startJSCoverage(),
          // page.coverage.startCSSCoverage(),
        ])
      }

      await use('autoTestFixture')

      // console.log('autoTestFixture teardown...');
      if (test.info().project.name === 'chromium') {
        const [jsCoverage] = await Promise.all([
          page.coverage.stopJSCoverage(),
          // page.coverage.stopCSSCoverage(),
        ])
        const coverageList = [...jsCoverage]
        await addCoverageReport(coverageList, test.info())
      }
    },
    {
      scope: 'test',
      auto: true,
    },
  ],
})

export { test, expect }

[BUG] Inaccurate code coverage

Hi @cenfun,

When instrumenting playwright tests for an angular app using istanbul via the babel-plugin-istanbul plugin the code coverage shows accurately whilst when using V8 + monocart-reporter some lines that should be covered show as uncovered even though there's an indication those lines have been executed X number of times.

The images below show a part of the code coverage for a file which show the problem.

As per our conversation on Playwright discord channel I've created a repo which allows you to easily reproduce the problem. Please see edumserrano/monocart-code-coverage-demo.

  • Istanbul code coverage report:

istanbul

  • Monocart code coverage report:

monocart

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.