github / lightcrawler Goto Github PK

View Code? Open in Web Editor NEW

1.5K 63.0 141.0 106 KB

Crawl a website and run it through Google lighthouse

License: ISC License

JavaScript 100.00%

google-lighthouse chrome crawler

lightcrawler's Introduction

lightcrawler

Crawl a website and run it through Google lighthouse

npm install --save-dev lightcrawler

lightcrawler --url https://atom.io/ --config lightcrawler-config.json

where lightcrawler-config.json looks something like this:

{
  "extends": "lighthouse:default",
  "settings": {
    "crawler": {
      "maxDepth": 2,
      "maxChromeInstances": 5
    },
    "onlyCategories": [
      "Accessibility",
      "Performance",
      "Best Practices"
    ],
    "onlyAudits": [
      "accesskeys",
      "aria-allowed-attr",
      "external-anchors-use-rel-noopener",
      "geolocation-on-start",
      "no-document-write",
      "no-mutation-events",
      "no-old-flexbox",
      "time-to-interactive",
      "user-timings",
      "viewport",
      "without-javascript"
    ]
  }
}

Enjoy!

lightcrawler's People

Contributors

Stargazers

Watchers

Forkers

agdolla robertjgabriel jebcat1982 whitebinary zhangtemplar greeneyes8 yoshuawuyts spillchek2 gedzakis kermits931 hsj314199 songshuhome wmacjn ramchentheersridhar patdesorcy akbarkhosravi6 lakshaytalwar bogdananton maicaijian mumuhui fexi12 nimishb merfaveli shraddhabhivgade mamedegmm cookingwp hansangyoon chen-jc igorkurash1967 heriqnaz palash96 shakilintldev moustafafci anafmoraes suhashankare lifullprasetyama manumanu199719 maxsaxedesignweb odkurzacz89 jangguik24 paulvanbill 742486680 lexluga domenlee pndkrwap86 network27 lusimba work-prjs txderfrt ilichev-aleksey germaniakaiser dkelln nir-mal ginhub jiamingchong flutman kelen1995 anandpvid dragonix11 secondstring shoetech akash05rock loa112 ayushmaanagrawal manish-mittal surisurya446 utsavestha jy2265922 ibiduno agustianto filipequiteque leonardodeschamps globalizacia rabca mnavarropina ffever zxy1131167456 anirudh5 kublaj justingeeslin dr3 life-without-barriers joellisenby fadysamirsadek va21eu mathewhd thezedwards decebal aaron033 efraineduardo iqostw waldyrious moemachef xinzhen2015 mayo2943 hhy5277 jayd2446 oldschoolonline prak2419 mpnkhan

lightcrawler's Issues

lightcrawler: command not found

It seems to install fine, but.

$ npm install lightcrawler

[email protected]
updated 1 package and audited 789 packages in 2.884s
found 0 vulnerabilities

$ lightcrawler --url https://atom.io/ --config lightcrawler-config.json

-bash: lightcrawler: command not found

Ability to use the extra-headers feature of lighthouse

Lighthouse has an --extra-headers cli flag that allows us to pass in additional header e.g. Cookies with each request. Is there a way of doing this with lightcrawler?

I have look through the code but I cannot see anyway of doing this at the moment.

Error: spawn UNKNOWN

I have created a test project, but it occured an error when trying to run

Error: spawn UNKNOWN
at exports._errnoException (util.js:1020:11)
at ChildProcess.spawn (internal/child_process.js:328:11)
at Object.exports.spawn (child_process.js:369:9)
....

Any ideas?

Add documentation to modify lightcrawler-config.json

There is a lightcrawler-config.json but too me a while to get to the architecture doc to learn how to configure it. A link here might be good:
https://github.com/GoogleChrome/lighthouse/blob/master/docs/architecture.md

Maybe there is a better one.

How can I get HTML report using lightcrawler

@kevinsawicki
As of now, I got performance output in terminal, Is it possible to generate HTML report using lightcrawler?

Feature: pass credentials to login

Web apps often have a large chunk of their functionality hidden beyond a login screen. It'd be cool if there was a way to fill in these credentials while crawling so that we can follow up on these pages. Simple crawler supports this, so all that's needed is support in the API. Thanks!

Are there page by page results?

If the site is crawled with Lighthouse where are all of the URLs and the page scores? I was expecting a CSV or JSON file output.

Use case: site report in HTML

A lighthouse user sent me a use case and I thought it may make sense on the lightcrawler side:

I have a clean WP site, I install a new theme, and we want to run a set of audits across the pages: crawl the pages, record LH reports for each page, take final screenshots of each page, and spit out a summary report with links to individual pages.

Is that UX something you'd consider in-scope for lightcrawler? It seems to have a little different approach as far as reporting (HTML vs stdout) so curious what you think.

Smsfaisalsherin

My personal websites