const LIFECYCLE = {
 InititalBinding: 1,
 BeforePageload: 2,
 AfterPageLoad: 3,
 AtRest: 4, 
… 


class Thingthing extends Gather {

  static gather(opts) {

    driver.waitFor(LIFECYCLE.BeforePageload)
    .then(_ => …)
    .then(_ => …)
    .then(_ => driver.waitFor(LIFECYCLE.AtRest)
    .then(_ => …)

Disclaimer: All the names are definitely wrong.

But that said, I'm thinking this seems a nice way of managing scheduling of all gather work, so we don't step on toes and minimize repetitive work.

All gathers MUST start with a waitFor. It's possible they have a second.

Thoughts?

Time 'To Interactive (or TTI) is a classic success metric across the web, games and application development. We're attempting to provide a robust version of it for the web platform.

Key goals for the metric:

• User should feel like the page is done enough to start interacting -- they may scroll, tap on notification icons, start reading and expect the page to be not move around any longer.
• The page should be ready for the user. It's not enough to present the appearance of done-ness. The user shouldn't have problems once they start engaging with the page, because it's either still busy or uninitialized. In other words, the page is not ready if main thread contention is preventing user interaction from being successful.

A working definition for Time To Interactive

Updated: April 6, 2016

There are two major timestamps, composed of smaller ones:

Timestamp: Visually ready for interaction

Visually ready is the Math.max() of these timestamps:

1. domContentLoadedEventEnd has fired
   • Definition: HTML parsing has finished, all DOMContentLoaded handlers have run
2. User-agent loading indicator is done
   • Current definition: Top frame and all iframes have fired window load event
   • Proposed definition (from cl/1860743002): top frame only: DCL ended and all layout-blocking resources, plus images that begun their request before DCL ended have finished.
   • Alternative definition (from Chrome on Android Progress Bar Enhancements - google-only, sry): top frame's DOMContentLoaded + top frame's images (who started before DCL) are loaded (IMO, we should use this one)
     • Perhaps only include we only include above-the-fold-ish images.
3. Above-the-fold layout has stabilized
   • Could use WebMeaningfulLayout.h
   • Could use ksakamoto's "largest layout"?
4. What about fonts?
   • They are not necessarily addressed by above items.
   • Do we wait for inflight fonts requests? Only if the text remains invisible (if the user allows a fallback font, then we don't wait for inflight)
   • Can we ignore icon-fonts like font-awesome?
   • We should at least collect the font-painted timestamps for all webfonts
   • The Time to First Meaningful Paint: a layout-based approach algorithm considers in-flight font requests that layout more than 200 characters.
5. Viewport and above-the-fold
   • None of the above considers what is in the viewport, and waits for resources that may be offscreen or 10,000px away.
   • Should we only consider images that are above-the-fold?

Timestamp: Main thread ready for interaction

Once visually ready has hit, we begin considering thread availability.

1. Main thread is available
   • Definition TBD.
   • Approximate: 90% percentile main thread task duration ~ under 300ms. Maximum main thread task duration is < 400ms
   • Perhaps: System Health "input readiness" Metric by @benshayden
   • Perhaps: Ask requestIdleCallback for deadlines
     • "You call rIC until you get at least 20ms. Treat it as a proxy for page being settled. Seemed to correlate pretty well" ~@paullewis & jochen

@skyostil added: "it's super hard to congest the compositor, but a fair indicator will be availability of time on main thread because a) a lot of work will be main-thread bound and b) CPU contention (he thinks) can sometimes block the compositor from allowing scroll."

@paullewis added: "some of this may be in our wording where we explain that we're treating main thread as a proxy for interactivity"

Main-thread dependent input events

• Question: Do we treat pages with slow-scrolling-regions (global touchstart, etc) the same as ones without?
• Verdict: yes, until there is a CSP-like policy that makes it impossible to add these handlers

Key Data, Goals, and Scoring

• milliseconds: navigationStart ─⇥ visually ready
• milliseconds: navigationStart ─⇥ thread ready
• milliseconds: visually ready ─⇥ thread ready
  • if the delta between "looks done" and "is ready" is bigger and bigger.. then you're tanking the UX and the score should represent that.

I think it'd make most sense to do this on our side. That'd also allow us to handle the err in a generalized fashion.

@brendankenny and I were discussing how this would help the ergonomics of development some so we have a unified async structure.

I'm unsure on priority of improving this vs getting a few more tests written. Thoughts?

try for instance

https://moji-brush.com

note how everything breaks.

Manual evaluation: manifest has name, background_color, theme_color, icons at 192px and up

input:

• parsed manifest - #22

audit:

• based on docs & comments from Mounir, the following are required:
  1. has manifest
  2. manifest is valid (enough)
  3. name
  4. background_color
  5. theme_color
  6. and icons at 192px and up

a brief interlude on.... icon sizes

.. please see #291 for all the details about icon sizes and whats required for splash, etc.

Docs: Splash screen on /web

See also "PWA validator" bookmarklet - #17

Afaik on the Android homescreen, if a short_name contains too many characters it will be trimmed to something like My Totally Awesome App -> My Totally Awe.... It may be useful to check the spec or with Mounir regarding expected lengths for these app names and whether flagging this via Lighthouse makes sense.

I've personally run into issues with needing to manually test this on-device (or rethink the names used) to avoid truncating, so throwing the idea out there.

Just tried running a lighthouse trace and hit the following issues:

• launch-chrome.sh is OS X. Would it be worth considering moving this to an npm script and using which to pick the version of chrome to use

• Node 5.9.1 is required. We could add a check for >= 5.9 and throw a warning otherwise

• On linux I get the following error:

  TypeError: Cannot read property '_destroySSL' of undefined
  at destroySSL (_tls_wrap.js:370:7)

Going to dig into that one and try and figure out the problem but wanted to raise an issue on it..

I suppose there's a few cases that we shouldn't bomb on.

• link rel manifest not found in DOM
• link rel manifest found, but it 404s
• manifest asset retrieved but it's invalid JSON
• manifest members are missing, and all that.

I think we have decent coverage here, but we may need to bulk up tests for the manifest gatherer.

@brendankenny wanna take a stab here?

We can either use the DevToolsTimelineModel (which we currently do in FMP), or we can require in traceviewer.

My preference is to try and use the latter wherever we can, but fall back to the former where it offers other insights into the data.

Thoughts?

Problem with apps that send different responses based on User-Agent.

• Looks like DevTools and lighthouse(extension) debugger cannot be open at the same time - https://bugs.chromium.org/p/chromium/issues/detail?id=129539 . So device mode emulation with lighthouse extension seems not possible
• lighthouse uses sendCommand to emulate device mode when opening a url. So you can use lighthouse extension after opening url from a lighthouse cli. But the app/manifest.json isn't called with device mode on as it is called in the scope of the extension without device mode on. So user agent check kicks in sends no manifest for desktop. And this fails all the checks depending on manifest.
• And opening chrome with device mode "on" is also not an option - https://bugs.chromium.org/p/chromium/issues/detail?id=373169&thanks=373169&ts=1400050662

Any ideas ?

None, some, jsdoc, esdoc? wdyt?

Requirement: All assets served on HTTPS
Manual evaluation: Look at DevTools security panel to verify no problems, all green.

input:

• Use Protocol Security domain securityStateChanged event

audit:

• to see if main origin is considered secure

when:

• page loaded and in idle state

Capturing how available the main thread is. This is a proxy metric to for potential input latency, as we cannot predict when input will hit the browser.

Update: See details at Timestamp: Main thread ready for interaction in #27

Scoring

S-curve handled by Ben's math. Basic expectations:

• Goal: 90% percentile main thread task duration ~ under 300ms
• Goal: Maximum main thread task duration is < 400ms
• 100/100 is 90% percentile figure is <12ms
• 0/100 is 90% percentile figure is <1000ms.
• TBD: Determine curve.

Brendan is put up a strawman for this.

It will include warning/severity levels and handling atomic results.

I figure we can add a control that causes a full navigation of the page so we can capture page load traces and network events. This makes the extension less 'automatic', but the alternative would be to do the capture on every page load, which would be bad.

@paulirish wdyt? Am I missing something?

Manual evaluation: manifest has short_name, 144x144 png icon

input:

• parsed manifest - #22

audit:

• has a registered SW
• valid start_url
• valid name
• valid short_name
• icon of size >= 144x144 and png (either type image/png or filename ending in .png)

update: ^ Copied from brendan's comment below.

See also "PWA validator" bookmarklet - #17

docs: install-to-homescreen on /web

when:

• page loaded and in idle state

Right now it looks like it might be an auditing tool?

What should an audit return for statuses like "test not applicable" or "metric couldn't be computed", etc?

It's not reporting a test failure, nor a runtime exception.. @samccone recommended a result.fault state, which is what i tried out here

thoughts?

Komoroske brought this up during review.

To manage expectations, we need to point people to this repo and indicate things are still in development

Non-async script resources that are layout-blocking.

Calculating the total time cost:

• Grab all blocking JS requests (script net requests with priority of {Medium, High, VeryHigh})
• Duration of each script == from fetchStart though responseEnd until ExecuteScript finishes.
• Last critical script == last script to executescript before first meaningful paint. Discard any following.
• Total duration == Activity time span from first script's start to last script's finish.

What we report:

• What are the resources
• Their individual times of (network + main thread)
• Total activity time span.

I'll do this when my current PRs are merged or closed.

It feels like the audits are a bit unwieldy as classes. I was curious what they'd look like a bit more nude..

Here's a before and after:

(I've removed annotations from both, as they're particularly nasty for audits.)

@paullewis I spoke to you briefly about this and I believe you weren't sold on it. I'm in no rush, but curious what you and others think.

Lighthouse full extension

We merge the two: LighthouseLite (#37) and Lighthouse-core.

• Swap in chrome.debugger API in for chrome-remote-interface.
  • E.g. Here's protocol via chrome.debugger: https://github.com/paulirish/emulation-popup-ext/blob/master/app/scripts/background.js#L95-L123
• Browserify lighthouse/extension.js, etc
• Redeploy on top of LHLite

Detect whether a site is using CSP rules. If no unsafe rules exist then say the site is using good CSP values. If an unsafe rule does exist, then warn that there could be possible issues with the policy being enforced.

Sound like a good audit to add?

A few audits rely on a parsedManifest..

gather:

• protocol -> grab manifest file resource - horo@ has a canonical technique for this.
• parse w/ https://github.com/mounirlamouri/manifest-validator

report errors in parsing to user

Over in my fMP pr I noted it'd be valuable to save the trace to disk.. mostly so that I can verify the metrics and summarized data is correct.

@brendankenny rightly pointed out that in addition to debugging, there's value for saving the artifacts locally for timeseries analysis (a la big-rig).

Personally, I'd be happy to save all my runs artifacts until i want to free up the disk space.

Any ideas on what setup would make sense? cmd-line option -> something?

Perhaps ask the user if they have launched chrome via our helper script.

I think we want Blink style for the JS. What do others think?

Perhaps we can extend https://github.com/google/eslint-config-google

I'd be happy with a line length max > 110 though. :)

The one in CWS is still based on the gist. Been a while, but I think the state of the master is good enough for CWS.

Currently we: 1) get all SW registrations 2) filter to to see we have any in an "activated" state. This is good for right now, but it doesn't yet verify that offline caching provided via Service Worker
and it successfully page loads while offline. That'll be the next phase of this test.

Manual evaluation: Flip page offline via DevTools network throttling. Try it out

input:

• all matched SW registrations

audit:

Something like this…

1. flip network throttling to offline.
2. maybe clear disk cache
3. reload the page (or more cleanly: load about:blank and then navigate to the url)
4. network domain can communicate if the page's network request was served via SW.
5. verify its a 200, and page is "good"

when:

• page loaded and in idle state
• everything else is done?

At the moment viewport audit is running a regex over the entire html. It would be useful for this (and future audits) to seperate the body and the head as inputs. The other alternative would be to have these as gathers, but that feels like it wouldn't be as flexible.

Thoughts on having input.html + input.head + html body?

Manual evaluation: manifest has theme_color, <head> has <meta name="theme-color" content="_any valid CSS color_">

input:

• parsed manifest - #22

audit:

• parse manifest for theme_color.
• Grab DOM and parse document.head for the theme-color meta.

Currently both are required. Chrome will be transitioning to use manifest theme-color by default: go/fizz-feature-backlog has a tracking item for this.

See also "PWA validator" bookmarklet - #17

Docs: Support for theme-color in Chrome 39 for Android, and whatwg/meta-theme-color: Spec for the theme-color meta extension

fewer dependencies?

https://github.com/GoogleChrome/lighthouse/blob/master/helpers/network-recorder.js#L21-L81

@brendankenny I think this is on you, sir. It's in case other people require('lighthouse') at some point in the future and we just, you know, stomp the global.

and drop into gatherers/theme-color.js

just pull out

Small bug noticed when demoing lighthouse: The extension and CLI report different numbers for this item. Likely a problem in the gatherers.

Ideally, the first paint where the user feels that the primary content is visible

This means paints with only the topnav don't qualify. Any primary text must be visible (and not waiting for fonts). If an image is critical to the page (e.g. e-commerce product page), then first meaningful paint requires it to be visible.

Manual evaluation: Network Tab in DevTools timeline. Reload with screenshots/filmstrip enabled. Select the first screenshot with a meaningful view.

Input:

• Use ksakamoto's work:
  • First meaningful paint? - Google Groups
  • First Meaningful Paint - manual
    evaluation
  • Time-to-first-X-paint metrics: status and refinement
    plans
  • "Is it useful?" metric (aka. First meaningful paint): layout-based approach - Google Groups

Audit Scoring

• Metric's value is milliseconds from navigationStarted until this paint hit swapBuffers from GPU/browser threads.
• Goal: first meaningful paint delivered from first 14K of the HTML document in 1000ms (on the reference device/network class)
• Metric's score will be expressed as n/100.
  • 100/100 score is fMP in <= 1000ms
  • 90/100 is ~= 3000ms
  • TBD: Determine curve of score down to 0/100.

Lighthouse Lite extension

• Immediately consume kinlan's gist bookmarklet
• Publish as internal-only chrome extension.
• Deploy as Chrome extension
  • Some deets in https://goto.google.com/lighthouse-minutes

@samccone is doing this now. Should have something that runs and reports the exact same thing quite soon.

(orig reported at #73 )

We can probably just use some basic bash hackery of if hash chromium 2>/dev/null etc etc.

Loving the plugin so far. One thing that might be beneficial is to link to explanations for things. ex: I'm missing a canonical URL on one of my sites but if I don't know what a canonical URL is there's no explanation for how I can add one :D

(orig reported at #73 )

Since we're using a bunch of es2015 features, node 4 isn't supported.

The cli should error and exit if you're not running using a 5.x runtime.

question: Should the error message recommend using nvm?

As Dion recommended, the extension can be a passive PWA inspection tool, lighting up as active when a manifest is detected.

I investigated using the chrome.declarativeContent API for this, however it is limited to visible elements, and a <link> element is not one.

The best solution appears to be a small content script, as "run_at": "document_idle", which looks for the manifest <link> in the DOM and sets the page action icon/badge accordingly.

Does that sound about right to folks?

Reference Device & Network Conditions

Network Latency

• 150ms RTT is a reasonable approximation for "typical mobile"
• First 14K can be delivered with TCP/TLS handshakes in 4 round trips.
• 150ms is ~= 57th percentile mobile latency
  • Based on industry research data
  • 57ms at 10th percentile. 135ms at 50th. 853ms at 90th.
  • 85ms to 280ms hits the quartiles.
  • 150ms is memorable and representative.
• ( 150ms latency * 4 RTTs ) = 600ms
• 1000ms = 600ms transfer time + 100ms server response time + 150ms control
  plane + 150ms rendering
  • see also Mobile Analysis in PageSpeed Insights
• 500ms (~85th percentile) would be a good approximation for "typical slow mobile"

Network throughput

• 1.6Mbps download throughput
  • 1.6 is used in WebPageTest's 3g preset. WebPageTest experimented to find
    its preset sweet spots and used
    facebook/augmented-traffic-control
    originally.
  • Still seeking improved numbers for typical and typical-slow mobile
    throughput.
    crbug/590880

Device Class

• Nexus 5 for now.
  • https://github.com/facebook/device-year-class
    approaches this problem. Others have looked at instruction count as a
    proxy.

Implementation

• emulateNetworkConditions should be set for the page and SW
• setCPUThrottlingRate can help us approximate.
  • Until calibration is implemented on the devtools backend, we may need to do a calibration run ourselves.
  • Let's doublecheck with @a1ph about Windows, as original commit says POSIX support only.
  • Workers may be blacklisted, let's also check about throttling service worker

Wondering if we should add this to both the root package.json and the extension, or just the extension. WDYT?

Should we have an audit to discourage use of plugins? E.g. we already list use of Flash as a common mistake in our "mobile friendly documentation"

Some types of videos or content are not playable on mobile devices, such as license-constrained media or experiences that require Flash or other players that are not broadly supported on mobile devices. Unplayable content, when featured on a page of any website can be very frustrating for users.

A "simple" way to ensure that this is the case:

• check that the site's CSP policy sets object-src 'none'
• check that iframes have a sandbox attribute

old tests will need to be converted:

• bigrig processor tests (@paullewis)
• manifest-parser (@brendankenny)
• npm uninstall --save-dev chai

Calling requestAppBanner will fetch the manifest, which we can pick up in the network requests record.

This avoids the DOM dance and make sure we're testing the same manifest that Chrome does.

for now you need --enable-add-to-shelf for this to work

We may want to get rid of the transpilation to make dev a bit more sane.

I'm curious about using https://github.com/JamesMGreene/breq (require() polyfill via sync xhr).

Thoughts?

(Unrelatedly, doing the work in the background page will be easier to debug than doing it in popup.html).

https://gist.github.com/PaulKinlan/ce7c707e4cb96c752816