chromium / caterpillar Goto Github PK

Project to investigate porting Chrome Apps to websites.

License: Apache License 2.0

Python 60.12% JavaScript 33.75% HTML 3.92% CSS 2.20%

caterpillar's Introduction

Caterpillar

Given the recent interest in progressive web apps, this experimental project investigates whether it is feasible to automatically port Chrome Apps to web sites that run offline in Chrome and other modern browsers.

Installation

Extract the code into a folder. Install dependencies with pip and npm:

$ pip install -r requirements.txt && npm install

Usage

./caterpillar.py config -i config.json
./caterpillar.py convert -c config.json my-chrome-app/ my-web-app/

This will convert an unpackaged Chrome App my-chrome-app/ into a progressive web app my-web-app/ and generate a conversion report inside a subdirectory of my-web-app/ (depending on the configuration options you set).

For more detailed documentation, see the manual.

Running Tests

Running all tests:

./run_tests.py

Running Python tests only:

./run_tests.py py

Running specific Python tests:

./run_tests.py py module_test
./run_tests.py py module_test.TestClass
./run_tests.py py module_test.TestClass.test_function

Running JavaScript tests only:

./run_tests.py js

The JavaScript tests assume you have Chrome and Firefox (version 45 or greater). If you want to run the tests on different browsers, edit karma.conf.js.

Disclaimer

This is not an official Google product (experimental or otherwise), it is just code that happens to be owned by Google.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

caterpillar's People

Contributors

Stargazers

Watchers

caterpillar's Issues

End-to-end test app conversion

There should be at least one end-to-end test of converting an app — some kind of regression test, so get the output of an app conversion, run it through Caterpillar, check that the output is correct, then save that somewhere and test that Caterpillar's output matches it.

Strange console error when server 404s (for non-cached files)

This is seen in JavaScript console:

Uncaught (in promise) TypeError: Failed to execute 'fetch' on 'ServiceWorkerGlobalScope': Cannot construct a Request with a Request whose mode is 'navigate' and a non-empty RequestInit.(…)

Report includes API usage from boilerplate code

The report usage should ignore anything inside Caterpillar's boilerplate folder.

Make a test runner

With two different languages and the directory structure, testing is non-trivial, so there should be a script that runs appropriate tests.

chrome.google.com URLs show up as APIs

Also, any URL of the form chrome.*, but that's generally chrome.google.com.

BeautifulSoup generating warnings during tests

./run_tests.py py

Tests are generating this warning:

/usr/local/lib/python2.7/dist-packages/bs4/__init__.py:166: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

To get rid of this warning, change this:

 BeautifulSoup([your markup])

to this:

 BeautifulSoup([your markup], "lxml")

  markup_type=markup_type))

Most calls to BeautifulSoup pass 'html.parser' as the parser, but there are lots of calls to BeautifulSoup with no explicit parser, which is what this is warning about. We can go through and set all of them to 'html.parser'.

But the test in report_test.py, TestGenerateSummary.test_generate_summary fails if you use 'html.parser'. It passes if you use 'lxml', but this indicates a problem with the test if the choice of parser matters. And also it means we have an unspecified dependency on lxml. So this is a legitimate warning that we don't want to ignore.

Polyfill manifest warnings member should be optional

The warnings member should really be optional.

Caterpillar should generate app.info.js

app.info.js is used by the runtime polyfill (and possibly in future polyfills) for querying the manifest and Chrome App ID (basically any metadata we were originally able to access in the Chrome App). This should be generated by Caterpillar and put into the root of the output web app.

Report code should be word-wrapped

Currently, code shown in the report overflows the container. This should be word-wrapped so it looks nicer and is easier to read.

This should be a matter of adding some CSS to report.css, e.g.

.prettyprint {
    word-wrap: break-word;
}

Notifications don't work in service workers

See here for why — we're using new Notification(...) when we should be using the service worker registration's showNotification.

Add tests for `notifications.onClicked`

Move `caterpillar` out of the `chrome` namespace

Karma tests should also run Firefox

Generating service worker fails when some Unicode paths need to be cached

Use this config file:

{
  "boilerplate_dir": "caterpillar-boilerplate-\ud83d\udcc2", 
  "report_dir": "report-caterpillar-\u2611", 
  "start_url": "index.html"
}

Convert an app. Service worker generation raises errors.

Default BeautifulSoup parser mangles DOCTYPEs

Input this HTML file into edit_code:

<!DOCTYPE html>
<html>
  <head>
  </head>
  <body>
  </body>
</html>

By default, BeautifulSoup will convert this into

<html>
 <body>
  <p>

   <!DOCTYPE html>
  </p>
 </body>
</html>

which is not valid HTML. Additionally, this then trips error #16.

Add tests for `tts.isSpeaking` and `tts.onEvent`

Slash URLs should serve the index page

The service worker should serve the start page when the user hits a URL ending in /. Currently it just forwards it to the server, which means it may or may not serve the start page depending on whether the user/server is online.

Disallow some web functionality

Some web functionality is disabled in Chrome Apps. To properly convert apps we should make sure that this functionality is disabled as much as possible in the output web apps (most notably user text selection).

Having no background script should raise a warning if the app is launched from a local path

Say we have an app which has a manifest member like the following. I think that this means it's a packaged app, but it's locally hosted.

 "app": {
    "launch": {
       "local_path": "index.html"
    }
 }

Caterpillar can still convert these to websites, and give a useful report, so we should change the current behaviour of raising an error on no background script to raising a warning if the app is launched from a local path. Note that this breaks the assumption that a background script exists, so part of the conversion process will also have to be updated.

Loading a polyfill manifest for an API with no polyfill should return a default "not implemented" manifest

polyfill_manifest.load is somewhere it makes sense to return a default value rather than raising an exception if you try and load the manifest of a polyfill that doesn't exist. This is because everywhere that a polyfill manifest is used expects polyfill manifests to exist even for APIs without polyfills. The default should specify status as "none" and have empty dependencies and warnings.

/ refers to the app root in a Chrome App, but the website root on the open web

/ refers to the app root in URLs in a Chrome App, but the website root on the open web. As a result of this difference, assets may fail in a converted app — maybe the converted app is hosted in a subdirectory of the website root, but the converted app looks for /css/styles.css. I'm not sure whether this is something we should try and fix?

Implement or stub missing methods in runtime polyfill

The runtime polyfill is missing many methods and some objects. These should be either implemented or explicitly stubbed.

Add a npm package.json to keep track of dependencies

`report.format_html` should not highlight APIs not included in the API dictionary passed as argument

e.g. highlighting chrome.tts.speak('hello'); with the API dictionary only containing information relating to the storage API should return chrome.tts.speak('hello'); with no modifications.

Injecting script tags into HTML with no body raises an AttributeError

JSON shouldn't have trailing commas

Most of the JSON written so far has used trailing commas, but that's not actually valid JSON. This should be fixed.

Add tests for `notifications.getAll`

This will require adding some way to reset the internal state of the notifications polyfill — otherwise, the output of getAll will be unpredictable, since it will return all notifications that have been involved in tests thus far.

Following the example of the storage polyfill tests (currently in a CL here), such a reset function should be included in the chrome.caterpillar.notifications namespace and should include a ForTests suffix, e.g. chrome.caterpillar.notifications.resetNotificationsForTests.

run_tests fails due to missing dependencies

Run run_tests.py js. Output:

/path/to/caterpillar/node_modules/karma/node_modules/di/lib/injector.js:9
      throw error('No provider for "' + name + '"!');
            ^
Error: No provider for "framework:sinon"! (Resolving: framework:sinon)
    at error (/path/to/caterpillar/node_modules/karma/node_modules/di/lib/injector.js:22:68)
    at Object.parent.get (/path/to/caterpillar/node_modules/karma/node_modules/di/lib/injector.js:9:13)
    at get (/path/to/caterpillar/node_modules/karma/node_modules/di/lib/injector.js:54:19)
    at /path/to/caterpillar/node_modules/karma/lib/server.js:138:20
    at Array.forEach (native)
    at Server._start (/path/to/caterpillar/node_modules/karma/lib/server.js:137:21)
    at invoke (/path/to/caterpillar/node_modules/karma/node_modules/di/lib/injector.js:75:15)
    at Server.start (/path/to/caterpillar/node_modules/karma/lib/server.js:102:18)
    at Object.exports.run (/path/to/caterpillar/node_modules/karma/lib/cli.js:243:26)
    at Object.<anonymous> (/path/to/caterpillar/node_modules/karma/bin/karma:3:23)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Function.Module.runMain (module.js:497:10)
    at startup (node.js:119:16)

Manually run npm install karma-sinon. New output:

Chrome 50.0.2657 (Linux 0.0.0) runtime getPlatformInfo gets info FAILED
  Died on test #1     at /path/to/caterpillar/tests/js/polyfills/runtime.polyfill.test.js:86:7: platform is not defined
  ReferenceError: platform is not defined
      at Object.<anonymous> (/path/to/caterpillar/tests/js/polyfills/runtime.polyfill.test.js:87:20)
      at runTest (/path/to/caterpillar/node_modules/qunitjs/qunit/qunit.js:905:28)
      at Object.Test.run (/path/to/caterpillar/node_modules/qunitjs/qunit/qunit.js:890:4)
      at /path/to/caterpillar/node_modules/qunitjs/qunit/qunit.js:1032:11
      at process (/path/to/caterpillar/node_modules/qunitjs/qunit/qunit.js:692:24)
      at begin (/path/to/caterpillar/node_modules/qunitjs/qunit/qunit.js:674:2)
      at /path/to/caterpillar/node_modules/qunitjs/qunit/qunit.js:734:4

Needs karma-sinon and platform packages added to package.json.

TTS polyfill should store errors in `chrome.runtime.lastError`

The TTS polyfill currently doesn't handle errors. These should be stored in chrome.runtime.lastError in accordance with the specification of chrome.tts. The helper function chrome.caterpillar.setError should be used for this.

Trying to convert a file (rather than a folder) gives a blank error

Remove custom formatters for `report.report.make_warning`

Currently, the make_warning function is:

make_warning(name, member, text, apis, formatter=None)

Non-default formatters are never used, so we should remove the formatter argument.

Report generation raises a KeyError if an unpolyfilled API appears in the same line of code as a polyfilled API

Caterpillar crashes while generating report

Remove unimplemented config options

There are some config options e.g. root that aren't actually used. These should be removed for now, an issue should be logged for the feature they are attached to, and the options can then be reintroduced when a corresponding feature implementation is added.

Polyfill webview

webview isn't really an API, but it's also widely used. We could probably polyfill at least some of the functionality.

Polyfill chrome.fileSystem

This is the one of the most popular APIs and we can definitely provide some functionality.

Multiple API usages on one line result in one TODO comment.

Say I have a line

chrome.tts.speak(chrome.storage.local.get('something')['something']);`

This will generate one TODO, either

// TODO(Caterpillar): Check usage of chrome.tts.speak.

// TODO(Caterpillar): Check usage of chrome.storage.local.get.

Both should be generated.

Implement `notifications.update`

app.info.js is incorrectly detected as having Chrome Apps APIs

This results in TODOs being added to app.info.js.

`inject_misc_tags` fails if the input HTML document has no head tag

Autostub unpolyfillable APIs

It should be pretty easy to automatically generate stubs for APIs that have no polyfill.

Symbolic links to Caterpillar can't copy Caterpillar JS

Copying polyfills, service worker scripts, etc., fails when caterpillar.py is run as a symbolic link. This is due to e96606f#diff-8730e37d4092a0ad07a30535d18bf8d1L70 as os.path.realpath was removed.

`notifications.create` doesn't work on Firefox

Before creating a notification, the notifications polyfill requests permission to create notifications using Notification.requestPermission. This returns a Promise, and the Promise is used in the polyfill.

However, on Firefox, Notification.requestPermission doesn't return a Promise — it uses a (deprecated) callback function. So notifications.create doesn't work on Firefox.

There should be a fallback of some sort for when Notification.requestPermission uses a callback instead of a Promise.

Globally rename ca_* and pwa_* to chrome_app_* and web_*

All variables should be renamed to fit this naming scheme, so no abbreviations are used.

"Cache buster" GET params should be ignored by the service worker

Resources with ?v=1212 or similar at the end, if cached, should still be served from the cache.

Add tests for configuration.py

Interactive configuration generation doesn't generate correct types

caterpillar config -i doesn't convert input strings into types expected in the output configuration file. For example, if you type false in response to "Whether to normalise all URIs in the web app to the root (True)", Caterpillar will write the string "false" (rather than false) to the configuration file.

This is currently only a problem for the update_uris option.

Add tests for `report.format_html`

`chrome.runtime` should be available to caterpillar.js

Currently, the runtime polyfill is only counted as a requirement if there's an explicit call to it. However, errors are stored in chrome.runtime.lastError. The runtime polyfill should always be a requirement and should be loaded before any other polyfills.

Generating app.info.js based on a manifest containing single-quotes results in JS syntax errors

This is due to replacing the double-quotes with single-quotes to fit the code style.