Giter VIP home page Giter VIP logo

chromium-history-version-crawler's Introduction

chromium-history-version-crawler

Crawlers to produce Chromium versions mapping to chromium_base_position.

Result JSON data: https://github.com/vikyd/chromium-history-version-position

View page: https://vikyd.github.io/download-chromium-history-version/

[TOC]

Steps

All output json files locate in json folder.

Step1

Find all available Chromiunm versions, then find the chromium_base_position of each version. And genrate: all-version.json, version-position.json .

# doc: `node version-position-crawler.js -h`
# `all mode` or `incremental mode`
node version-position-crawler.js

all-version.json:

[
  "90.0.4399.1",
  "90.0.4399.0",
  "90.0.4398.1",
  "90.0.4398.0"
  // ...
]

version-position.json:

{
  "90.0.4399.1": "846615",
  "90.0.4399.0": "846615",
  "90.0.4398.1": "846545",
  "90.0.4398.0": "846545"
  // ...
}

Step2

Find all available Chromiunm chromium_base_position of each OS. And generate: position/position-Mac.json etc.

node position-crawler.js
# about 90 seconds

position-Mac.json:

[
  "15734",
  "15749",
  "15839",
  "15942"
  // ...
]

Step3

Compare version-position.json and each position/position-os.json, extract the intersection of the chromium_base_position, generate the final json files: version-position-Mac.json etc.

node ver-pos-os-generator.js

Mac-ver-pos.json:

{
  "90.0.4398.1": "846545",
  "90.0.4398.0": "846545",
  "90.0.4396.2": "845872",
  "90.0.4396.1": "845872"
  // ...
}

json steps

all-version.json -> version-position.json ->
                                              -> ver-pos-[os].json
                       position-[os].json ->

Full mode

It is hard to fetch all position successfully at one loop.

If you really want a full fetch, you may need to modify parameters in the js file.

Missing data

Because of some problems of the position api:

Most version before v38.0.0.0 is missing.

chromium-history-version-crawler's People

Contributors

sxzz avatar vikyd avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

chromium-history-version-crawler's Issues

No more data from omahaproxy

https://omahaproxy.appspot.com/

*** Service Permanently Offline ***

OmahaProxy has been turned down.
Please migrate to https://chromiumdash.appspot.com.

Announcement: https://groups.google.com/a/chromium.org/g/chromium-dev/c/uH-nFrOLWtE/m/PhUj_inyAQAJ

I'm looking into https://developer.chrome.com/blog/chrome-for-testing/ data and JSON endpoints for mapping milestone versions to revisions โ€” and once you have a revision you can link to the chromium snapshots archive with it (instead of the chrome-testing endpoints it was designed for) so there might be viable alternatives.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.