Giter VIP home page Giter VIP logo

har-reader's Introduction

Creating my own HAR Reader

I generally use a complex rendering engine to create a few content heavy websites locally. When I do, the original files that generate the website will be mostly using a server side language like PHP or Node JS (along with more complexity). What I might need would be a static site with a few assets and HTML files to host it in GitHub or other similar platforms.

What I generally do is to run them locally using http://localhost/ and just filter out the only assets that are called while rendering the page. This is done by capturing all the network requests in a HTTP Archive and then getting the URLs that are needed and rebuilding the target static website.

HAR & HAR Reader

So what's this HAR? HAR (HTTP Archive) is a file format used by several HTTP session tools to export the captured data. The format is basically a JSON object with a particular field distribution. In any case, please note that not all the fields are mandatory, and many times some information won't be saved to the file.

Also beware, HAR files contain sensitive data!

  • content of the pages you downloaded while recording
  • your cookies, which will allow anyone with the HAR file to impersonate your account
  • all the information that you submitted while recording: personal details, passwords, credit card numbers...

I have been using Google HAR Analyzer, a part of G Suite toolbox. This is a nice app by Google to analyse the captured HAR files, but there's a big downside to it.

Motivation & Current Issues

The HAR Analyzer by Google works on client side and depending on your machine and browser, it may not handle huge files. I am using a MacBook Pro 15" Retina with 2.8 GHz CPU and 16 GB of RAM, but unfortunately, even such a system is not capable of handling more than 50 MB of JSON file processing. I had to think of a way and had a thought, if it's just processing JSON, all I need would be a JavaScript engine!

Node JS to the rescue!

Google Chrome has been praised as containing one of the best JavaScript engines, which is V8 and that's the same engine being used in Node JS. All I need is just the engine and not a browser. Now that I have got the JavaScript engine, these are the steps I thought of:

  1. Read the HAR file as string and store it locally.
  2. Parse the string as JavaScript, as HAR files is just a big JSON file.
  3. Get the path of the URL entries and push it to an array.
  4. Join the contents of the array in the way I want.
  5. Writing it to a new file as showing it is too mainstream.

Blog Post

I have covered the whole thing in my blog post here. Read more on Creating my own HAR Reader on my blog.

har-reader's People

Contributors

prathibhas avatar praveenscience avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.