Giter VIP home page Giter VIP logo

puppeteer-examples's Introduction

Puppeteer examples

Deprecation notice: We are moving our efforts to theheadless.dev, a new free & open source knowledge base for Puppeteer AND Playwright. Idea is the same: practical examples and guides, by the community. We — the team at Checkly — are going to pour a lot of love & care into this new project. Feel free to contribute!

Puppeteer Headless Chrome examples for real life use cases. Clone this repo and run them directy with a simple node command.

git clone https://github.com/checkly/puppeteer-examples
cd puppeteer-examples
npm i
node 2.\ search/amazon.js

You can run these scripts in the puppeteer sandbox.

1. basics

The very basic on getting useful info from web page. Highlights the basic Puppeteer functions.

alerts

Create an alert dialog and close it again.

1. basics/alerts.js

Download file / upload file

Find an image by class selector, downloads the image, saves it to disk and read it again. Use this together with a .fileUpload() method.

1. basics/download_upload.js

emulate devices

Use the built in devices descriptors to emulate an Iphone 6. These are actually shortcuts for calling page.setUserAgent() and page.setViewPort().

1. basics/emulate_devices.js

get the value of common form elements

Gets the value of commonly used HTML form elements using page.$eval()

1. basics/forms.js

get list of links

Scrapes Hacker News for links on the home page and returns the top 10

1. basics/get_list_of_links.js

get text value of an element

Gets the text value of an element by using the page.$eval method

1. basics/get_text_value.js

get title

Get the title of a page and print it to the console.

1. basics/get_title.js

hover

The hover function is a combination of scrolling and putting the mouse into a hover state over the requested element. This example hovers the first track we find on the soundcloud.com homepage, which should trigger the play and like buttons to be visible

1. basics/hover.js

keyboard

types into a text editor

1. basics/keyboard.js

location_faker

Fake the location for the geolocation API used by the browsers

1. basics/location_faker.js

mouse

Most of the things you can click using straight .click() handlers, but for some situation directly instructing the mouse might be convenient. This example load a page that plays back what mouse actions are used on the page.

1. basics/mouse.js

pdf

Renders a PDF of the Puppeteer API spec. This is a pretty long page and will generate a nice, A4 size multi-page PDF.

1. basics/pdf.js

request interception

Uses Puppeteer request interception, blocks images from loading, then snaps a basic screenshot of the full New York Time homepage and saves it a .png file.

1. basics/request_interception.js

screenshots

Snaps a basic screenshot of the full New York Time homepage and saves it a .png file.

1. basics/screenshots.js

screenshots clipped

Grabs and clips out just the stock tickers on the Yahoo finance page

1. basics/screenshots_clipped.js

set cookie

Sets the "login_email" property in a Paypal cookie so the login screen is pre-filled with an email address.

1. basics/set_cookie.js

2. search

Common search input and select methods on search results.

Amazon search

Looks for a "nyan cat pullover" on amazon.com, goes two page two clicks the third one.

2. search/amazon.js

Booking.com search

Finds accommodations in Berlin on Booking.com, takes a screenshot and logs the top 10.

2. search/booking.js

Duck Duck Go search

undefined

2. search/duck_duck_go.js

Youtube search

Looks for Fleetwood Mac's "Dreams" video on youtube.com and clicks on the third video. Waits for 5 seconds for the video to load.

2. search/youtube.js

3. login

Common login scenarios on popular website. Credentials mostly supplied with setting ENV variables.

Github

Logs into Github. Provide your username and password as environment variables when running the script, i.e: GITHUB_USER=myuser GITHUB_PWD=mypassword node github.js

3. login/github.js

Google Social Login

Logs into Checkly using Google social Login. Provide your username and password as environment variables when running the script, i.e: GOOGLE_USER=myuser GOOGLE_PWD=mypassword node google_social.js

3. login/google_social.js

Instagram

Logs into Instagram with credentials. Provide your username and password as environment variables when running the script, i.e: INSTAGRAM_USER=myuser INSTAGRAM_PWD=mypassword node instagram.js

3. login/instagram.js

Microsoft Live Login

Logs into MS Live. Provide your username and password as environment variables when running the script, i.e: linux: MSLIVE_USER=myuser MSLIVE_PWD=mypassword node mslive.js Windows users: SET MSLIVE_USER=myuser SET MSLIVE_PWD=mypassword node mslive.js for more info see here: https://stackoverflow.com/questions/9249830/how-can-i-set-node-env-production-on-windows

3. login/mslive.js

4. shopping-carts

How to handle shopping cart functions like adding and removing items.

Staples shopping cart

Goes to the face paint category and adds to the shopping cart.

4. shopping-carts/staples.js

Walmart shopping cart

Looks for a Nintendo's Mario Odyssey and adds it to the shopping cart.

4. shopping-carts/walmart.js

5. parallel-pages

How to handle Allow parallel processing pages.

Screenshots parallel pages

Allow parallel processing screenshot

5. parallel-pages/screenshots_parallel.js

Screenshots parallel pages in batches

parallel screenshotting of an array of Websites with small example

5. parallel-pages/screenshots_parallel_cologne_colleges.js

A. mocha-tests

Mocha test runner scripts that use Puppeteer and the standard assert library to check specific properties and actions on a page. They can be executed like:

cd "a. mocha-tests"
npx mocha alibaba.js

Alibaba product search

Searches Alibaba.com for a product and checks if the results show up

a. mocha-tests/alibaba.js

Amazon product search

Searches Amazon.com for a product and checks if the results show up

a. mocha-tests/amazon.js

codesandbox.io

Goes to codesandbox.io, creates a new sandbox and selects the Vue.js template

a. mocha-tests/codesandbox.js

Duck Duck Go search

Goes to duckduckgo.com, searches for "chrome puppeteer", asserts the result and snaps a screenshots

a. mocha-tests/duck_duck_go.js

Etsy shopping cart

Goes to etsy.com, select the first knick knack and adds it to the shopping cart.

a. mocha-tests/etsy.js

Gmail signup

Checks the signup flow from the landing page. Clicks the

a. mocha-tests/gmail.js

Google search

Searches Google.com for a term and checks if the first link matches. This check should fail.

a. mocha-tests/google.js

Staples shopping cart

Goes to staples.com and adds a some facepaint to an empty shopping cart. Validates the correct amount.

a. mocha-tests/staples.js

Walmart shopping cart

Goes to walmart.com and adds a Nintendo game to an empty shopping cart. Validates the correct amount.

a. mocha-tests/walmart.js

B. jest-tests

Jest test runner scripts that use Puppeteer to check specific properties and actions on a page. Very similar to the Mocha tests, but using the Jest expect assertions. Install Jest as a global dependency and run them as any other script

npm i -g jest
jest jest-tests/alibaba.js

Alibaba product search

Searches Alibaba.com for a product and checks if the results show up

b. jest-tests/alibaba.spec.js

Amazon product search

Searches Amazon.com for a product and checks if the results show up

b. jest-tests/amazon.spec.js

codesandbox.io

Goes to codesandbox.io, creates a new sandbox and selects the Vue.js template

b. jest-tests/codesandbox.spec.js

Etsy shopping cart

Goes to etsy.com, select the first knick knack and adds it to the shopping cart.

b. jest-tests/etsy.spec.js

Google search

Searches Google.com for a term and checks if the first link matches. This check should fail.

b. jest-tests/google.spec.js

Walmart shopping cart

Goes to walmart.com and adds a Nintendo game to an empty shopping cart. Validates the correct amount.

b. jest-tests/walmart.spec.js

puppeteer-examples's People

Contributors

aofdev avatar batuhan avatar chuongtrh avatar dependabot[bot] avatar evanhalley avatar hack-tramp avatar movd avatar neil-s avatar ragog avatar rinoldsimon avatar saranshdhingra avatar tnolet avatar umutuzgur avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

puppeteer-examples's Issues

Getting Error

(node:4492) UnhandledPromiseRejectionWarning: Error: Navigation Timeout Exceeded: 30000ms exceeded
    at Promise.then (F:\git\Testing\puppeteer-examples\node_modules\puppeteer\lib\NavigatorWatcher.js:71:21)
    at <anonymous>

This error occurs for every script. I think this may be a problem with puppeteer. Anyways I think we should handle this error also. I would like to make a PR regarding this. What say ?

I can UnhandledPromiseRejectionWarnings on everything I run in this library

$ node --version                                                                                          
v8.9.4
$ node 1.\ basics/alerts.js                                                                                
This message is inside an alert box
(node:41930) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Protocol error (Runtime.callFunctionOn): Target closed.
(node:41930) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
node:41789) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Protocol error (Runtime.callFunctionOn): Target closed.
(node:41789) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I don't understand. I have a clean node 8 install on a new Macbook Pro.

cannot find puppeteer

C:\ub16_prj\puppeteer-examples\a. mocha-tests>cnpm i -g puppeteer
Downloading puppeteer to C:\web\nvm\v11.6.0\node_modules\puppeteer_tmp
Copying C:\web\nvm\v11.6.0\node_modules\puppeteer_tmp_puppeteer@1.13.0@puppeteer to C:\web\nvm\v11.6.0\node_modules\puppeteer
Installing puppeteer's dependencies to C:\web\nvm\v11.6.0\node_modules\puppeteer/node_modules
[1/8] proxy-from-env@^1.0.0 installed at node_modules_proxy-from-env@1.0.0@proxy-from-env
[2/8] progress@^2.0.1 installed at node_modules_progress@2.0.3@progress
[3/8] debug@^4.1.0 installed at node_modules_debug@4.1.1@debug
[4/8] mime@^2.0.3 installed at node_modules_mime@2.4.0@mime
[5/8] ws@^6.1.0 installed at node_modules_ws@6.2.0@ws
[6/8] https-proxy-agent@^2.2.1 installed at node_modules_https-proxy-agent@2.2.1@https-proxy-agent
[7/8] rimraf@^2.6.1 installed at node_modules_rimraf@2.6.3@rimraf
[8/8] extract-zip@^1.6.6 installed at node_modules_extract-zip@1.6.7@extract-zip
execute post install 1 scripts...
[1/1] scripts.install [email protected] run "node install.js", root: "C:\web\nvm\v11.6.0\node_modules\puppeteer"
Downloading Chromium r637110 - 143.1 Mb [====================] 100% 0.0s
Chromium downloaded to C:\web\nvm\v11.6.0\node_modules\puppeteer.local-chromium\win64-637110
[1/1] scripts.install [email protected] finished in 1m
All packages installed (42 packages installed from npm registry, used 1m(network 733ms), speed 619.44kB/s, json 39(66.79kB), tarball 387.27kB)

C:\ub16_prj\puppeteer-examples\a. mocha-tests>

C:\ub16_prj\puppeteer-examples\a. mocha-tests>node google.js
internal/modules/cjs/loader.js:605
throw err;
^

Error: Cannot find module 'puppeteer'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:603:15)
at Function.Module._load (internal/modules/cjs/loader.js:529:25)
at Module.require (internal/modules/cjs/loader.js:657:17)
at require (internal/modules/cjs/helpers.js:22:18)
at Object. (C:\ub16_prj\puppeteer-examples\a. mocha-tests\google.js:7:19)
at Module._compile (internal/modules/cjs/loader.js:721:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:732:10)
at Module.load (internal/modules/cjs/loader.js:620:32)
at tryModuleLoad (internal/modules/cjs/loader.js:560:12)
at Function.Module._load (internal/modules/cjs/loader.js:552:3)

Run Puppeteer on remote desktop.

Is it possible to run puppeteer on remote desktop.
So here what my ask is:
First I have my scripts created on Computer A
By giving command from Computer A, my script should invoke a browser in Computer B and should execute my script on the same.

ReferenceError: before is not defined

C:\ub16_prj\puppeteer-examples\a. mocha-tests>node google.js
C:\ub16_prj\puppeteer-examples\a. mocha-tests\google.js:11
before(async () => {
^

ReferenceError: before is not defined
at Object. (C:\ub16_prj\puppeteer-examples\a. mocha-tests\google.js:11:1)
at Module._compile (internal/modules/cjs/loader.js:721:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:732:10)
at Module.load (internal/modules/cjs/loader.js:620:32)
at tryModuleLoad (internal/modules/cjs/loader.js:560:12)
at Function.Module._load (internal/modules/cjs/loader.js:552:3)
at Function.Module.runMain (internal/modules/cjs/loader.js:774:12)
at executeUserCode (internal/bootstrap/node.js:342:17)
at startExecution (internal/bootstrap/node.js:276:5)
at startup (internal/bootstrap/node.js:227:5)

C:\ub16_prj\puppeteer-examples\a. mocha-tests>

Error: DispInvoke: execScript Exception occurred.

My Code

const puppeteer = require('puppeteer-ie');
const screenshot = 'github2.png';
(async () => {
    const browser = await puppeteer.launch({ headless: false })
    const page = await browser.newPage()
    var url;
    url = 'https://google.com';
    try {
        await page.goto(url, {
            timeout: 10000,
            waitUntil: 'networkidle0',
        });
        console.log(`await page.goto('https://github.com/login')`);
        await page.screenshot({ path: screenshot });
        console.log(`await page.screenshot({ path: screenshot })`);
	    browser.close()
	    console.log('See screenshot: ' + screenshot)
    } catch (e) {
    	console.log('xxx');
        console.log(e);
    }
})()

error

λ node index.js
xxx
Error: DispInvoke: execScript Exception occurred.

    at Dispatch.<anonymous> (<anonymous>)
    at ExecutionContext.require (C:\Users\This PC\Desktop\pptr-ie\node_modules\puppeteer-ie\source\ExecutionContext.js:39:16)
    at ExecutionContext.attach (C:\Users\This PC\Desktop\pptr-ie\node_modules\puppeteer-ie\source\ExecutionContext.js:58:20)
    at Page.waitForNavigation (C:\Users\This PC\Desktop\pptr-ie\node_modules\puppeteer-ie\source\Page.js:64:29)
    at async C:\Users\This PC\Desktop\pptr-ie\index.js:34:9 {
  errno: -2147352567,
  code: -2147352319,
  description: 'Could not complete the operation due to error 80020101.'
}

Please help me

Error Installing Puppeteer on AWS linux Ec2 server

Hello,
i am installing puppeteer using below node command.
"npm i puppeteer"
however it is failing with below error
"ERROR: Failed to download Chromium r662092!"

please suggest any solution to overcome this issue.

modifying to accomodate URL list

Excellent code! I am modifying to pull from a URL list, with the choice of either local or Google Sheets. Do you have any best practice suggestions to accomplish this?

Why does google_social.js not work when I set it headless to false?

Time exceed out in while waiting for the password :
await page.waitForSelector('input[type="password"]', { visible: true })
And thus it returns me
(node:9950) UnhandledPromiseRejectionWarning: Error: waiting failed: timeout 30000ms exceeded

Any idea how to fix this?

ENV variable

sir how to provide envoriment variable pls help

How can get cookies

Hi sir,

When my proccess finish i must get cookies from pupeteer on json format how can i do.
thanks and regards

Unable to retriev element data

Hi, really struggling to extrach one bit of data from this web page.
When inspecting the data element and copy the xPath into the console I get an emprt response when clearly the element has a value of 939.
Any help to explain where I'm going wrong would be most appreciated.
I'm using Puppeteer-extra and stealth
Thanks Chris

xPath

Broken sites: trypuppeteer.com and puppeteersandbox.com

https://puppeteersandbox.com/ gives error when running examples:

WebSocket connection to 'wss://a28klffomxbxm8.iot.eu-central-1.amazonaws.com/mqtt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJ7Z7C26WYKEAIL7A%2F20181116%2Feu-central-1%2Fiotdevicegateway%2Faws4_request&X-Amz-Date=20181116T033232Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=d0575f9bad51f919ba710bb2088bfe40d9a461d6bf466e13c8988a4f79307b2d' failed: Error in connection establishment: net::ERR_CERT_SYMANTEC_LEGACY

and https://trypuppeteer.com/ does not respond at all.

basics/hover.js cannot take snapshot without sleep for some time

Hi Team, nice examples!

basics/hover.js example

For me I have to add a sleep() function to make sure await page.screenshot({ path: 'hover.png' }) can capture the hover effect.

const puppeteer = require('puppeteer');

function sleep(ms) {
  return new Promise(resolve => {
    setTimeout(resolve, ms)
  })
}

(async () => {
  const browser = await puppeteer.launch()
  const page = await browser.newPage()
  await page.goto('https://soundcloud.com/')
  await page.hover('.playableTile__artwork')
  await sleep(1000)  // <-- sleep 
  await page.screenshot({ path: 'hover.png' })
  await browser.close()
})()

Is it only me have this issue? Is there anyway I can get rid of sleep() ?

Thanks!

[without sleep]
hover

[with sleep]
hover

Note: we are moving to headless.dev

Hi, I started this repo some years ago. It's still really cool, but also a bit limited. And now there is Playwright. Me and my team decided to put some extra love and care into creating a "puppeteer-example 2.0", called theheadless.dev.

Idea is the same: free & open source examples and knowledge about Puppeteer AND Playwright, all code and content still on GitHub https://github.com/checkly/theheadless.dev

However, we will support this, update it and expand on it to make it the #1 knowledge base on everything relating the new wave of headless browser automation

Multiple browsers with one having tabs/pages?

(browsers set to 3)
Why does the script open 3 separate browsers and then load three pages/tab with the URLs in the LAST of the three browsers?

Three browsers then 3 tabs/pages in the last browser, while the previous two browsers do nothing?

Already Logged In

How can we check if the user is already logged in (cookies are saved already). I want to know whats the way to only go through the login process if the user is not already logged in.

Currently i am doing it like this. Is there a better way?

try {
   await page.waitForSelector('#login-username', {timeout: 3000})
   await page.type('#login-username', USERNAME);
   await page.type('#login-password', PASSWORD);

   await page.click(btnLoginSelector);

   await page.waitForSelector(btnAccountSettingsSelector);
   console.log( 'successfully logged in');
 } catch ( err ) {
   await page.waitForSelector(btnAccountSettingsSelector);
   console.log( 'already logged in');
 }

Wait for user Login

I am looking for an example where the script

Navigates to the Login page of website/application
Waits for the human to login into the website
Detects that the Login is successful
Takes action as programmed in the script
Logsout
Scripts like this can help automate tasks for people. Privacy is preserved as the script does not store/see the login credentials

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.