phantombuster / nickjs Goto Github PK

I'm not sure if one or both of these options is being ignored:

$ node build/test.js
tab.open()
> Tab 1: Aborted (blacklisted by /.*\.js.*/): https://mixergy.com/wp-content/plugins/swiftype-search/assets/install_swiftype.min.js?ver=4.8.2
> Tab 1: Aborted (blacklisted by /.*\.js.*/): https://mixergy.com/wp-includes/js/jquery/jquery.js?ver=1.12.4
> Tab 1: Aborted (blacklisted by /.*\.js.*/): https://mixergy.com/wp-includes/js/jquery/jquery-migrate.min.js?ver=1.4.1
...

test.js:

Nick = require('nickjs');

nick = new Nick(options = {
  printNavigation: false,
  printResourceErrors: false,
  printPageErrors: false,
  blacklist: [/.*\.js.*/]
});

(async() => {
  var tab;
  tab = (await nick.newTab());
  console.log('tab.open()');
  return tab.open('https://mixergy.com/interviews/');
})();

I just paste npm install nickjs and I received some warnings.

Shouldn't a chromium browser gets downloaded as part of nickJS installation instead of manually pointing chromium executable path?

Thanks for the library.

Is there a possible future support for mouse simulations?

hi , when i run the demo, it appears Error: getaddrinfo ENOTFOUND localhost undefined:9222

How do you insert a text inside a textarea that is not belongs to a form tag?

Please enable changing NickJS options after the initial constructor. Specifically, I would like to modify:

• loadImages flag
• blacklist/whitelist
• (and the print* flags would be nice, too)

Motivation: I would like to use a special configuration just when logging in

• Login page requires loadImages=true and no blacklists
• but other pages are OK with loadImages=false and blacklists (and load faster without images & JS files)

When I try to launch the example I get this error :

SyntaxError: Unexpected token '>'

  phantomjs://code/test.js:4 in injectJs
  phantomjs://code/bootstrap.js:456

To launch the script I use :

node_modules/casperjs/bin/casperjs test.js

test.js contains the example from readme.md

EDIT :

I just discoered that casper.js or phantom is not suporting ES6

setTimeout( () =>  console.log('Hello'), 3000 );

this fail :'(

Hi, it's me Again.
It sound dumb (and not really relative to nick) but how can i use Node API like fs (for writing my scrapping results in a json file) ?
Here my code

require('babel-polyfill')
const Nick = require('nickjs')
const nick = new Nick()
var fs = require('fs');
nick.newTab().then(async function (tab) {
  await tab.open('http://example.com')
  await tab.waitUntilVisible('.products-grid')
  await tab.inject('./jquery.js')
  const urls = await tab.evaluate((arg, callback) => {
    const urls = []
    $("li.item").each((index, element) => {
      urls.push($(element).find("div.product-content-wrapper").find('h3.product-name').find('a').attr('href'))
    })
    callback(null, urls)
  })
  var data = []
  for (var i = 0; i < urls.length; i++) {
    await tab.open(urls[i])
    await tab.waitUntilVisible('.catalog-product-view')
    await tab.inject('./jquery.js')
    const type = await tab.evaluate((arg, callback) => {
      var type = $('div.product-name').find('h1').text()
      callback(null, type)
    })
    const img = await tab.evaluate((arg, callback) => {
      var img = $('div#map_container').find('img').attr('src')
      callback(null, img)
    })
    data.push({ type: type, img: img })    
    await tab.screenshot('./uploads/' + i + '.png')
  }
  fs.writeFile('./myjsonfile.json', data, 'utf8', ()=>{
    nick.exit()
  });  
}).catch((err) => {
  console.log('Oops, an error occurred: ' + err)
  nick.exit(1)
})

Here at the end I want to write a json file with my results (var data is an Object).
But in the execution i have an error

undefined is not a function (evaluating 'fs.writeFile('./myjson.json', data, function () {

})')

For my defense, i'm really new to PhantomJs/CasperJs and Babel.
I know my script is executed in a CasperJS context, but i can't figured out why i can't use fs (even with the require('babel-polyfill')

Anyone to help me learn and make me a better dev ?

Hello there,

Is reset option working?

I found that it is not working and so the previous value is left in the input box.

Here is my code.

await tab.sendKeys('#username', id, {reset:true,keepForcus:false});

Can you confirm that?

OS: Ubuntu 16.04
NickJS: The latest one
Google Chrome: 67.0.3396.62

Best regards,

Need some help getting Chrome running in different environments.

In a different thread, you said nickjs looks for google-chrome-unstable before checking CHROME_PATH in environment variables.

Is using google-chrome-unstable the best way to have the same code execute in both a local (Mac) and remote (EC2/Elastic Beanstalk) node.js setup?

If so, can you help with download/install instructions of google-chrome-unstable?

If not, can you provide help with getting nickjs to recognize Chrome in an EC2 instance?

Thx - I know these are probably noob questions.

I am trying to use a SOCKS5 proxy to connect and "http://" is getting appended to the front. The SOCKS5 proxy url takes this form "socks5://username:password@domain:1080".

I get this :
Error: timeout: load event did not fire after xxx ms
while I'm trying to open website such as azlyrics.com or genius.com, even when I extend the timeout option to 60000ms.
I can quickly connect to https://news.ycombinator.com/ or other websites though. Are some websites preventing headless browser connections ? If so, is there any workaround ?

could not connect to chrome debugger after 24 tries (10s): Error: connect ECONNREFUSED 127.0.0.1:9222

this just popped up for some reason on heroku, i have legacy dynos running the same code and working perfectly. tried to add a fresh dyno and getting this error for the first time

settings are:
Stack: heroku-16
node: 8.9.0
npm: 5.5.1

In ubuntu Server, there is not chrome instead of chromium. Is it the same way to set the chromium path to Chrome_Path environment variable?

Hi and first thanks for NickJs and PhantomBuster
This is an Help Me Please issue.
Im'not very familiar with async/await function, so i'm practicing with this.

My question is:
How can i grab a bunch of urls (Like my example code)
And loop through an array to open new tab for each url, take a screenshot, close the tab, close nick.
I'm a user of Phantombuster too (in experimentation in my company) and i'm really really stuck with this.
Anyone to help me ?

import 'babel-polyfill'
import Nick from 'nickjs'
const nick = new Nick()
nick.newTab().then(async function (tab) {
  await tab.open('https://news.ycombinator.com/')
  await tab.waitUntilVisible('#hnmain')
  await tab.inject('https://code.jquery.com/jquery-3.1.1.slim.min.js')
  const urls = tab.evaluate((arg, callback) => {
    const data = []
    $('.athing').each((index, element) => {
      data.push($(element).find('.storylink').attr('href'))
    })
    callback(null, data)
  })
  return urls
}).then((urls) => {
  for (var i = 0; i < urls.length; i++) {
    // here i want open a new tab for each url i have in my urls array
    // And i want to perform a screenshot or a evaluate function
  }
//and after that i want to quit nick
  nick.exit()
})
  .catch((err) => {
    console.log('Oops, an error occurred: ' + err)
    nick.exit(1)
  })

Best Regards

Hi !

I begin using your lib and I'm wondering if and how I could use proxy to connect to the web ?

thanks for your work

Hugo

First off, Nick is amazing. Thank you.

I tried upgrading to .0.3.6 (from .0.3.0) get around the Chrome subprocess killed by signal SIGTERM issues.

Now I get a Error: timeout: load event did not fire after 10002ms on my code. Any ideas? The HN example still works so I'm not sure what's wrong with my code.

const Nick = require("nickjs");
const nick = new Nick();

(async () => {
  const tab = await nick.newTab();

  let urls = ["https://www.lawfwfewef.com", "https://www.awfwaffe.com"];

  let outsideresults = [];

  for (let url of urls) {
    const insideresults = [];

    await tab.open(url);

    await tab.wait(9000);

    await tab.untilVisible("html"); // Make sure we have loaded the page
  } // for of loop
})()
  .then(() => {
    console.log("Job done!");
    nick.exit();
  })
  .catch(err => {
    console.log(`Something went wrong: ${err}`);
    nick.exit(1);
  });

I'm using NickJs browser emulation and am attempting to add a header when opening a page. I'm monitoring the request in Fiddler and see that the header is not being added.

How do I add the header?

Here's my current code:

await tab.open("https://www.google.com/", {headers: {"Accept-Language": "en-US,en;q=0.9"}})

With Node 6.11.3 LTS and Node 8.5.0
Chrome Version 61.0.3163.91 (Official Build) (64-bit)

Using sample on https://nickjs.org/
It always say: "Error: Could not start chrome: Error: spawn google-chrome-unstable ENOENT"

Same environment, I run puppeteer normally

const puppeteer = require('puppeteer');
(async() => {
    const browser = await puppeteer.launch();
    console.log(await browser.version());
    browser.close();
})();

Result is HeadlessChrome/62.0.3198.0

#What I've tried
*Reinstalling Chrome
*Modifying my installation of Nickjs to try enabling downloads
*Reinstalling a clean Nickjs installation

#Expected behavior
Downloadable files will be downloaded when navigated to

#Actual behavior
Downloadable files are not downloaded

Nickjs: 0.3.6
Chrome: 67.0.3396.62
Ubuntu: 16.04.4

I am trying to crawl google map and got following error.

2018-06-01 10:34 : CHROME STDERR: [0601/103455.634767:ERROR:gpu_process_transport_factory.cc(1017)] Lost UI shared context.
2018-06-01 10:34 : CHROME STDERR:

Best regards,

Hi dev team,
It seems the headless chrome remote interface can make full page screenshots, by setting the right resolution in Emulation.setVisibleSize

I wasn't able to find such option in Nickjs, and I am getting the half of the screenshot:

What would you recommend?

How is scaling supposed to work with nickjs?
Does instantiating a new nickjs starts a new session (suppose you want to keep tabs independent from each other)

I am making web based api with express4 and pm2.

When I call nick.exit(), my api auto-restarts because it is calling exit() function.

So, is there any way that I can finish using nickjs? like not auto-restarting the api?

Hello there,

Possibly, this is bug on headless chrome, but, I want to confirm if this is a bug.

When I crawl like following on google map, the result is less than I use just chrome browser.

1. Open this URL.
   https://www.google.co.jp/maps/@35.6603142,139.7055783,21z

2. sendkey like shibuya web consultant

The result of google chrome has more than 20 search results.

On the other hand, the result of chrome headless browser has aorund 8 search results.

Do you have the same result?

My environment is as following.

ubuntu 18.04
Google Chrome 67.0.3396.99
NickJs the latest one
Regards,

Is there a way to log/view the client request headers and the server response headers?

This error occurs after upgrading to latest version of Chrome:

error when initializing new chrome tab: Error: 'Network.setRequestInterceptionEnabled' wasn't found

Confirmed with these release channels of Chrome:

• Release (Version 64.0.3282.119 (Official Build) (64-bit))
• Beta (Version 64.0.3282.119 (Official Build) (64-bit))
• Dev (Version 65.0.3325.18 (Official Build) dev (32-bit))

Environment:

• Windows 10
• NickJS 0.2.6
• Node v8.9.1

The code:

Nick = require 'nickjs'
nick = new Nick()

nick.newTab().then (tab) =>
    await console.log "Testing..."
.then () =>
    console.log 'Completed with no errors!'
    nick.exit 0
.catch (err) =>
    console.log 'Something went wrong:'
    console.log err
    nick.exit 1

I've run heroku buildpacks:add https://github.com/heroku/heroku-buildpack-google-chrome and rooting around on the forums but not having much luck.

Seems like I'm heading in the right direction:

Hello guys!

You've been doing a great job with NickJS library!

But working with NickJs on NodeJs I had the following server error: "Chrome subprocess killed by signal SIGTERM". In fact, Chrome is killed by SIGTERM kernel process while NickJs tabs are evaluating.

It seems the problem occurs when code open/close tabs for long time, in particular when pages have a lot of content and scripts.
Test script is written to reflect the problem on the real project, where pages also run simultaneously in different tabs for long time. During the work CPU and Memory consumption in Task Manager looks fine, so killing the Chrome process is very unexpected.

My environment: NodeJS v8.8.1, NickJs + Desktop Chrome 63.0.3239.108 (Official Build) (64-bit).
Additional packages: Bluebird - for advanced Promise usage.
OS: Windows 8 (But the same error occurs on Ubuntu 16.04)

Here is my test file 'pureNickTest.js':

const Promise = require('bluebird');
const nick = new NickJs();
const testUrl = "http://edition.cnn.com/2017/12/15/politics/next-alabama-states/index.html";

const runTab = async (url) => {
    const tab = await nick.newTab();
    await tab.open(url);
    await tab.untilVisible("body");
    await tab.inject(`${__dirname}/libs/jquery-3.2.1.min.js`);
    await tab.wait(3000);
    await tab.scrollToBottom();

    const results = await tab.evaluate((arg, callback)=>{
        const blocks = $("body").find('div');
        callback(null, blocks.length);
    });

    await tab.close();
};

const runTestTabs = () => {
    const fakeArray = new Array(100);

    return Promise.each(fakeArray, () => {
        return runTab(testUrl);
    });
}

(async ()=> {
    await runTestTabs();
    console.log("Test complete!");
    nick.exit();
})();

Equivalent code for puppeteer works fine and doesn't crush with this error.

I would appreciate any help on this issue!

how solvethis error :
when run with thiscommand

./node_modules/casperjs/bin/casperjs lib/myNewBot.js

How to run on GCF? getting this error -
could not start chrome: Error: spawn google-chrome-beta ENOENT

Environemnt:
Ubuntu 16.04
Nickjs ^0.3.6
Nodejs v8.11.1

Hello there,
I am getting this even I use the tab.close().
Do you know why I am getting more chrome process?
Should I use nick.exit() after the execution if I want to kill all chrome process?

Regards,

As the following code, last console.log should be printed but the app would be forced exit as nick.exit() called. By remove nick.exit() line, console.log print the result but the app would not end as it should be. (I will have to Ctrl-c to terminate)

const nick = new Nick()
let tab = await nick.newTab()
await tab.open(url)
await tab.untilVisible('.tbl')
let content = await tab.getContent()

nick.exit()

console.log('This line should be printed with content', content)

I tried running NickJs in node9-alpine image and I get this error:

CHROME STDERR: [0204/101402.974481:WARNING:dns_config_service_posix.cc(326)] Failed to read DnsConfig.
CHROME STDERR: [0204/101402.984254:ERROR:devtools_http_handler.cc(759)] Error writing DevTools active port to file
> It took 105ms to start and connect to Chrome (1 tries)
{
  "level": "error",
  "message": "cannot connect to chrome tab: Error: Unknown command: protocol"
}

I have added the chromium browser to the image and set the CHROME_PATH to the chromium-browser. It is also present there, I checked with running a bash in the container.
Does anyone know what to do?

My Dockerfile:

FROM node:9-alpine

ENV CHROME_PATH=/usr/bin/chromium-browser
ENV NICKJS_NO_SANDBOX=1
# ╒═════════════════════════
# │ Copy from builder
# ╘═════════════════════════
RUN mkdir -p /app
WORKDIR /app

# Node modules
COPY --from=builder /app-prod/node_modules ./node_modules

# App files
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/res ./res

COPY --from=builder /app/package.json .
COPY --from=builder /app/util/healthcheck.js .
COPY --from=builder /app/start.sh .

RUN rm -f /app/dist/source.js /app/dist/source.js.map

# ╒═════════════════════════
# │ INSTALL NATIVE
# ╘═════════════════════════
# Install native app dependencies
RUN set -ex; \
  apk add --no-cache \
    rsync \
    openssh \
    udev \
    ttf-freefont \
    chromium

# ╒═════════════════════════
# │ SETUP HEALTH CHECK
# ╘═════════════════════════
HEALTHCHECK --interval=12s --timeout=12s --retries=3 --start-period=60s \
 CMD node ./util/healthcheck.js

# Run the container under "node" user by default
#USER node
CMD [ "npm", "run", "serve" ]

Does the evaluate function include all the execution context in the page? Like the elements in an iframe

https://stackoverflow.com/questions/48776320/iframes-document-selectors-return-null-in-devtools-console-but-then-return-ele

Hi,
thank you for your time in advance.

I am having login issue with nickjs.

I took the cookies from my browser and I set it up before opening the URL.

This is my code. As you can see from the screenshot, it fills the email and the password (the "name" attributes are the same with the input's keys) but it stops there (you can see my shell console output bellow).

I tried to navigate to a url which requires log-in, but it gives me the login form again, which means the login process wasn't successfull.

Would you help me to solve this issue? I would appreciate your help.

Regards,
Manol.

`
const selector = "form[name='signIn']";
const inputs = {
'email': "real email",
'password': "real password"
};

try {
  await tab.waitUntilVisible(selector, 5000)
  await tab.fill(selector, inputs, { submit: true })
  //await tab.click("input#signInSubmit")
  console.log("Form sent!")
  await tab.screenshot('./login.png')

} catch (err) {
  console.log("Form not found:", err)
}`

`root@06131ed62a3b:/app# node index.js

Fatal: Chrome subprocess exited with code 1

Tab 1: Navigation (open): https://www.amazon.com/ap/signin?_encoding=UTF8&accountStatusPolicy=P1&openid.assoc_handle=usflex&openid.claimed_id=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.ns.pape=http%3A%2F%2Fspecs.openid.net%2Fextensions%2Fpape%2F1.0&openid.pape.max_auth_age=0&openid.return_to=https%3A%2F%2Fwww.amazon.com%2Fgp%2Fcss%2Forder-history%3Fie%3DUTF8%26ref_%3Dnav_youraccount_orders&pageId=webcs-yourorder&showRmrMe=1
Tab 1: Navigation (formSubmissionPost): https://www.amazon.com/ap/signin
Form sent!
Job done!
`

Google Chrome: 66.0.3359.117
Nickjs: 0.3.6

Error:

2018-05-15 11:17 : CHROME STDERR: [0515/111735.657188:ERROR:gles2_cmd_decoder.cc(3350)] ContextResult::kFatalFailure: fail_if_major_perf_caveat + swiftshader
2018-05-15 11:17 : CHROME STDERR: [0515/111735.660133:ERROR:gles2_cmd_decoder.cc(3350)] ContextResult::kFatalFailure: fail_if_major_perf_caveat + swiftshader
2018-05-15 11:17 : CHROME STDERR: [0515/111745.945705:ERROR:gles2_cmd_decoder.cc(3350)] ContextResult::kFatalFailure: fail_if_major_perf_caveat + swiftshader
2018-05-15 11:17 : CHROME STDERR: [0515/111745.948669:ERROR:gles2_cmd_decoder.cc(3350)] ContextResult::kFatalFailure: fail_if_major_perf_caveat + swiftshader
2018-05-15 11:17 : CHROME STDERR: [0515/111748.365746:WARNING:spdy_session.cc(2991)] Received HEADERS for invalid stream 767

Do you know how to solve this issue?

My NickJS script was working fine a few weeks ago. I don't think I changed anything, but it stopped working. I tried updating from NickJS 2.0 to 2.3 and Chrome dev to the latest version with no improvement.

Apparently it is hanging at the call to tab.open(url). I also tried running the sample code at https://nickjs.org/, and it hangs at the same method. How do I figure out why NickJS is hanging there? (The PhantomBuster demo still works, so I think it must be something with my environment)

Environment:

• Windows 10 64-bit
• Node: v8.4.0
• NickJS 2.0, 2.3
• Chrome: 63.0.3236.0 (developer stream)
• Executing from Git bash 2.12.0.windows.1

Simple code to reproduce (CoffeeScript):

Note the callback is never called.

Nick = require 'nickjs'
nick = new Nick()

callback = (err, httpCode, httpStatus, url) =>
    console.log 'Callback:'
    console.log err

( () =>
    tab = await nick.newTab()
    console.log 'tab.open()'
    tab.open 'https://mixergy.com/interviews/', callback
)()

Simple code to reproduce (JS):

Note the callback is never called.

var Nick, callback, nick;

Nick = require('nickjs');

nick = new Nick();

callback = (err, httpCode, httpStatus, url) => {
  console.log('Callback:');
  return console.log(err);
};

(async() => {
  var tab;
  tab = (await nick.newTab());
  console.log('tab.open()');
  return tab.open('https://mixergy.com/interviews/', callback);
})();

Hi.
Do you have an image to dockerize nickjs (latest version with chrome) ?
I tried to dockerize my app with a very simple dockerfile with the latest node image

FROM node:latest
ADD ./app /app
WORKDIR /app
ENV CHROME_PATH=/usr/bin/google-chrome
RUN apt-get update && install google-chrome-stable
RUN npm install
CMD npm start

My app it's your example with ycombinator and nothing more.
The google-chrome-stable version is 62 (in accordance with your documentation).

The image build is ok, chrome's installation is ok path is ok (Check with which google-crome and echo "$CHROME_PATH").
But when i run a new container or when i run node index.js to launch my node script inside a new container i got this error:

Fatal: Chrome subprocess exited with code 1

And i can't figured out what is my problem.

Can nick run inside an existing running electron instance and use the api to control it?
If so, how would the configuration look like ?

Is there a way to recover or instantiate a tab again when it crashed? Whats the value of the tab when it crashed?

Hello there,

I would like to change userAgent after initialization of Ncickjs.

What I found is following.

tab = await nick.newTab();
tab.driver.client.Network.setUserAgentOverride({"userAgent":userAgent});

Is this correct way to do?

regards,

What would be the best way to set a different proxy and user-agent on each tab open, testing this out currently but not sure if this is the right approach:

const nick = new Nick({
  printNavigation: true,
  printResourceErrors: true,
  printPageErrors: false,
  resourceTimeout: 10000,
})

// start loop logic

;(async () => {
        nick.options.userAgent = randomUserAgent;
        nick.options.httpProxy = randomProxy;

        const tab = await nick.newTab()
        await tab.open(url)

        // Clean up
        await nick.deleteAllCookies()
        await tab.close()
})()

// end loop logic

Initially I had const nick = new Nick inside my loop logic and that crashed node after 100 tab opens, pretty sure why :)

Hi guys,

You made a great tool, but I'm running in an issue.

when I try the sample code in node.js when the code is reaching nick.exit(0); the whole node process is killed I was making some tests with postman and as a result the response from the server is always : "Could not get any response" but in the node logs I have :
"server started on: 3000"
"Tab 1: Navigation (open): http://:www.google.com"
"Process finished with exit code 0"
and after the node server is not responding and is killed.

I tried also on a remote server the node server is restarting but every time the process is killed.

For now I found a workaround "tab.close()" but it looks like the chrome is staying there in background and every time a new Nick is created a new chrome is launched.

Can you give me some advice about this or is it a bug ?

How do i simulate a Enter event, like pressing enter after sending keys in a textarea?

I want to use casper + phantom, not headless chrome.

Then, I have following source code.

require('babel-polyfill')
const Nick = require('nickjs')
const Promise = require('bluebird')

const nick = new Nick()

nick.newTab().then(async function(tab) {
  await tab.open("news.ycombinator.com")
  await tab.untilVisible("#hnmain") // Make sure we have loaded the page
  await tab.inject("/home/vagrant/test/nickjs_casper/src/js/jquery-3.3.1.slim.min.js") // We're going to use jQuery to scrape                                                                                
  const hackerNewsLinks = await tab.evaluate((arg, callback) => {
    // Here we're in the page context. It's like being in your browser's inspector tool
    const data = []
    $(".athing").each((index, element) => {      data.push({
        title: $(element).find(".storylink").text(),
        url: $(element).find(".storylink").attr("href")
      })
    })
    callback(null, data)
  })
  console.log(JSON.stringify(hackerNewsLinks, null, 2))
})
.then(() => nick.exit())
.catch((err) => {
    console.log('Oops, an error occurred: ' + err)
    nick.exit(1)
})

But, I am getting the following error.
Oops, an error occurred: Error: in evaluated code (initial call): EvalError: Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "script-src 'self' 'unsafe-inline' https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/ https://cdnjs.cloudflare.com/".

Do you know why I am getting this error?

Tabs

• goBack
• goForward
• selectOption

i think there is a lot to be learned from cypress
https://www.cypress.io/

it just isn't made to do scrapping.