• The getting started guide doesn't mention how to enqueue requests manually, I found it somewhere in the SDK docs. Maybe giving an example of enqueueing requests manually in the getting started guide would be helpful for people starting out with writing a web-scraper.
• Again about the context.enqueueRequest() function I was not sure if I can use it asynchronously or not? for example, is it ok to pass an async function to each():

subCategoriesSelector.each(async function() {
            // Suffix sub category link with a limit of 96
            var scLink = $(this).attr('href').replace('#', '?limit=96');
            //log.info(`SubCategory Link: ${scLink}`);
            await context.enqueueRequest({
                url:`${baseURL}/${scLink}`,
                userData: {
                    "label": "LIST"
                    }
                }
            );

For example Network iddle is useful for scraping of AJAX websites.

https://apify.com/apify/cheerio-scraper - link to https://sdk.apify.com/docs/api/autoscaledpool
https://apify.com/apify/puppeteer-scraper - links to https://sdk.apify.com/docs/api/autoscaledpool and https://sdk.apify.com/docs/api/puppeteerpool

Many pages are not fully loaded with current configuration and network0 works for them. We should test this on some use cases to see if the websites won't timeout.

Added sections to web-scraper input schema ->

Link for example proxy and browser masking to https://help.apify.com/en/articles/1961361-several-tips-how-to-bypass-website-anti-scraping-protections

Before releasing this make sure that markdown in section description is supported at Apify app.

For consistency with other actor projects...

If I make a task for this actor and hit the run button, I got an error message that Start URLs is not defined even there are two. Then I also have to set Max pages per crawl.

Make it consistent with CheerioCrawler documentation

With a comment describing what is it good for. This will help users to quickly learn about this cool feature.

Following the instructions to run locally returns the following error:

Error: Evaluation failed: EvalError: Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "script-src 'self' 'unsafe-inline' https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/".


    at <anonymous>:6:43
    at ExecutionContext.evaluateHandle (act-crawler/node_modules/puppeteer/lib/ExecutionContext.js:66:13)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)

Here's the RESULTS-1.json:

[
  {
    "id": 1,
    "label": "",
    "url": "https://news.ycombinator.com",
    "loadedUrl": "https://news.ycombinator.com/",
    "uniqueKey": "https://news.ycombinator.com",
    "referrerId": null,
    "requestedAt": "2017-12-28T09:34:17.387Z",
    "loadingStartedAt": "2017-12-28T09:34:18.609Z",
    "loadingFinishedAt": "2017-12-28T09:34:18.627Z",
    "responseStatus": 200,
    "responseHeaders": {
      "date": "Thu, 28 Dec 2017 09:34:18 GMT",
      "content-encoding": "gzip",
      "referrer-policy": "origin",
      "server": "cloudflare-nginx",
      "x-frame-options": "DENY",
      "content-type": "text/html; charset=utf-8",
      "status": "200",
      "x-xss-protection": "1; mode=block",
      "cache-control": "private; max-age=0",
      "content-security-policy": "default-src 'self'; script-src 'self' 'unsafe-inline' https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/; frame-src 'self' https://www.google.com/recaptcha/; style-src 'self' 'unsafe-inline'; img-src 'self' 'unsafe-inline'",
      "strict-transport-security": "max-age=31556900",
      "cf-ray": "3d4385df4ee52fcf-MAA",
      "vary": "Accept-Encoding",
      "x-content-type-options": "nosniff"
    },
    "pageFunctionStartedAt": null,
    "pageFunctionFinishedAt": null,
    "type": "START_URL",
    "depth": 0,
    "pageFunctionResult": null,
    "interceptRequestData": null,
    "downloadedBytes": 196571,
    "willLoad": true,
    "queuePosition": "LAST",
    "errorInfo": [
      "Error: Evaluation failed: EvalError: Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: \"script-src 'self' 'unsafe-inline' https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/\".\n\n\n    at <anonymous>:6:43\n    at ExecutionContext.evaluateHandle (/private/tmp/act-crawler/node_modules/puppeteer/lib/ExecutionContext.js:66:13)\n    at <anonymous>\n    at process._tickCallback (internal/process/next_tick.js:188:7)",
      "Error: Evaluation failed: EvalError: Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: \"script-src 'self' 'unsafe-inline' https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/\".\n\n\n    at <anonymous>:6:43\n    at ExecutionContext.evaluateHandle (/private/tmp/act-crawler/node_modules/puppeteer/lib/ExecutionContext.js:66:13)\n    at <anonymous>\n    at process._tickCallback (internal/process/next_tick.js:188:7)",
      "Error: Evaluation failed: EvalError: Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: \"script-src 'self' 'unsafe-inline' https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/\".\n\n\n    at <anonymous>:6:43\n    at ExecutionContext.evaluateHandle (/private/tmp/act-crawler/node_modules/puppeteer/lib/ExecutionContext.js:66:13)\n    at <anonymous>\n    at process._tickCallback (internal/process/next_tick.js:188:7)",
      "Error: Evaluation failed: EvalError: Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: \"script-src 'self' 'unsafe-inline' https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/\".\n\n\n    at <anonymous>:6:43\n    at ExecutionContext.evaluateHandle (/private/tmp/act-crawler/node_modules/puppeteer/lib/ExecutionContext.js:66:13)\n    at <anonymous>\n    at process._tickCallback (internal/process/next_tick.js:188:7)"
    ],
    "retryCount": 3,
    "skipOutput": false,
    "outputSeqNum": 1
  }
]

Hi,

Are there any chances we could get the crawler's webhook feature finishWebhookUrl implemented in this act?

Thanks

Facing issues with the HTTP_REFERER header for an apify/web-scraper Task, I tried to dump $_SERVER on a test page after a POST request enqueued as

context.enqueueRequest(
    {
        url: "https://example.org/print_post_request.php",
        payload: "fname=lol&lname=wut",
        headers: {
            "Referer": "http://example.org"
        },
           method: "POST"
    }
);

but the output stated REQUEST_METHOD => GET. Then I checked setting the StartURL method to POST and the output was the same.

I conclude that POST settings are completely ignored by apify/web-scraper.

If proxy IP doesn't work a user needs a way to retire the browser to obtain a new IP. This should wait for SessionPool to be added and this will simply invalidate the session.

The legacy Crawler had one use case that doesn't seem to work nicely in this actor. Let's say you wanted to check if some webpage can be easily scraped or not. So you just created a new crawler, set the Start URL to the URL you wanted to check, click Run and see a screenshot of the page. When you do the same in this actor, the live view doesn't even show up and the actor finishes straight away. Is there some way to make this case work?

Two users already had a use-case where they want to store persistent data. Right now they have to use raw API which is not user friendly.

Where is the documentation for how the INPUT_SCHEMA.json fields get render on the task UI?

Make it consistent with CheerioCrawler documentation

I was the puppetter scraper tutorial. https://docs.apify.com/scraping/puppeteer-scraper

I was testing injecting qjuery. When I removed the line:
await Apify.utils.puppeteer.injectJQuery(page); // <-------- Inject jQuery.

I am still getting all of the results in the dataset!

Why put such a strict limit there? I understand that Puppeteer can accumulate garbage and crash on memory over looong amount of time but that's nowhere near 360 seconds.

The copy in input schema needs improvements, many things are not clear, documentation is missing.

Why am i getting this error message with Puppeteer Scraper? I cannot find this error in the source code.

https://github.com/apifytech/actor-scraper
https://github.com/apifytech/apify-js
ACTOR: The actor run has reached the timeout of 300 seconds, aborting it. You can increase the timeout in run options.

Based on my testing, the PhantomJS actor on the apify platform is 8x more efficient than puppeteer actor. With the same configuration and 70 minutes of crawling, PhantomJS crawled 4830 pages, while puppeteer 604. Your customers could get more value by using PhantonJS.

https://my.apify.com/tasks/mDp69rvzdfC7LGk36#/runs/cycaJnAzPqzvmDq6q - PhantomJS
https://my.apify.com/tasks/h4RBR5Z72rnrwryPY#/runs/zPSyicXMJejz2rjX2 - puppeteer

The functions section of the readme contains info about jquery, but not about underscore
https://www.dropbox.com/s/izu694xzrxik5fz/Screenshot%202019-04-04%2010.39.42.png?dl=0

Since both have useSessionPool: true, and you have access to Apify inside, useful for using requestAsBrowser using the same session

The API URL to view the output log for a task doe not return the log.
https://api.apify.com/v2/key-value-stores/vDLfk985Hqw8SiXh3/records/OUTPUT?disableRedirect=1

Instead returns:

{
  "error": {
    "type": "record-not-found",
    "message": "Record was not found"
  }
}

I would like to have the option to receive an email notification when the task is scheduled, starts and ends. I would prefer the default email address to be my account email but have a field or additional addresses.

Since you already have the email actor, the code already seems to be there.

JSON input is not user-friendly. This needs to be implemented at the platform first and enabled in input schema.

This means that on a website which uses hashtags to control it's routing, web-scraper does not work. Example would be this website https://dagsordener.vejle.dk/ (I can provide a testing configured Task for this).

i am trying to understand the use case when you would want to not useRequestQueue. what value does this provide to the user?

Now it's only available in "Page function". Customers requested this (see https://app.intercom.com/a/apps/kod1r788/inbox/inbox/conversation/26871291243)

Default puppeteer page.waitFor function has parameter timeout in options, where you can set a timeout in ms.

We should have the same param or at least document it that we have a different one in the web-scraper readme.

Hi,

I am going through the documentation and on the below mentioned URL, you have written that window & document can be accessed. Now I don't want to inject jQuery as I can do all the stuff in Javascript only. But I am not able to access window/document

https://apify.com/apify/web-scraper#page-function

While document is returning object as below:

{
2021-01-25T13:54:00.587Z location: {
2021-01-25T13:54:00.587Z ancestorOrigins: {},
2021-01-25T13:54:00.588Z href: 'chrome-error://chromewebdata/',
2021-01-25T13:54:00.588Z origin: 'null',
2021-01-25T13:54:00.589Z protocol: 'chrome-error:',
2021-01-25T13:54:00.589Z host: 'chromewebdata',
2021-01-25T13:54:00.589Z hostname: 'chromewebdata',
2021-01-25T13:54:00.590Z port: '',
2021-01-25T13:54:00.590Z pathname: '/',
2021-01-25T13:54:00.591Z search: '',
2021-01-25T13:54:00.591Z hash: '',
2021-01-25T13:54:00.591Z assign: {},
2021-01-25T13:54:00.592Z reload: {},
2021-01-25T13:54:00.592Z replace: {},
2021-01-25T13:54:00.593Z toString: {}
2021-01-25T13:54:00.593Z }
2021-01-25T13:54:00.594Z }

While I was expecting the actual DOM object. Please help me out on this.

The current version 0.13.0 is almost 3 months old. It would be good to move to the latest version 1.0.0 which would be more stable.

Currently only a tags are supported. Support for divs with href attribute is needed.
Is this something which is a quick fix? If someone can point in the right direction, then maybe I can add the fix and send out a PR.
Thanks

Run log

Not sure why it did not work, the options seem to passed correctly.

When I remove the value of a Link selector ('a'), save the configuration and reload the page, the value is back again set to 'a'

I have not researched this further, but it seems that when an error is thrown or some other issue happens in the browser, proxy is not correctly closed, or it tries to reconnect or I don't know what and it causes the scraper to get into an infinite loop of restarting browsers. More info in SDK slack channel.

When I want to set the crawler to use a new IP for each request, no idea how to do it.

It would be great to have cookies persistency across runs.
We can get some inspiration from legacy phantomjs scraper cookies persistency option.

People are trying different workarounds to accomplish a simple thing.

Feedback from user: I see there is a link to the cheerio docs in the getting started guide, maybe a link from the Task > Input page as well could be useful.

Feedback from users:

• Don't know what is input in the documentation about, there is no input in the UI that it is referring to, solved with custom data
• Also I wasn't using the Pseudo URLs box but I wasn't sure if I should uncheck the 'Use Request Queue' checkbox or not in that case as I was adding to the request queue in the code.

apify/crawlee#789

Lately some of the integration tests were failing. We need to investigate why and fix the problem.

We should probably add some option for the requestListSources input schema editor to hide the HTTP method since only GET is supported.

It puts everything to a default one, setValue is passed together with the store name.

anchor links like not jumping to section

https://www.tripadvisor.com.au/Restaurant_Review-g255060-d12544090-Reviews-The_Meat_Wine_Co-Sydney_New_South_Wales.html

https://my.apify.com/view/runs/czTCEblwfeqrA5r33