interactive shell vs real code about core HOT 5 CLOSED

@xciser77,
I didn't have that. But perhaps the site uses post JS render.

Try using ExecuteJavascriptMiddleware in your code:
https://roach-php.dev/docs/downloader-middleware/#executing-javascript

from core.

i am trying, do I only have to include use RoachPHP\Downloader\Middleware\ExecuteJavascriptMiddleware
in my spider or do I have to declare something in the downloader middleware also ?

from core.

Do you have an example repository illustrating this issue? Because the interactive shell uses the same mechanism to download a site's HTML as a spider does.

from core.

if got this link (https://www.douglas.nl/nl/p/5009960042) and I am trying to get the prizes.

`<?php

namespace App\Spiders;

use Generator;
use RoachPHP\Downloader\Middleware\RequestDeduplicationMiddleware;
use RoachPHP\Downloader\Middleware\ExecuteJavascriptMiddleware;
use RoachPHP\Extensions\LoggerExtension;
use RoachPHP\Extensions\StatsCollectorExtension;
use RoachPHP\Http\Response;
use RoachPHP\Spider\BasicSpider;
use RoachPHP\Spider\ParseResult;

class douglasnl extends BasicSpider
{
public array $startUrls = [
'https://www.douglas.nl/nl/p/5009960042'
];

public array $downloaderMiddleware = [
    RequestDeduplicationMiddleware::class,
];

public array $spiderMiddleware = [
    //
];

public array $itemProcessors = [
    //
];

public array $extensions = [
    LoggerExtension::class,
    StatsCollectorExtension::class,
];

public int $concurrency = 1;

public int $requestDelay = 2;

/**
 * @return Generator<ParseResult>
 */
public function parse(Response $response): Generator
{
  
   $product_id = $response->filterXpath('//link[@rel="canonical"]')->link();  
   $prizes =  $response->filterXpath('//div[@class="product-detail__variant-row product-detail__variant-row--spread-content"]')->eq(0);

   yield $this->item([
    'product_id' => $product_id,
    'prizes' => $prizes
  ]);
}

}
`

If I try the response filterXpath ($response->filterXpath('//div[@Class="product-detail__variant-row product-detail__variant-row--spread-content"]')->eq(0);) in the interactive shell, I get two results, in my laravel app I get an error ( 0 => "Line 175, Col 44974: No match in entity table for 'Gabbana'")

Roach::startSpider(douglasnl::class); $items = Roach::collectSpider(douglasnl::class);

from core.

So I've tested this locally and you actually get back the same result both times. The difference is that the REPL processes the raw HTML a little bit before showing the results.

The issue is that you're yielding the entire Crawler object in your spider instead of just the string contents of the node. So your parse method should look something like this instead:

/**
 * @return Generator<ParseResult>
 */
public function parse(Response $response): Generator
{
    $product_id = $response->filterXpath('//link[@rel="canonical"]')
        ->link()
        // Return the actual URI string instead of the `Link` object.
        ->getUri();

    $prizes =  $response
        ->filterXpath('//div[@class="product-detail__variant-row product-detail__variant-row--spread-content"]')
        ->eq(0)
         // Return the actual text contents of the node instead of the entire
         // `Crawler` object. 
        ->text();

    yield $this->item([
        'product_id' => $product_id,
        'prizes' => $prizes
   ]);
}

Another thing is that you should call either Roach::startSpider or Roach::collectSpider but not both since that would actually cause the spider to run twice. Roach::collectSpider already starts the spider.

from core.