A Spider Crawler Extension for Pimcore Dynamic Search.
- Pimcore >= 5.8.0
- Pimcore Dynamic Search
dynamic_search:
context:
default:
data_provider:
service: 'web_crawler'
options:
always:
own_host_only: true
full_dispatch:
seed: 'http://your-domain.test'
valid_links:
- '@^http://your-domain.test.*@i'
user_invalid_links:
- '@^http://your-domain.test\/members.*@i'
single_dispatch:
host: 'http://your-domain.test.test'
normalizer:
service: 'web_crawler_localized_resource_normalizer'
Name | Default Value | Description |
---|---|---|
own_host_only |
false | |
allow_subdomains |
false | |
allow_query_in_url |
false | |
allow_hash_in_url |
false | |
allowed_mime_types |
['text/html', 'application/pdf'] | |
allowed_schemes |
['http'] | |
content_max_size |
0 |
Name | Default Value | Description |
---|---|---|
seed |
null | |
valid_links |
[] | |
user_invalid_links |
[] | |
max_link_depth |
15 | |
max_crawl_limit |
0 |
Name | Default Value | Description |
---|---|---|
host |
null |
Identifier: web_crawler_default_resource_normalizer
Normalize simple documents
Options: none
Identifier: web_crawler_localized_resource_normalizer
Scaffold localized documents
Options:
Name | Default Value | Allowed Type | Description |
---|---|---|---|
locales |
all pimcore enabled languages | array | |
skip_not_localized_documents |
true | bool | if false, an exception rises if a document/object has no valid locale |
Identifier: http_response_html_scaffolder
Simple object scaffolder.
Supported types: VDB\Spider\Resource
with content-type text/html
.
Identifier: http_response_pdf_scaffolder
Simple object scaffolder.
Supported types: VDB\Spider\Resource
with content-type application/pdf
.
Identifier: pimcore_element_scaffolder
Simple object scaffolder.
Supported types: Asset
, Document
, DataObject\Concrete
.
Identifier: resource_uri_extractor
Supported Scaffolder: http_response_html_scaffolder
, http_response_pdf_scaffolder
Return Type: string|null
Options: none
Identifier: resource_language_extractor
Supported Scaffolder: http_response_html_scaffolder
, http_response_pdf_scaffolder
Return Type: string|null
Options: none
Identifier: resource_meta_extractor
Supported Scaffolder: http_response_html_scaffolder
Return Type: string|null
Options: none
Identifier: resource_html_tag_content_extractor
Supported Scaffolder: http_response_html_scaffolder
Return Type: string|null
Options: none
Identifier: resource_text_extractor
Supported Scaffolder: http_response_html_scaffolder
, http_response_pdf_scaffolder
Return Type: string|null
Options: none
Identifier: resource_title_extractor
Supported Scaffolder: http_response_html_scaffolder
, http_response_pdf_scaffolder
Return Type: string|null
Options: none