Not extension in Google font css about node-website-scraper HOT 6 CLOSED

The important question is: how would you want to do type recognition, apart from checking the extension? I think it is hard to implement a solution which covers all scenarios. To do that you would have to combine the following techniques:

1. Recognize type based on extension.
2. Recognize type based on where it occurs (e.g.: link[rel="stylesheet"] => css).
3. Recognize type based on magic file bytes

I don't think it would be realistic to implement all that. It would make more sense if users would write a custom filenameGenerator to deal with their scenario.

from node-website-scraper.

Hi
Now scraper takes names from url. Url from example has no extension so filename is generated without extension. It is known behavior

Possible ways to solve it:

• Add option like addExtensions. If it is true scraper adds missing extensions for resources with known types (.html for html files and .css for css files)
• Get rid of sorting by extensions in subdirectories and sort by type, for example:

subdirectories: [
        {directory: 'css', types: ['css']}
],

• Extend existing sorting by extensions with types to provide ability for sorting by extension, by type or by both of them, for example:

subdirectories: [
        {directory: 'images', types: ['image'], extensions: ['.jpg', '.png']},    // resource should satisfy both ext and type
        {directory: 'css', types: ['css']}, // resource satisfy only type 
        {directory: 'scripts', extensions: ['.js']}, // rsource satisfy ony extensions
],

Way 2 and 3 requires additional types (image, script, font, etc)

@marklagendijk can I ask your advice for this please?

from node-website-scraper.

Thanks for the answer

subdirectories: [
        {directory: 'css', types: ['css']}
],

This code is not working. CSS is not stored in the CSS folder.
URL example: http://newtest.beauby.ru
My config:

subdirectories: [
        {directory: 'img', extensions: ['.jpg', '.png', '.svg', '.gif']},
        {directory: 'source', extensions: ['.jpg', '.png', '.svg', '.gif']},
        {directory: 'js', extensions: ['.js']},
        {directory: 'fonts', extensions: ['.ttf', '.woff', '.woff2', '.eot', '.otf']},
        {directory: 'css', types: ['css']}
      ],
      sources: [
        {selector: 'img', attr: 'src'},
        {selector: 'source', attr: 'srcset'},
        {selector: 'img', attr: 'srcset'},
        {selector: 'link[rel="stylesheet"]', attr: 'href'},
        {selector: 'script', attr: 'src'}
      ],

Better 3

Extend existing sorting by extensions with types to provide ability for sorting by extension, by type or by both of them

Вы говорите по русски? Вы я вижу вы из Киева, я тоже из из Киева. Проще на русском я думаю.
Вариант 3 самый лучший, а еще лучше вот так {directory: 'css', [types: ['css'], extensions: ['.css']]}

И еще есть моменты которые хотелось бы обсудить.

from node-website-scraper.

Hi again
Sorry, seems I didn't explain it properly. Nothing of suggestions above is working now. Module should be extended to support one of them.
I just wanted to ask guys @marklagendijk @aivus to look at possible solutions and discuss what is better.

I suggest continue using English on github in order to keep it clear for everyone

We can discuss everything you want on gittter in Russian and then just put summary here https://gitter.im/s0ph1e/node-website-scraper

from node-website-scraper.

Problem in distribution according to folders. Ok, let will be without expansion, but that types were distributed according to folders.

from node-website-scraper.

Adding extensions for html and css files is implemented in #59
Will be released in 1.0.0

from node-website-scraper.