davidglezz / tfg Goto Github PK

Prototype of a web application that indexes microdata (schema.org) exposed by electronic stores and allows the search of products.

License: MIT License

TypeScript 74.26% JavaScript 1.93% HTML 17.34% CSS 5.93% Shell 0.53%

json-ld rdfa microdata schema-org products search aggregator price-tracker crawler scraper

tfg's Introduction

Agregador de Productos y comparador de precios

Proyecto Fin de Grado de David González García que consiste en una aplicacion web que extrae e indexa productos a la venta de tiendas electrónicas mediante los datos estructurados que estas ponen a disposición.

Demostración

http://tfg.davidgagnon.org/

tfg's People

Contributors

Stargazers

Watchers

Forkers

ebarahona evo42

tfg's Issues

htmlmetaparser TypeError: Cannot read property 'text' of undefined

TypeError: Cannot read property 'text' of undefined
at Handler.onclosetag (D:\Clase\TFG\src\crawler\node_modules\htmlmetaparser\dist\index.js:445:9)

Fetch node as Chrome extensions for distributed crawl

https://developer.chrome.com/extensions/samples

Handle as / all non /api routes

Support for canonical urls

HTML tag: <link rel="canonical" href="http://www.example.com/" />
HTTP header: Link: <http://www.example.com/>; rel="canonical"

Update frontend to Angular 5

https://angular-update-guide.firebaseapp.com/

Allow to add shops

TaskUpdateUrls stops after long time.

After several hours of operation, the task stops working.

Support robots.txt

Support robots.txt to get the sitemap url and restrictions.

SitemapParser Error read ECONNRESET

Al parsear algunos sitemaps:

[ERROR] Error read ECONNRESET
Error: write after end
    at writeAfterEnd (_stream_writable.js:236:12)
    at PassThrough.Writable.write (_stream_writable.js:287:5)
    at Unzip.ondata (_stream_readable.js:640:20)
    at emitOne (events.js:115:13)
    at Unzip.emit (events.js:210:7)
    at Unzip.Readable.read (_stream_readable.js:476:10)
    at flow (_stream_readable.js:847:34)
    at resume_ (_stream_readable.js:829:3)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
Error: write after end
    at writeAfterEnd (_stream_writable.js:236:12)
    at PassThrough.Writable.write (_stream_writable.js:287:5)
    at Unzip.ondata (_stream_readable.js:640:20)
    at emitOne (events.js:115:13)
    at Unzip.emit (events.js:210:7)
    at Unzip.Readable.read (_stream_readable.js:476:10)
    at flow (_stream_readable.js:847:34)
    at resume_ (_stream_readable.js:829:3)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)

TypeError: Cannot read property 'text' of undefined
     at Handler.onclosetag (/app/crawler/node_modules/htmlmetaparser/src/index.ts:753:5)
     at Parser.onend (/app/crawler/node_modules/htmlparser2/lib/Parser.js:307:14)
     at Tokenizer._finish (/app/crawler/node_modules/htmlparser2/lib/Tokenizer.js:838:12)
     at Tokenizer.end (/app/crawler/node_modules/htmlparser2/lib/Tokenizer.js:829:25)
     at Parser.end (/app/crawler/node_modules/htmlparser2/lib/Parser.js:338:18)
     at Stream.<anonymous> (/app/crawler/node_modules/htmlparser2/lib/WritableStream.js:15:10)
     at Object.onceWrapper (events.js:314:30)
     at emitNone (events.js:105:13)
     at Stream.emit (events.js:207:7)
     at finishMaybe (_stream_writable.js:604:14)

QueryFailedError: Incorrect string value: '\xF4\x80\x80

Problem:
QueryFailedError: ER_TRUNCATED_WRONG_VALUE_FOR_FIELD: Incorrect string value: '\xF4\x80\x80\x80<2...' for column 'description' at row 1

Posible solution:
Change encoding from utf8 to utf8mb4

https://stackoverflow.com/questions/10957238/incorrect-string-value-when-trying-to-insert-utf-8-into-mysql-via-jdbc

Ignore SSL errors

SSL validation errors:
[ERROR] Error certificate has expired

URL when characters with accents appear

Problem with accents:

%25C3 stretches to:

%2525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525C3