-
It can extract main information about the website including title, description, media link.
-
It can find all the bait products by surfing all the pages recursively.
-
It can extract product information including title, description, price, sku, all media link
- BASE_URL :
Base API URL - PORT :
Port number - OPENAI_API_KEY :
Custom OpenAI API key - CHROMIUM :
Path where chromium is installed. - TIME_OUT :
API response timeout
-
npm run start -
dev mode:
npm run dev
-
Docker build:
docker-compose build -
Docker run process:
docker-compose up -d
-
Use puppeteer to fetch all content from the webpage and visit all subpages.
-
Use ChatGPT APIs, extract information from the content.
-
After getting all pages, filter product page.
-
Get detailed information from the product page
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent