hlp-ai / mt-data Goto Github PK

Given a set of domains or web sites, crawl pages in them.

1. Multi-thread
2. Only crawl pages in given domains
3. Keep polite to web sites
4. Store crawled pages

For sentences in a Wet site, align them based on cross-lingual sentence embeddings, e.g., LaBSE. So, the basic steps are as follows:

1. For pages in a Web site, segment text into sentences;
2. Convert sentences into dense vectors using LaBSE;
3. Find the most similar senteces. i.e., parallel sentences, based on embedding vectors.

1. Parse text from HTML code of Web page;
2. Split text into sentences.

Align web pages based on URL patterns.