am9zzwy / tuer Goto Github PK

Adjust tokenizing, because as of now we still have some shitty words lol.
We probably remove important punctuations and characters that might be usefull. They also might skew the words into something they are not.

Good way of testing it is using example text and test the tokenizer.

Maybe switch to lemmatisation instead of tokenization

• Web Crawling
• Text-Representation
• Indexing
• DuckDB Integration
• Statistical Ranking
• L2R
• Link analysis
• Evaluation

Content:

• Title
• Link
• HTML-Body

• Easy Synonym Expansion: synonyms from thesaurus and WordNet (synonyms, hypernyms, hyponyms)
• Heavy Semantic Expansion: word2vec embedding for words in our corpus to check cosine similarity with query words (closest match)
• Connect with Pipeline

• read in query.txt
• generate result.txt

QUERY:

WARNING: the posted code examples below and in the jupyter notebook include wrong formatting for TABS! Make sure that we separate number of query from query text via \t instead of for query terms.

The batch file will be formatted to have one query per line, listing the query number, and query text as tab-separated entries.
An example of the batch file for the first two queries looks like this:

1   tübingen attractions
2   food and drinks

RESULT:

The text file should be formatted to produce one ranked result per line, listing the query number, rank position, document URL and relevance score as tab-separated entries. An example of the first three lines of such a text file looks like this:

1   1   https://www.tuebingen.de/en/3521.html   0.725
1   2   https://www.komoot.com/guide/355570/castles-in-tuebingen-district   0.671
1   3   https://www.unimuseum.uni-tuebingen.de/en/museum-at-hohentuebingen-castle   0.529
...
1   100 https://www.tuebingen.de/en/3536.html   0.178
2   1   https://www.tuebingen.de/en/3773.html   0.956
2   2   https://www.tuebingen.de/en/4456.html   0.797
...

• Ignore hidden stuff (hidden tags)
• Text in link should not be considered

(especially for @PhictionalOne)

Basic operations:

• remove whitespaces
• remove stopwords (check if there are different granularity levels)