riponcse10 / nlp-fall-2019-project-team2 Goto Github PK

By Raiful Hasan and Mohammad Aminul Hoque

• Overview
• Dataset
• How to run
• Results

Identifying the duplicate questions is challenging, the sentence composition and word selection vary among persons. There is a way to find the document similarity based on the degree of overlapping in classic natural language processing. However, in duplicate question detection, it not perform well, because most of the questions are too small and the degree of overlapping is insufficient. We have implemented an LSTM based architecture that will detect the duplicate question if they have the same intent.

For this project, we have used the Quora question pair dataset. It contains publicly available Quora question. The dataset has been labeled manually by humans. Table 1 shows the sample data of the dataset.

Table 1: Sample data in Quora dataset

Here is the description of the field of the dataset -

The attributes of the datasets are -

• Create a folder named "data" inside the current directory

• Download the pre-trained word vector from https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit and put inside the data folder

• Download train.csv and test.csv from https://l.facebook.com/l.php?u=https%3A%2F%2Fuab365-my.sharepoint.com%2F%3Af%3A%2Fg%2Fpersonal%2Fmahoque_uab_edu%2FEnn7NCg-eoZIl5y5oC4lR8kB5-tk9_pUiXULt-PV8ivKfQ%3Fe%3DjVcKYB%26fbclid%3DIwAR3We1Dd__4X9XAPo78GiAbWq8g6Au-ewAAN4iXrW67kNbh5xdli5qJSHdA&h=AT19KIqXOrBD56HvsOLfyiynNM2IJtwDlmRM8ffNImY7wSWot6MtzPbedeE0n-D9Zo7hJQDpVabvyhE9TDmvws3vlKFKRt9CIZUJdODOEfMq5T0Y6lozINknCErxI5drvigAGFJ4jXA and put the file inside the data folder

• Open Question-Similarity-Checker.ipynb in Jupyter notbook and run

Accuracy – 82.65%