Giter VIP home page Giter VIP logo

nlp-fall-2019-project-team2's Introduction

Duplicate Question Pair Detection in Natural Language Processing

By Raiful Hasan and Mohammad Aminul Hoque

Overview

Identifying the duplicate questions is challenging, the sentence composition and word selection vary among persons. There is a way to find the document similarity based on the degree of overlapping in classic natural language processing. However, in duplicate question detection, it not perform well, because most of the questions are too small and the degree of overlapping is insufficient. We have implemented an LSTM based architecture that will detect the duplicate question if they have the same intent.

Dataset

For this project, we have used the Quora question pair dataset. It contains publicly available Quora question. The dataset has been labeled manually by humans. Table 1 shows the sample data of the dataset.

id qid1 qid2 question1 question2 is_duplicate
25 53 54 What is web application? What is the web application framework? 0
139 279 280 What is the ideal life after retirement? What's life after retirement? 0
197 395 396 What are some must watch TV shows before you die? Are there any must watch TV shows? 1
221 443 444 What is my puk code? What's the PUK for TF64SIMC4? 1

Table 1: Sample data in Quora dataset

Here is the description of the field of the dataset -

Field Description
id unique id for each question pair
qid1 the id for question 1 in the pair
qid2 the id for question 2 in the pair
question1 the full text for question1
question2 the full text for question2
is_duplicate 1- if questions are duplicate or 0 โ€“ if questions are not duplicate

The attributes of the datasets are -

Attribute Values
Total number of entries 4,04,290
Total duplicate pairs 149302 (37%)
Training set 3,04,290
development set 50,000
Testing set 50,000

How to run

Results

Accuracy โ€“ 82.65%

nlp-fall-2019-project-team2's People

Contributors

raiful avatar riponcse10 avatar

Watchers

James Cloos avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.