niderhoff / nlp-datasets Goto Github PK

View Code? Open in Web Editor NEW

5.7K 5.7K 955.0 195 KB

Alphabetical list of free/public domain datasets with text data for use in Natural Language Processing (NLP)

nlp-datasets's Introduction

👋 Hi, I’m Nicolas Iderhoff and I work as a Solution Architect for IT Systems and Software with AI at its core
👀 I’m interested in Data Management, LLMs, Automation, and DevOps.
🌱 I currently oversee the creation of a LLM-based Document Search for enterprises.
💞️ I’m looking to collaborate on all things regarding LLMs and RAG.
📫 How to reach me: [email protected]

nlp-datasets's People

Contributors

Stargazers

Watchers

Forkers

ligadous chiranjibsuruf caomw sagaruprety seominlee harris-chiu panyang chagge yzy5630 bensnw 7hacker yeswanthcheekati abhisheksawarkar naetherm sfailsthy yangqiokay rgaonkar machao-github nulledexceptions uscdatascience prhldk rudreshv stephensebastin bzheng6 squiba ltoscano wx1ng yezhejack fangzheng354 leezqcst harrytanme mengqhui alokranjan1234 lampts kmario23 dpappas single430 arhumishtiaq rkroll jayden11 sam-iitj vseledkin mydeeplearning kv19971 kyubyong dsksd kefirski stella-gao akki2825 locosoft1986 ahmedfadhil tuhinsharma121 backtrackbaba richard-mf clesleycode wanjizheng eric013 riomus zmq2017 roysh yugrocks priyasin kmr0877 petartodorov lawrenceliu lihaossu jamesbconner fatmas1982 onpoeet phpmind phillette trungkak joizhang2012 redeipirati fogside yhyu13 nahidalam clairett gedman4b bilash yumoxu rugezhao niladrimohanty sabirdvd sidbrahma tantrantp7 tanthml afrazhussain isr-wang fei090620 fromppf shubhampachori12110095 vishalbelsare miketam1021 rameshjes j-greer orangeplumber lvious vedic-partap shaypal5

nlp-datasets's Issues

Corporate messaging

Corporate messaging is pointing to common crawl.

Multiple Historical News Headlines Datasets

Some datasets you can review and add. (disclaimer: authored by me), Specifically the 4 regarding headlines and 1 regarding urban dictionary. Thanks!

https://www.kaggle.com/therohk/datasets

Missing metadata for Amazon Reviews dataset

Hello @niderhoff

Thank you so much for releasing these datasets. I downloaded the 'Amazon reviews' dataset. But I am unable to make any sense out of it. Its a mixture of review title and review text for all the products on amazon. There is no classification as such.
I was looking for dataset of reviews for mobile phones. And such particular reviews are all mixed up in the given dataset. Also, there is no labeling done as such.
Could you please any possible solution to these problems?

Thanks again

dataset with people's dialogues

Tell me, are there datasets with people's dialogues? I'm looking for it to simulate the model of people's communication using BIG DATA. For example, often people communicate so

 First_man(question)     Second_man (answer)

[1,] "how are you" "Fine, thank you, and how are you?"
Are there such data? I.E. data of everyday communication between people

Political Social Media data set now points to paid service Appen instead of Crowdflower

The political data set now points here instead of crowdflower.

https://appen.com/resources/datasets/

Del.icio.us dataset returns 404

hi, there,

the Del.icio.us dataset is no longer reachable.
the following links do not work.
214k data
1250k data

Twitter links broken

Geolocated links for twitter UK and USA broken

Gutenberg dataset

Hi,

https://github.com/pgcorpus/gutenberg is a very cool resource which contains scripts to download and preprocess the Gutenberg corpus. There is also an accompanying paper by @martingerlach and @fontclos https://arxiv.org/abs/1812.08092. I think it'd be nice to add this to the list.

Hate speech identification

Hate speech link broken. Correct reference is https://github.com/t-davidson/hate-speech-and-offensive-language (CF actually released a subset of our labelled data without notifying us, we released a larger set of labeled data that you can access using the link here).

You might want to include

Hi,
You have compiled an interesting list. Here is another interesting NLP datatset.
Including it would help value to your list.

https://nlp.stanford.edu/links/statnlp.html

entities: people, businesses, etc.

Please consider whether this resource would be good for your list. It a large collection of data about entities such as people, businesses, and organizations. It also includes code to ETL this data for use in NLP projects.

https://github.com/az0/entity-metadata/