hundredblocks / concrete_nlp_tutorial Goto Github PK

I am having trouble setting up the environment and i think many are facing similar problems like #2 .If we can have a requirements.txt or environment.yml file,please add in the repo.It would be very helpful

import keras
import nltk
import pandas as pd
import numpy as np
import re
import codecs

input_file = codecs.open("/home/vikash/gen/social_media-disaster-tweets-DFE.csv", 'r'
, encoding='utf-8', errors='replace')
print input_file
output_file = open("social_media_relevant_cols_clean.csv", "w")

def sanitize_characters(raw, clean):
print("enter the function")
for line in input_file:
output_file.write(line)

sanitize_characters(input_file, output_file)
questions = pd.read_csv("social_media_relevant_cols_clean.csv")
questions.columns = ('text', 'choose_one', 'choose_label')
questions.head()
questions.tail()

The following is the error

Traceback (most recent call last):
File "/home/vikash/gen/concept.py", line 22, in
questions.columns = ('text', 'choose_one', 'choose_label')
File "/home/vikash/.local/lib/python2.7/site-packages/pandas/core/generic.py", line 4385, in setattr
return object.setattr(self, name, value)
File "pandas/_libs/properties.pyx", line 69, in pandas._libs.properties.AxisProperty.set
File "/home/vikash/.local/lib/python2.7/site-packages/pandas/core/generic.py", line 645, in _set_axis
self._data.set_axis(axis, labels)
File "/home/vikash/.local/lib/python2.7/site-packages/pandas/core/internals.py", line 3323, in set_axis
'values have {new} elements'.format(old=old_len, new=new_len))
ValueError: Length mismatch: Expected axis has 13 elements, new values have 3 elements

I am new to ML and NLP do we need to add the Column class_label.
I have downloaded from this link https://www.figure-eight.com/data-for-everyone/ pertaining to Disasters on social media as in your blog do help. Thanks in advance.

Hi there,

This is there error that comes up when trying to run the code. Any suggestions?
I'm currently running python 3.7.4.

Thanks,
Stef

def plot_important_words(top_scores, top_words, bottom_scores, bottom_words, name):
y_pos = np.arange(len(top_words))
top_pairs = [(a,b) for a,b in zip(top_words, top_scores)]
top_pairs = sorted(top_pairs, key=lambda x: x[1])

bottom_pairs = [(a,b) for a,b in zip(bottom_words, bottom_scores)]
bottom_pairs = sorted(bottom_pairs, key=lambda x: x[1], reverse=True)

top_words = [a[0] for a in top_pairs]
top_scores = [a[1] for a in top_pairs]

bottom_words = [a[0] for a in bottom_pairs]
bottom_scores = [a[1] for a in bottom_pairs]

fig = plt.figure(figsize=(10, 10))  

plt.subplot(121)
plt.barh(y_pos,bottom_scores, align='center', alpha=0.5)
plt.title('Irrelevant', fontsize=20)
plt.yticks(y_pos, bottom_words, fontsize=14)
plt.suptitle('Key words', fontsize=16)
plt.xlabel('Importance', fontsize=20)

plt.subplot(122)
plt.barh(y_pos,top_scores, align='center', alpha=0.5)
plt.title('Disaster', fontsize=20)
plt.yticks(y_pos, top_words, fontsize=14)
plt.suptitle(name, fontsize=16)
plt.xlabel('Importance', fontsize=20)

plt.subplots_adjust(wspace=0.8)
plt.show()

top_scores = [a[0] for a in importance[1]['tops']]
top_words = [a[1] for a in importance[1]['tops']]
bottom_scores = [a[0] for a in importance[1]['bottom']]
bottom_words = [a[1] for a in importance[1]['bottom']]

plot_important_words(top_scores, top_words, bottom_scores, bottom_words, "Most important words for relevance")

Following is the erro
KeyError Traceback (most recent call last)
in ()
33
34
---> 35 bottom_scores =[a[1] for a in importance[1]['bottom']]
36 bottom_words = [a[0] for a in importance[1]['bottom']]
37 top_scores =[a[1] for a in importance[1]['tops']]

KeyError: 1

When running questions = pd.read_csv("socialmedia_relevant_cols.csv")

It has been replaced by Concatenate, so the code will not run as is.

Hi,

Can you please provide the requirements.txt for the versions of python and all libraries?

Thanks.

On writing the cleaned file -
output_file = open("socialmedia_relevant_cols_clean.csv", "w")

It gave error -
UnicodeEncodeError: 'charmap' codec can't encode characters in position 64-65: character maps to

@hundredblocks

ValueError: Classification metrics can't handle a mix of continuous and multiclass targets