simonseo / nyuad-spammer Goto Github PK

I want to use json2csv programmatically within the flask server. How do I do this?

Ideally there will be a method that takes a 2 arguments: either of [json string/python dict/json filename/json file] and an output filename. It will save any of these to a CSV file of the given output filename.

Maybe we can just write a few lines of code that does this. I think the json2csv library is unnecessarily big for our usage.

:(

Scrapper stops when attempting to create a new file because OS does not want to fork()

Currently, the "message" field of our CSV contains HTML-formatted content. We shouldn't pass this straight to the chatbot. We want the users to see plaintext with buttons instead of links.

To reduce time for setting up in the future, I made a wiki article outlining the initial steps to run everything.

Fill in the blanks when you have time please!!!

https://github.com/simonseo/nyuad-spammer/wiki/Setting-up-repository

It should also run locally

How does Telegram identify each user? do they have a unique ID that the chatbot can see?

Currently, the order in which the announcements are listed on the student portal (more or less according to postId) is different from the order in which the announcements are sent to the user when there are multiple posts in between scraping.

Syncing the order will definitely be helpful to see if the bot's doing things right, but don't know if this is really necessary for other users

lambda lastTimeUserWasUpdated: DB(everyThingSinceThatTime)

Student Portal has changed the categories. Need to decide whether to:

1. Users subscribe to everything
2. Update all topics/categories

We choose 1. Not because we are lazy though.

Both Student Portal and Facebook support some text formatting like bold, italics, etc.

We should make them translate smoothly.

There are two columns in the csv file we get from scraping the Student Portal.
The first is category_name and the second are topic. We want to decide which one we should give as options to the user to subscribe to.

category_name are things like [News & Information, Learning & Development, Job Opportunities, Internships & Volunteering, etc]
topic are things like [Academic Affairs, Academics, Campus Life, Community Outreach, etc]

For now, we are moving on with topics as we thought it is closer to the original idea we had. However, we're open to discussions about this.

I tried to clean install but it fails because the database does not automatically set up

for debugging purposes, it'll be nice to have timestamps

Only current students (not including people on leave, I believe) have access to student portal, but anyone with FB account can access the information. Should we/can we filter users?

1. scrapper sends json as string in the request body to /json
2. db server parses the string and change it into csv file
3. use readcsv function that's already defined by khaled to read csv file
4. use db.inject to inject data