titipata / penn-events-calendar Goto Github PK

Currently, we use the following script to remove duplicates. We should group by owner (except a few) and remove duplicates by URL and get last scraped events instead.

• Create a server with permanent (elastic) IP
• Ask IT (or ISC) at Penn to associate the domain name to the IP (waiting for MindCORE to grant temporary domain name)

• Add ability to filter events by owners in drop-down menu e.g.:

  • all
  • Computational Neuroscience Initiative
  • English department
  • ...

Adding instruction on how to use the website and also how to report the missing events in issue page.

• Create SQLite database to store data instead of JSON file
• In fetch_events.py, we will check if the event is new and add to the database
• In api.py, fix the API to retrieve data from the database instead of reading JSON file
• Run cron job to fetch events everyday and update the database

sometimes there is missing location information or, worse, locations could be changed. A link would allow people to be sure (and it is a great fallback in case parsers fail)

Had a discussion with @titipata about the direction of the project after porting to Gatsby. Here is a summary:

• we would need a user for recording their response on the event.
  • what will we use here? store data on the server or on the web storage?
    • need to think about, would we allow the user to login from another browser but still have the same selected event?
• add like/dislike button on the event item
• add more tabs, i.e., home, selected events, and recommendations

Create a nginx configuration file for reverse proxy and also port management.

There are some data issues seen on rendering. Here is a list:

• Event with no title
  • This is fixed by filtering during grouping the events by date.
  • However, this should be fixed from the scraping/parsing process as there is actually a title when I went to visit event URL.
• Location with False as string (not boolean) (image 1)
  • I believe this is from the scraping/parsing process too.
  • I will let this slip for now but want your idea (@titipata) on how we should handle the events with no location. My suggestions are:
    • Hide the location label.
    • Show the word unknown. (image 2)

image 1

image 2

• All somehow have 07 as minute which is weird, may be this is timezone error (+7 to min instead of hr).
• Time of English Department school have no endtime. Actually there is but sometimes it's in description so that we may need to parse from there. Using regex can be one option such as below case: (fixed in #24)

Forms of Justice: The Rise of the Legal Procedural in the Era of Due Process Graduate Lounge,
Fisher-Bennett Hall Room 330 2:00pm - 3:00pm Public portion of defense 3:00pm - 4:00pm
Private portion of defense 4:00pm - 4:30pm Public celebration Please join Sara Sligar and her
dissertation committee for her defense at

We can get all time here by pattern /[0-9]+\:[0-9]+[ ]*[ap]m[ ]*-[ ]*[0-9]+\:[0-9]+[ ]*[ap]m/. With re.match() we should get a list of all time and we can then process to get the actual time we need.

• need to handle (All Day) from English Department.

TODOs

• find cause and fix 07 in minute.
• handle (All Day) events.

• #62 add pagination to a container, we should decide how many events should be shown per page
  • https://www.gatsbyjs.org/docs/adding-pagination/
  • I think we should limit how many months we will show events instead
• #71 sort event in each day by? (starttime, endtime, duration, title)
• #97 mplement responsive navbar (there should be a way to fix width of the logo image)
• add loading indicator when switching from page to page
• add feedback to interactive DOM like button and card (as mentioned in #20)
  • now we applied keyframes to some of the interactive DOMs
  • we can also add transition to some
  • this will need to be elaborated on which DOM to do, probably creating a new issue if we found something off
• add mini calendar so that user can navigate to each day
  • https://www.gatsbyjs.org/docs/gatsby-link/#recommendations-for-programmatic-in-app-navigation
  • lelivrescolaire/react-light-calendar#5
  • will open this up later if needed

As we implement DOM from scratch, there are no user feedback (effect or animation) at all when user clicked on clickable DOM (with onClick or onPress). Most of the DOM we see this effect on general website are such as a and button.

We need to implement this in some way. First idea is to look for those of Bootstrap.

I will write the production installation document later. This issue is for my memo.

Auto start service on boot Ubuntu 16.04

systemctl enable penn.service

• Add options on the search page for upcoming and include previous week on frontned.
• Add hug API to allow search back for one week

I wonder if one could, based on all the scrapers built so far, produce a scraper that would work on all pages. Simples would be to try all the scrapers we have so far and then choose a working one if we have one. But alternatives may be more ML based. I think building a universal scraper (for talk announcements only) might be a great masters project

After serving the site with --prefix-paths, any request to the backend seems to be wrong. We need to figure this out.

Also, when there is no selected events yet, it looks like the loading page will continue indefinitely.

• It should take selected event_index as a list
• Perform averaging of selected events --> preference_vector
• Make a cosine distance between --> preference_vector, all_existing_events --> argsort for the closest distance
• Return list of recommended_event_index as an output.

Implement completion suggester feature from elasticsearch

Reference:

• https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters.html#completion-suggester
• https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters.html#completion-suggester
• https://kb.objectrocket.com/mongo-db/how-to-use-python-to-update-api-elasticsearch-documents-259

es.update(
    index='penn-events',
    id=1,
    doc_type='event',
    body={
        'doc':{
            'suggest': ['Pratik Chaudhari', 'Barchi']
        }
    },
    refresh=True,
)

Then for loop through all events and update.

doc = {
    'size' : 10000,
    'query': {
        'match_all' : {}
    }
}
indices = [int(r['_id']) for r in es.search(
    index='penn-events', 
    body=doc,
    scroll='1m'
)['hits']['hits']]

To search events

suggest_body = {
    "suggest": {
        "field-suggest" : {
            "prefix" : "prati", 
            "completion" : { 
                "field" : "suggest" 
            }
        }
    }
}
es.search(index='penn-events', body=suggest_body)

Workflow:

• user type search text in the frontend
• then, suggest all titles coming from backend
• allow them to click on one of the search result
• take the event index to search result page, get recommendation of that particular event
• list under the search one label You may also interested in ...

Use https://github.com/Supervisor/supervisor to schedule when to fetch and index events.

See temp demo at http://54.201.64.205/.

TODOs

• change cursor to pointer on the card when there is detail, only be pointer on Collapse area when expanded.
• change See Details to Collapse when description is expanded, also change arrow direction
• change onclick handler, only collapse when clicking on Collapse button
• add URL to event, only show when the detail is expanded show next to event title because all have URL but not description -> open this URL in new tab on click
• adjust spacing in description, the rendered just doesn't look good as seen in this card:
  • this is due to description is also HTML so it contains a lot of   stuffs (see raw string below).

Fisher-Bennett Hall room TBD

V21: Victorian Studies for the 21st Century is sponsoring its fourth annual 
summer reading group series in 2018, with groups convening in multiple cities. 
 Topics are “Work," “Blackness," and “Sex."  The Philadelphia-area 
group will meet at Penn on June 21, July 19, and August 
16 2-4pm.  All are welcome!

We have to clear the localStorage if we observe outlier event_index. This can happen when the data is updated and the former localStorage still stores the previous event_index which might exceed index in the dataset.

The current favicon is from Gatsby. Probably using Penn logo here.

• remove navbar, change footer background color probably white or very light grey
• footer text:

Made at kordinglab.com at University of Pennsylvania
Help us make it better at titipata/penn-events-calendar

• Penn logo in the middle follow by upcoming events below for the beginning (will discuss about font with @bluenex later today!)
• group display talks by date
• put date as a header. Display today date as today, tomorrow as tomorrow and afterwards in Month Date format e.g. Jun 25

Currently, it's is in a tight space so we can't see what is being typed in to the input. We should make it responsive so that the user on mobile device has a better experience.

• Changing logo of the page, it is ReactJS currently
• Adding URL to event if existed to the detail extension

The aim is to make automation for site deployment. Once it is done, the new version should be deployed automatically. Candidate tools for this are:

• Jenkins
• Travis CI
• GitHub Actions

Make it HTTPS using https://letsencrypt.org/

Create a script that can check and kill process related to supervisord and gatsby. Currently, we use ps aux | grep supervisord to see the pid. However, there can sometimes be processes running in port 8888, 9200 (elasticsearch), and 8000 or 9000 (gatsby).

Backend

• implemented in #24
• new endpoint /api/v1/getsimilarevents/<string:event_id> is created

Frontend

• add redux-thunk to enable async dispatch
• get events from endpoint when focus
• when the card is clicked, even though there is no detail on the event, the event id will be dispatched to fetch similar events.
• similar events will be in redux store events.similarEvents
• design UI for similar events

Currently, we use spacy to extract person and organization entities from the document as search candidates. It works fine but still getting a lot of entities wrong. Here, we want to see if there are techniques words that can better extracts keywords from the document.

Possible solutions:

• use react-virtualized to lazy load list.
• implement a true pagination using GraphQL as suggested by Gatsby docs.
• change response of endpoints to include requested data.

• selected events should also show events in the past
  • need localStorage to be implemented
  • eslint for Hooks should be added too

Create an elastic search for current events.

• Backend index all events using elasticsearch library

• Frontend create search tab calling port 9200 for getting search events, sort output by date and display. We should also implement the auto-complete function as described here.

We notice that once we run the program supervisor-elasticsearch, it runs correctly. However, when we try to stop it, only the parent process is killed leaving the child processes running indefinitely.

We may be able to resolve this with stopasgroup option as mentioned in https://stackoverflow.com/a/20566215/4010864.

• https://blog.daftcode.pl/react-calendar-with-google-calendar-as-cms-tutorial-5f5d81e425a9

The event is available at https://web.sas.upenn.edu/mindcore/events/. Not sure if there is an easy to download format for the page or not.

We use event_index to track selected and recommended events by saving them to localStorage of the browser. This works fine because we only append more events to the file. However, when we need to refresh the data after some time, event_index will no longer be the same anymore. This will cause incorrect results of search and recommendation.

To solve this we will add metadata to the events.json:

{
  "name": "...",
  "refresh_count": 1,
  "fetch_date": "...", 
  "modified_date": "",
  "data": ["list_of_events"],
}

• refresh_count: increase only when fetch_date updated once every 1-2 years
• fetch_date: date of the first fetch date
• modified_date: this is updated regularly (weekly) but it should be "" when first_date updated
• data: a list of events

Create a script for setting up uwsgi with gunicorn or nginx. Not sure if this is the good one to follow.

@kittinan do you know where is a good site to follow for setting up uwsgi/gunicorn for Python backend?

Let's do MIT License @bluenex. Let me know if you're fine with it.

• Create Google form with the following input date, description, endtime, location, owner (drop down), speaker, starttime, title, url.
• All backend to read CSV file from Google and add it to the current events
• Edit Usage instrcutions` so that people can add events to Google form

Here are list of calendar at Penn

• Med School
• CLUNCH
• English department

• Fetch MindCORE events
• Fix "Invalid date"
• allow recommendation sort by relevances/ timeline
• reduce recommendation size to 15

As from React version 16.8.0, React Hooks is officially included in the stable release. We should adopt hooks for this repo for simplicity sake.

Create a parameter file specifying the IP address or IP name so that we can run outside the localhost. Here, we need to update script in

• package.json

    "deploy": "concurrently -n 'hug,gatsby' -c 'bgBlue,bgMagenta' 'cd ../backend && 
hug -f hug_api.py -p 8888' 'gatsby build && gatsby serve -H {{IP_NAME}}'"

• src/pages/recommendations.js

useEffect(() => {
    fetch('http://localhost:8888/recommendations', {
    ...

We save and use data from within repository. This makes API call break when time past for a while because there is no data to fetch. We should have a simple script to generate JSON of events and similar events locally.

• typos on README:
  • https://github.com/titipata/penn-events-calendar/blame/master/README.md#L35
    • there should be a link to gatsby_site dir as well
  • https://github.com/titipata/penn-events-calendar/blame/master/README.md#L37
    • no full-stop, there should be a link to devops dir
  • https://github.com/titipata/penn-events-calendar/blame/master/README.md#L26
    • there should be :
  • https://github.com/titipata/penn-events-calendar/blame/master/gatsby_site/README.md#L4
    • there should be :
  • https://github.com/titipata/penn-events-calendar/blame/master/gatsby_site/README.md#L22
    • there should be :
  • https://github.com/titipata/penn-events-calendar/tree/master/devops
    • add supervisor-fetch-events
• add loading indicator on frontend page
• add prefix to gatsby so that it serves at http://bleen.seas.upenn.edu/penn-events instead of http://bleen.seas.upenn.edu:9000/
• change project name and details in package.json

Update serve_elasticsearch.sh so that it detects the operating system and download the right elasticsearch from https://www.elastic.co/downloads/elasticsearch

Add button in event to sync it to Google Calendar. Mainly, we have to see the format of how to add to the Google calendar.

• University Museum #f64242
• Weiss Tech House #ffb51c
• Department of Mechanical Engineering and Applied Mechanics #49a9a6
• Nursing Bulkload #a3e8dc
• Department of Chemical and Biomolecular Engineering #ba2644
• Kelly Writers House #ff6b6b
• School of Arts and Sciences Office of Advancement #ff83bd
• Department of Bioengineering #aaaaaa
• Department of Materials Science and Engineering #61c0bf
• Institute for Research in Cognitive Science #bbded6
• Department of History and Sociology of Science #fae3d9
• Department of Electrical and Systems Engineering #ffb6b9
• Department of Microbiology #7af0b7
• Penn Humanities Forum #493714
• Office of Government and Community Affairs #df8f4d
• GRASP Laboratory #00818a
• Center for Advanced Study of India #ffcd77
• The Working Group in Applied Mathematics and Computational Science #d7907b
• Bhakti Yoga Band #eee1b3
• School of Engineering and Aplied Science #d7907b
• Center for Neuroscience & Society #9a8c98
• Wharton Entrepreneurship #7c98b3
• Wharton Leadership Program #accbe1
• Career Services #ebbab9
• Department of Germanic Language and Literature #ff8a22
• Pereman Quadrangle #b72b34
• Nano/Bio Interface Center #708775
• PRECISE #b63b3b
• Urban Studies #505050
• SEAS Library #f9c414
• idom #27567b
• Department of Computer and Information Science #8fc0e7
• Wharton Entrepreneurial Programs #f5b70c
• Slavic Department #fbf1f1
• Libraries #ff7562
• School of Medicine #00cccc
• Center for Human Modeling and Simulation #24160d
• Singh Center for Nanotechnology #3a8ba2
• School of Design #00e595
• Faculty Senate #b1f2d7
• Society for Creative Anachronism #fde583
• SEAS Deputy Dean's Office #d7b8ff
• Penn Humanities Forum- Bulkload #07988d
• Institute for Law and Economics #099e72
• Center for Healthcare Improvement & Patient Safety #ca8b65
• Theatre Arts Program #2d174d
• Chaplain's Office #ff6200
• University Communications #a53f2b
• Med School Bulkload #f5f621

From backend, we will send relevance value together with event_index through a response like:

[{event_id: 123, relevance: 82}, ...]

When frontend received the response, we will translate the value into percentage and draw color on top of the star. Here is a reference on this https://codepen.io/bluenex/pen/oNvOZJY