olin-build / abe Goto Github PK

The current names of the heroku instances are confusing and no longer relevant. The DNS for abe.olin.build will need to be updated to reflect the new heroku instances.

Right now import is only for web-hosted calendars that ABE subscribes to, but I'm seeing a lot of individual ICS files for events that are being emailed out that I want to upload to the calendar, and I'm pretty sure we can do that easily with the ICS support we already have on the in ABE.

CloudWatch should send an alert on various resource utilization costs; maybe just data transfer.

This leads to recurring event exceptions being created before their parent definitions which is sad

Currently JSON fields are not ordered. Options include using OrderedDict or python 3.6 (where all dictionaries are ordered).

Currently we're using Heroku's free logging service, which caps out at 1,500 messages. There are a variety of other services available.

Notes from research:

• Heroku Log Drains documentation
• Heroku Logging Addons
• Article about connecting logdrains to AWS cloudwatch
• Web app from article

I went ahead and connected coralogix to abe-dev and abeweb, which holds logs for 5 days.

I was trying to add a birthday to ABE through python that repeated annually forever. After adding it /events/ requests were unresponsive and timed out. I didn't add the recurrence_end field initially, which may have been the issue. Either way more investigation is needed.

db.Event.save(
    **{
            "title": "Franklin Walter Olin's Birthday",
            "start": parser.parse("1860-01-08T20:00:00Z"),
            "end": parser.parse("1860-01-09T20:00:00Z"),
            'recurrence_end': datetime(2038, 1, 19),
            "visibility": "public",
            "labels": ["featured"],
            'recurrence': {
                'frequency': 'YEARLY',
                'interval': '1',
            }
    }
)

mongodb results:

{
	"_id" : ObjectId("596f636fe44d461db93be2e0"),
	"_cls" : "Event",
	"title" : "Franklin Walter Olin's Birthday",
	"start" : ISODate("1860-01-08T20:00:00Z"),
	"end" : ISODate("1860-01-09T20:00:00Z"),
	"visibility" : "public",
	"labels" : [
		"featured"
	],
	"recurrence" : {
		"frequency" : "YEARLY",
		"interval" : "1",
		"by_day" : [ ]
	},
	"sub_events" : [ ],
	"recurrence_end" : ISODate("2038-01-19T00:00:00Z")
}

This issue was originally “Implement access control for items that modify the calendar”. The first phase of access control is to restrict API access entirely. I've repurposed this issue to represent that requirement.

Monthly and yearly recurrences create dummy objects a week later

I can't seem to import any ICS feeds. Tried to get US holidays from Google Calendar

Also tried this one in case it was GCal specific, it's not.

I'm getting 500: Internal Server Error every time.

When trying to update a recurring event from the Web frontend, this happens:

Request

URL format: https://abe.olin.build/events/<sid>
Params:

{
    description: "Some text",
    rec_id: "2017-08-29T20:00:00-04:00",
    sid: "59774841e44d46221dfd0fa8",
    start: "2017-08-29T20:00:00-04:00",
    end: "2017-08-29T23:00:00-04:00"
}

Response

500 INTERNAL SERVER ERROR

Log output

File "/app/abe/resource_models/event_resources.py", line 138, in put
|     received_data['rec_id'] = dateutil.parser.parse(str(received_data['rec_id']))
| NameError: name 'dateutil' is not defined

For debugging and non-fullcalendar purposes this would be useful

In response to https://abe-dev.herokuapp.com/events/?start=2017-6-24&end=2017-8-5

{
    "description": "Fika (Swedish pronunciation: [\u00b2fi\u02d0ka]) is a concept in Swedish (and Finnish) culture with the basic meaning \"to have coffee\", often accompanied with pastries, cookies or pie.", 
    "end": "Thu, 06 Jul 2017 20:45:00 GMT", 
    "labels": [
      "summer", 
      "library"
    ], 
    "location": "Library UL", 
    "sid": "5967cf21e44d4623ad11c961", 
    "start": "Thu, 06 Jul 2017 19:00:00 GMT", 
    "title": "Coffee Break"
}

In response to https://abe-dev.herokuapp.com/events/5967cf21e44d4623ad11c961/2017-07-06T15:00:00

{
  "description": "Fika (Swedish pronunciation: [\u00b2fi\u02d0ka]) is a concept in Swedish (and Finnish) culture with the basic meaning \"to have coffee\", often accompanied with pastries, cookies or pie.", 
  "end": "Thu, 06 Jul 2017 16:45:00 GMT", 
  "labels": [
    "summer", 
    "library"
  ], 
  "location": "Library UL", 
  "rec_id": "Thu, 06 Jul 2017 15:00:00 GMT", 
  "sid": "5967cf21e44d4623ad11c961", 
  "start": "Thu, 06 Jul 2017 15:00:00 GMT", 
  "title": "Coffee Break"
}

It would be useful to attach photos/flyers to events, and potentially store them in a separate endpoint at /files/

A couple things that might be helpful regarding file uploading:

• It looks like files are accessed with request.files in Flask requests, but I'm not sure what the difference is between sending files, json data, and non-json put data and how flask handles it http://flask.pocoo.org/docs/0.11/patterns/fileuploads/
• Multiple files with flask: https://stackoverflow.com/questions/11817182/uploading-multiple-files-with-flask
• The field to add to document_models.py: http://docs.mongoengine.org/apireference.html#mongoengine.fields.FileField
• Amazon S3 Buckets

Currently ABE outputs datetimes in this format: "Sun, 06 Aug 2017 09:00:00 GMT".

It would be better to use a standard like ISO8601: "2017-08-06T09:00:00". Futureboard was having trouble parsing that specific string with their date library, and it makes sense to have control over the output of the API.

This could be done by extending Flask's JSONEncoder like so:

from flask.json import JSONEncoder

class CustomJSONEncoder(JSONEncoder):

    def default(self, obj):
        if isinstance(obj, datetime):
            return obj.isoformat()
        else:
            return JSONEncoder.default(self, obj)


app = Flask(__name__)
app.json_encoder = CustomJSONEncoder

Urls like https://abe.olin.build/events/1234 return a 500 when they should return a 404.

There was mention how caching could improve the performance of rrule (https://kite.com/docs/python/dateutil.rrule.rruleset)

Currently there's no way to see past request data in the logs, so requests have to be replicated and inspected from the sending side for debugging. Adding a logging.debug of request.__dict__ would be very helpful for debugging.

Current behavior

• When a label defaults to visible, default: "True" (string)
• When a label defaults to invisible, default is undefined

Desired behavior

• When a label defaults to visible, visible: true (boolean)
• When a label defaults to invisible, visible: false

Request

URL: https://abe.olin.build/labels/598d366ece40c1000d1e4d5f
Mode: PUT
Content-Type: application/json
Data: {"name":"PGP","color":"#ED037C","description":"Post-Graduate Planning (Career Services)","default":true}

Response

{ "message": "Internal Server Error" }

Heroku Log

heroku[router]: at=info method=PUT path="/labels/598d366ece40c1000d1e4d5f" host=abe.olin.build request_id=6e3593d5-8d15-44f1-a4c9-b980fd32c20f fwd="208.91.53.193" dyno=web.1 connect=1ms service=9ms status=500 bytes=238 protocol=https

DEBUG:ABE: || Label requested: 598d366ece40c1000d1e4d5f

| [2017-08-11 04:49:20,330] ERROR in app: Exception on /labels/598d366ece40c1000d1e4d5f [PUT]

| Traceback (most recent call last):

File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request

rv = self.dispatch_request()

File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request

return self.view_functionsrule.endpoint

File "/app/.heroku/python/lib/python3.6/site-packages/flask_restful/init.py", line 480, in wrapper

resp = resource(*args, **kwargs)

File "/app/.heroku/python/lib/python3.6/site-packages/flask/views.py", line 84, in view

return self.dispatch_request(*args, **kwargs)

File "/app/.heroku/python/lib/python3.6/site-packages/flask_restful/init.py", line 595, in dispatch_request

resp = meth(*args, **kwargs)

File "/app/abe/resource_models/label_resources.py", line 70, in put

result.update(**received_data)

| NameError: name 'received_data' is not defined

| INFO:ABE: || 10.45.50.47 - - [11/Aug/2017 04:49:20] "PUT /labels/598d366ece40c1000d1e4d5f HTTP/1.1" 500 -

For example: if I update the location in an event, the response to my PUT request has the old data in the location field.

Recurring events that shift days in the transfer from EST to UTC are expanded using the UTC datetime are outputted as a day earlier:

{
  "_id": ObjectId("59774841e44d46221dfd0fa8"),
  "_cls": "Event",
  "title": "Swingdancing",
  "start": ISODate("2017-07-06T00:15:00Z"),
  "end": ISODate("2017-07-06T02:00:00Z"),
  "allDay": false,
  "recurrence_end": ISODate("2017-08-27T13:00:00Z"),
  "visibility": "students",
  "labels": [
    "burstthebubble",
    "featured"
  ],
  "recurrence": {
    "frequency": "WEEKLY",
    "interval": "1",
    "until": ISODate("2017-08-31T00:00:00Z"),
    "by_day": [
      "WE"
    ],
    "by_month": [ ],
    "by_year_day": [ ],
    "forever": false
  },
  "sub_events": [ ],
  "location": "Cambridge"
}

is output like this:

[
  {
    "end": "2017-07-12T02:00:00", 
    "labels": [
      "burstthebubble", 
      "featured"
    ], 
    "location": "Cambridge", 
    "sid": "59774841e44d46221dfd0fa8", 
    "start": "2017-07-12T00:15:00", 
    "title": "Swingdancing"
  }, 
  {
    "end": "2017-07-19T02:00:00", 
    "labels": [
      "burstthebubble", 
      "featured"
    ], 
    "location": "Cambridge", 
    "sid": "59774841e44d46221dfd0fa8", 
    "start": "2017-07-19T00:15:00", 
    "title": "Swingdancing"
  }, 
  {
    "end": "2017-07-26T02:00:00", 
    "labels": [
      "burstthebubble", 
      "featured"
    ], 
    "location": "Cambridge", 
    "sid": "59774841e44d46221dfd0fa8", 
    "start": "2017-07-26T00:15:00", 
    "title": "Swingdancing"
  }, 
  {
    "end": "2017-08-02T02:00:00", 
    "labels": [
      "burstthebubble", 
      "featured"
    ], 
    "location": "Cambridge", 
    "sid": "59774841e44d46221dfd0fa8", 
    "start": "2017-08-02T00:15:00", 
    "title": "Swingdancing"
  }, 
  {
    "end": "2017-08-09T02:00:00", 
    "labels": [
      "burstthebubble", 
      "featured"
    ], 
    "location": "Cambridge", 
    "sid": "59774841e44d46221dfd0fa8", 
    "start": "2017-08-09T00:15:00", 
    "title": "Swingdancing"
  }, 
  {
    "end": "2017-08-16T02:00:00", 
    "labels": [
      "burstthebubble", 
      "featured"
    ], 
    "location": "Cambridge", 
    "sid": "59774841e44d46221dfd0fa8", 
    "start": "2017-08-16T00:15:00", 
    "title": "Swingdancing"
  }, 
  {
    "end": "2017-08-23T02:00:00", 
    "labels": [
      "burstthebubble", 
      "featured"
    ], 
    "location": "Cambridge", 
    "sid": "59774841e44d46221dfd0fa8", 
    "start": "2017-08-23T00:15:00", 
    "title": "Swingdancing"
  }, 
  {
    "end": "2017-08-30T02:00:00", 
    "labels": [
      "burstthebubble", 
      "featured"
    ], 
    "location": "Cambridge", 
    "sid": "59774841e44d46221dfd0fa8", 
    "start": "2017-08-30T00:15:00", 
    "title": "Swingdancing"
  }
]

Reduce the time it takes for data to be returned from a get request

Event:

Subscribed calendar:

@audreywl may have more details

[Oliver changed this to:]

The requirements are now in Pipfile. The import list in that file can probably be pared down to something like:

arrow = ">=0.10.0"
celery = ">=3.1.25"
celery-with-mongodb = ">=3.0"
flask = ">=0.12.2"
flask-cors = ">=3.0.2"
flask-restful = ">=0.3.6"
flask-socketio = ">=2.9.0"
flask-sslify = ">=0.1.5"
icalendar = ">=3.11.4"
isodate = ">=0.5.4"
mccabe = ">=0.6.1"
mongoengine = ">=0.13.0"
pymongo = ">=3.4.0"
python-dateutil = ">=2.6.0"
pytz = ">=2017.2"
requests = ">=2.18.1"
gunicorn = "*"

I know we decided not to devote a lot of time to this before, but it would be nice to have for events the happen every couple days. I'm building the parsing functionality on the frontend just in case.

I dealt with it on the frontend since we should be able to parse different date formats anyway, but it should probably be standardized for the API.
Normal Event:
start: "2017-07-13 21:00:00"

Dummy Recurring event:
start: "Mon, 24 Jul 2017 19:00:00 GMT"

Because you can't highlight pop-ups in Chrome, and also it's more convenient.

Single strings can be passed in for the labels field, which are parsed incorrectly somewhere between the database and the json response, splitting up the string into an array of characters.
link to issue on frontend.

Here's an example event in mongodb:

{
	"_id" : ObjectId("5967ddc6e23b8c000487d147"),
	"_cls" : "Event",
	"title" : "Midnight Memes",
	"description" : "![now including surreal memes](https://i.imgur.com/I1Nvebi.jpg)",
	"location" : "EH3nw",
	"start" : ISODate("2017-07-15T03:59:53Z"),
	"end" : ISODate("2017-07-15T05:00:53Z"),
	"visibility" : "students",
	"labels" : "Library",
	"sub_events" : [ ]
}

and the JSON output:

{
    "_cls": "Event", 
    "description": "![now including surreal memes](https://i.imgur.com/I1Nvebi.jpg)", 
    "end": "2017-07-15 05:00:53", 
    "id": "5967ddc6e23b8c000487d147", 
    "labels": [
      "L", 
      "i", 
      "b", 
      "r", 
      "a", 
      "r", 
      "y"
    ], 
    "location": "EH3nw", 
    "start": "2017-07-15 03:59:53", 
    "title": "Midnight Memes", 
    "visibility": "students"
}

This happens when I send an empty string to update the location or the description, without updating anything else.

Request

Request URL: https://abeweb.herokuapp.com/events/5978c0a67412b80011ab8220
Request method: PUT
JSON: location: ""

For Olin IP addresses, send back events with 'public' and 'olin' visibility. For all other, send back just 'public'

Depending on the order of tasks, it may be expedient to skip this and go straight to #216.

Y'all should make a boolean property on all labels that we can reference on the frontend to learn whether that label should be visible by default.

ABE can subscribe to ical feeds, but the ability to import ics files into ABE could be a useful feature. The ability to parse the ical file format is already present, and adding this feature would make an email interface for ABE easier to implement.

This could be implemented as a POST request to /events/, and maybe a dry-run flag or something similar to get the ABE version of the event(s) back without saving to the database.

Current label fields:

• name
• description
• url

Additional data that could be stored with labels:

• suggested color (issue on frontend)
• suggested visibility
• suggested default
• parent labels

https://blog.miguelgrinberg.com/post/using-celery-with-flask

olin-build/abe-web#93 (opened by @osteele) is really a backend issue, I believe.

Plz. I'll start passing you a boolean allDay field once they're supported on the frontend.

We don't want to allow subscribing an ABE instance to itself. This could be bad.

Potential solution: Just block URL of ABE instance

If I submit an ics feed in California, the events will appear in ABE on the east coast as if they were on the east coast. For example, an event starting at 9am PST will appear as it were starting at 9am EST on ABE (if ABE is on EST at the time).
We need to grab the information on the timezone from the ics feed and translate it to the times that mongodb saves.

In the interest of addressing some issues/ideas that have risen up in the past couple weeks, mainly:

• structuring the project in an organized and logical way to encourage contributions and development from Olin students with a variety of programming/Python experience
• implementing unit tests for more of the project (started with #62)
• using the same database connection across resource models
• implementing sockets for event creation/deletion for f-board integration
• implementing Sphinx and Read the Docs for documentation similar to the Olin API
• the ability to add additional features/endpoints and new mongoengine document models to ABE, like /files/ as mentioned in #52

It seems that we've reached the comfortable limit of a flat project structure - I recently rearranged the project files in #44 based on the guidelines from The Hitchhiker's Guide to Python. Looking at the Olin API's folder structure, I think that splitting up document_models.py and resource_models.py would be a nice first step. After that helpers.py needs to be pulled apart and reorganized - I'm still not sure what the best way to structure it would be.

Some resources to help with Flask, sockets, and celery along the way:

• http://flask.pocoo.org/docs/0.12/patterns/appfactories/#app-factories
• https://stackoverflow.com/questions/13284858/how-to-share-the-global-app-object-in-flask
• http://flask.pocoo.org/docs/0.12/blueprints/
• http://flask.pocoo.org/docs/0.12/api/#flask.current_app
• https://gist.github.com/jzempel/3201722

I'm proposing a project structure similar to this:

.
├── abe
│   ├── __init__.py
│   ├── app.py
│   ├── database
│   │   ├── __init__.py
│   │   ├── connection.py
│   │   ├── mongo_config_sample.py
│   │   └── mongo_config.py
│   ├── resources
│   │   ├── __init__.py
│   │   ├── event_model.py
│   │   └── label_model.py
│   ├── helpers
│   │   ├── __init__.py
│   │   ├── postdeploy.py
│   │   ├── pr-predestroy.py
│   │   ├── sample_data.py
│   │   ├── mongo_helpers.py
│   │   └── ical_helpers.py
│   ├── documents
│   │   ├── __init__.py
│   │   ├── event_document.py
│   │   └── label_document.py
│   └── templates
│       ├── add_event.html
│       ├── add_label.html
│       └── splash.html
└── tests
│   ├── __init__.py
│   ├── context.py
│   └── test.py
├── docs
│   └── [documentation stuff]
├── app.json
├── .gitignore
├── LICENSE
├── Procfile
├── README.md
├── requirements.txt
├── run.py
└── runtime.txt

https://github.com/mongomock/mongomock