olin-build / abe Goto Github PK
View Code? Open in Web Editor NEWAmorphous Blob of Events
Home Page: https://abe.olin.build/
License: GNU Affero General Public License v3.0
Amorphous Blob of Events
Home Page: https://abe.olin.build/
License: GNU Affero General Public License v3.0
For example: if I update the location in an event, the response to my PUT request has the old data in the location field.
When trying to update a recurring event from the Web frontend, this happens:
URL format: https://abe.olin.build/events/<sid>
Params:
{
description: "Some text",
rec_id: "2017-08-29T20:00:00-04:00",
sid: "59774841e44d46221dfd0fa8",
start: "2017-08-29T20:00:00-04:00",
end: "2017-08-29T23:00:00-04:00"
}
500 INTERNAL SERVER ERROR
File "/app/abe/resource_models/event_resources.py", line 138, in put
| received_data['rec_id'] = dateutil.parser.parse(str(received_data['rec_id']))
| NameError: name 'dateutil' is not defined
URL: https://abe.olin.build/labels/598d366ece40c1000d1e4d5f
Mode: PUT
Content-Type: application/json
Data: {"name":"PGP","color":"#ED037C","description":"Post-Graduate Planning (Career Services)","default":true}
{ "message": "Internal Server Error" }
heroku[router]: at=info method=PUT path="/labels/598d366ece40c1000d1e4d5f" host=abe.olin.build request_id=6e3593d5-8d15-44f1-a4c9-b980fd32c20f fwd="208.91.53.193" dyno=web.1 connect=1ms service=9ms status=500 bytes=238 protocol=https
DEBUG:ABE: || Label requested: 598d366ece40c1000d1e4d5f
| [2017-08-11 04:49:20,330] ERROR in app: Exception on /labels/598d366ece40c1000d1e4d5f [PUT]
| Traceback (most recent call last):
File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functionsrule.endpoint
File "/app/.heroku/python/lib/python3.6/site-packages/flask_restful/init.py", line 480, in wrapper
resp = resource(*args, **kwargs)
File "/app/.heroku/python/lib/python3.6/site-packages/flask/views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "/app/.heroku/python/lib/python3.6/site-packages/flask_restful/init.py", line 595, in dispatch_request
resp = meth(*args, **kwargs)
File "/app/abe/resource_models/label_resources.py", line 70, in put
result.update(**received_data)
| NameError: name 'received_data' is not defined
| INFO:ABE: || 10.45.50.47 - - [11/Aug/2017 04:49:20] "PUT /labels/598d366ece40c1000d1e4d5f HTTP/1.1" 500 -
default: "True"
(string)default
is undefinedvisible: true
(boolean)visible: false
Currently ABE outputs datetimes in this format: "Sun, 06 Aug 2017 09:00:00 GMT"
.
It would be better to use a standard like ISO8601: "2017-08-06T09:00:00"
. Futureboard was having trouble parsing that specific string with their date library, and it makes sense to have control over the output of the API.
This could be done by extending Flask's JSONEncoder like so:
from flask.json import JSONEncoder
class CustomJSONEncoder(JSONEncoder):
def default(self, obj):
if isinstance(obj, datetime):
return obj.isoformat()
else:
return JSONEncoder.default(self, obj)
app = Flask(__name__)
app.json_encoder = CustomJSONEncoder
Y'all should make a boolean property on all labels that we can reference on the frontend to learn whether that label should be visible by default.
In the interest of addressing some issues/ideas that have risen up in the past couple weeks, mainly:
It seems that we've reached the comfortable limit of a flat project structure - I recently rearranged the project files in #44 based on the guidelines from The Hitchhiker's Guide to Python. Looking at the Olin API's folder structure, I think that splitting up document_models.py
and resource_models.py
would be a nice first step. After that helpers.py
needs to be pulled apart and reorganized - I'm still not sure what the best way to structure it would be.
Some resources to help with Flask, sockets, and celery along the way:
I'm proposing a project structure similar to this:
.
├── abe
│ ├── __init__.py
│ ├── app.py
│ ├── database
│ │ ├── __init__.py
│ │ ├── connection.py
│ │ ├── mongo_config_sample.py
│ │ └── mongo_config.py
│ ├── resources
│ │ ├── __init__.py
│ │ ├── event_model.py
│ │ └── label_model.py
│ ├── helpers
│ │ ├── __init__.py
│ │ ├── postdeploy.py
│ │ ├── pr-predestroy.py
│ │ ├── sample_data.py
│ │ ├── mongo_helpers.py
│ │ └── ical_helpers.py
│ ├── documents
│ │ ├── __init__.py
│ │ ├── event_document.py
│ │ └── label_document.py
│ └── templates
│ ├── add_event.html
│ ├── add_label.html
│ └── splash.html
└── tests
│ ├── __init__.py
│ ├── context.py
│ └── test.py
├── docs
│ └── [documentation stuff]
├── app.json
├── .gitignore
├── LICENSE
├── Procfile
├── README.md
├── requirements.txt
├── run.py
└── runtime.txt
Recurring events that shift days in the transfer from EST to UTC are expanded using the UTC datetime are outputted as a day earlier:
{
"_id": ObjectId("59774841e44d46221dfd0fa8"),
"_cls": "Event",
"title": "Swingdancing",
"start": ISODate("2017-07-06T00:15:00Z"),
"end": ISODate("2017-07-06T02:00:00Z"),
"allDay": false,
"recurrence_end": ISODate("2017-08-27T13:00:00Z"),
"visibility": "students",
"labels": [
"burstthebubble",
"featured"
],
"recurrence": {
"frequency": "WEEKLY",
"interval": "1",
"until": ISODate("2017-08-31T00:00:00Z"),
"by_day": [
"WE"
],
"by_month": [ ],
"by_year_day": [ ],
"forever": false
},
"sub_events": [ ],
"location": "Cambridge"
}
is output like this:
[
{
"end": "2017-07-12T02:00:00",
"labels": [
"burstthebubble",
"featured"
],
"location": "Cambridge",
"sid": "59774841e44d46221dfd0fa8",
"start": "2017-07-12T00:15:00",
"title": "Swingdancing"
},
{
"end": "2017-07-19T02:00:00",
"labels": [
"burstthebubble",
"featured"
],
"location": "Cambridge",
"sid": "59774841e44d46221dfd0fa8",
"start": "2017-07-19T00:15:00",
"title": "Swingdancing"
},
{
"end": "2017-07-26T02:00:00",
"labels": [
"burstthebubble",
"featured"
],
"location": "Cambridge",
"sid": "59774841e44d46221dfd0fa8",
"start": "2017-07-26T00:15:00",
"title": "Swingdancing"
},
{
"end": "2017-08-02T02:00:00",
"labels": [
"burstthebubble",
"featured"
],
"location": "Cambridge",
"sid": "59774841e44d46221dfd0fa8",
"start": "2017-08-02T00:15:00",
"title": "Swingdancing"
},
{
"end": "2017-08-09T02:00:00",
"labels": [
"burstthebubble",
"featured"
],
"location": "Cambridge",
"sid": "59774841e44d46221dfd0fa8",
"start": "2017-08-09T00:15:00",
"title": "Swingdancing"
},
{
"end": "2017-08-16T02:00:00",
"labels": [
"burstthebubble",
"featured"
],
"location": "Cambridge",
"sid": "59774841e44d46221dfd0fa8",
"start": "2017-08-16T00:15:00",
"title": "Swingdancing"
},
{
"end": "2017-08-23T02:00:00",
"labels": [
"burstthebubble",
"featured"
],
"location": "Cambridge",
"sid": "59774841e44d46221dfd0fa8",
"start": "2017-08-23T00:15:00",
"title": "Swingdancing"
},
{
"end": "2017-08-30T02:00:00",
"labels": [
"burstthebubble",
"featured"
],
"location": "Cambridge",
"sid": "59774841e44d46221dfd0fa8",
"start": "2017-08-30T00:15:00",
"title": "Swingdancing"
}
]
Because you can't highlight pop-ups in Chrome, and also it's more convenient.
Monthly and yearly recurrences create dummy objects a week later
For debugging and non-fullcalendar purposes this would be useful
I dealt with it on the frontend since we should be able to parse different date formats anyway, but it should probably be standardized for the API.
Normal Event:
start: "2017-07-13 21:00:00"
Dummy Recurring event:
start: "Mon, 24 Jul 2017 19:00:00 GMT"
If I submit an ics feed in California, the events will appear in ABE on the east coast as if they were on the east coast. For example, an event starting at 9am PST will appear as it were starting at 9am EST on ABE (if ABE is on EST at the time).
We need to grab the information on the timezone from the ics feed and translate it to the times that mongodb saves.
There was mention how caching could improve the performance of rrule (https://kite.com/docs/python/dateutil.rrule.rruleset)
Currently there's no way to see past request data in the logs, so requests have to be replicated and inspected from the sending side for debugging. Adding a logging.debug
of request.__dict__
would be very helpful for debugging.
This issue was originally “Implement access control for items that modify the calendar”. The first phase of access control is to restrict API access entirely. I've repurposed this issue to represent that requirement.
Reduce the time it takes for data to be returned from a get request
Urls like https://abe.olin.build/events/1234 return a 500 when they should return a 404.
In response to https://abe-dev.herokuapp.com/events/?start=2017-6-24&end=2017-8-5
{
"description": "Fika (Swedish pronunciation: [\u00b2fi\u02d0ka]) is a concept in Swedish (and Finnish) culture with the basic meaning \"to have coffee\", often accompanied with pastries, cookies or pie.",
"end": "Thu, 06 Jul 2017 20:45:00 GMT",
"labels": [
"summer",
"library"
],
"location": "Library UL",
"sid": "5967cf21e44d4623ad11c961",
"start": "Thu, 06 Jul 2017 19:00:00 GMT",
"title": "Coffee Break"
}
In response to https://abe-dev.herokuapp.com/events/5967cf21e44d4623ad11c961/2017-07-06T15:00:00
{
"description": "Fika (Swedish pronunciation: [\u00b2fi\u02d0ka]) is a concept in Swedish (and Finnish) culture with the basic meaning \"to have coffee\", often accompanied with pastries, cookies or pie.",
"end": "Thu, 06 Jul 2017 16:45:00 GMT",
"labels": [
"summer",
"library"
],
"location": "Library UL",
"rec_id": "Thu, 06 Jul 2017 15:00:00 GMT",
"sid": "5967cf21e44d4623ad11c961",
"start": "Thu, 06 Jul 2017 15:00:00 GMT",
"title": "Coffee Break"
}
Currently we're using Heroku's free logging service, which caps out at 1,500 messages. There are a variety of other services available.
Notes from research:
I went ahead and connected coralogix to abe-dev
and abeweb
, which holds logs for 5 days.
[Oliver changed this to:]
The requirements are now in Pipfile
. The import list in that file can probably be pared down to something like:
arrow = ">=0.10.0"
celery = ">=3.1.25"
celery-with-mongodb = ">=3.0"
flask = ">=0.12.2"
flask-cors = ">=3.0.2"
flask-restful = ">=0.3.6"
flask-socketio = ">=2.9.0"
flask-sslify = ">=0.1.5"
icalendar = ">=3.11.4"
isodate = ">=0.5.4"
mccabe = ">=0.6.1"
mongoengine = ">=0.13.0"
pymongo = ">=3.4.0"
python-dateutil = ">=2.6.0"
pytz = ">=2017.2"
requests = ">=2.18.1"
gunicorn = "*"
This leads to recurring event exceptions being created before their parent definitions which is sad
olin-build/abe-web#93 (opened by @osteele) is really a backend issue, I believe.
ABE can subscribe to ical feeds, but the ability to import ics files into ABE could be a useful feature. The ability to parse the ical file format is already present, and adding this feature would make an email interface for ABE easier to implement.
This could be implemented as a POST request to /events/
, and maybe a dry-run
flag or something similar to get the ABE version of the event(s) back without saving to the database.
The current names of the heroku instances are confusing and no longer relevant. The DNS for abe.olin.build will need to be updated to reflect the new heroku instances.
I was trying to add a birthday to ABE through python that repeated annually forever. After adding it /events/
requests were unresponsive and timed out. I didn't add the recurrence_end
field initially, which may have been the issue. Either way more investigation is needed.
db.Event.save(
**{
"title": "Franklin Walter Olin's Birthday",
"start": parser.parse("1860-01-08T20:00:00Z"),
"end": parser.parse("1860-01-09T20:00:00Z"),
'recurrence_end': datetime(2038, 1, 19),
"visibility": "public",
"labels": ["featured"],
'recurrence': {
'frequency': 'YEARLY',
'interval': '1',
}
}
)
mongodb results:
{
"_id" : ObjectId("596f636fe44d461db93be2e0"),
"_cls" : "Event",
"title" : "Franklin Walter Olin's Birthday",
"start" : ISODate("1860-01-08T20:00:00Z"),
"end" : ISODate("1860-01-09T20:00:00Z"),
"visibility" : "public",
"labels" : [
"featured"
],
"recurrence" : {
"frequency" : "YEARLY",
"interval" : "1",
"by_day" : [ ]
},
"sub_events" : [ ],
"recurrence_end" : ISODate("2038-01-19T00:00:00Z")
}
Currently JSON fields are not ordered. Options include using OrderedDict or python 3.6 (where all dictionaries are ordered).
It would be useful to attach photos/flyers to events, and potentially store them in a separate endpoint at /files/
A couple things that might be helpful regarding file uploading:
request.files
in Flask requests, but I'm not sure what the difference is between sending files, json data, and non-json put data and how flask handles it http://flask.pocoo.org/docs/0.11/patterns/fileuploads/document_models.py
: http://docs.mongoengine.org/apireference.html#mongoengine.fields.FileFieldCurrent label fields:
name
description
url
Additional data that could be stored with labels:
Right now import is only for web-hosted calendars that ABE subscribes to, but I'm seeing a lot of individual ICS files for events that are being emailed out that I want to upload to the calendar, and I'm pretty sure we can do that easily with the ICS support we already have on the in ABE.
We don't want to allow subscribing an ABE instance to itself. This could be bad.
Potential solution: Just block URL of ABE instance
Plz. I'll start passing you a boolean allDay field once they're supported on the frontend.
I can't seem to import any ICS feeds. Tried to get US holidays from Google Calendar
Also tried this one in case it was GCal specific, it's not.
I'm getting 500: Internal Server Error every time.
I know we decided not to devote a lot of time to this before, but it would be nice to have for events the happen every couple days. I'm building the parsing functionality on the frontend just in case.
Single strings can be passed in for the labels field, which are parsed incorrectly somewhere between the database and the json response, splitting up the string into an array of characters.
link to issue on frontend.
Here's an example event in mongodb:
{
"_id" : ObjectId("5967ddc6e23b8c000487d147"),
"_cls" : "Event",
"title" : "Midnight Memes",
"description" : "![now including surreal memes](https://i.imgur.com/I1Nvebi.jpg)",
"location" : "EH3nw",
"start" : ISODate("2017-07-15T03:59:53Z"),
"end" : ISODate("2017-07-15T05:00:53Z"),
"visibility" : "students",
"labels" : "Library",
"sub_events" : [ ]
}
and the JSON output:
{
"_cls": "Event",
"description": "![now including surreal memes](https://i.imgur.com/I1Nvebi.jpg)",
"end": "2017-07-15 05:00:53",
"id": "5967ddc6e23b8c000487d147",
"labels": [
"L",
"i",
"b",
"r",
"a",
"r",
"y"
],
"location": "EH3nw",
"start": "2017-07-15 03:59:53",
"title": "Midnight Memes",
"visibility": "students"
}
This happens when I send an empty string to update the location or the description, without updating anything else.
Request URL: https://abeweb.herokuapp.com/events/5978c0a67412b80011ab8220
Request method: PUT
JSON: location: ""
For Olin IP addresses, send back events with 'public' and 'olin' visibility. For all other, send back just 'public'
Depending on the order of tasks, it may be expedient to skip this and go straight to #216.
CloudWatch should send an alert on various resource utilization costs; maybe just data transfer.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.