anishathalye / gavel Goto Github PK

Haven't gotten around to this yet…

In practice, this shouldn't be a problem, but still, it's nice to be thorough.

Feedback from HackBeanpot:

One issue that judges mentioned was that they wanted to know how many projects they had judged and how many more they needed to judge. I do think however if I better explained that they didn't need to judge every project then that probably wouldn't have been an issue.

It might still be nice to show judges some stats about how many projects that they've judged. Or maybe even showing them all the projects they've judged, as well as their votes (and this could double as an interface to edit votes).

Related to #22

To reproduce: start gavel, add a judge (e.g. Michael,,) but don't add any projects. Then head over to that judge's login page and enter. Stack trace here.

System information:
macOS 10.12
Vagrant 1.8.6
Gavel 88e72e4

At some hackathons, judges didn't know when to stop judging and kept going even after the organizers downloaded the results.

It would be nice to add a feature to explicitly open and close the system for judging.

Thanks to @zacharyliu for reporting this issue!

From his experience at HackPrinceton, @zacharyliu said that some issues that they had were:

• weird issue where one judge had 69 recorded votes within a few minutes, while the others had only ~20 votes by the end
• some judges started hitting random buttons even when we told them not to
• unable to delete bad votes from when judges hit random buttons before starting
• no way to undo accidental votes

Maybe we can improve this by adding:

• an ability to undo the last vote (to the judge UI)
• an ability for the organizers to clear votes for a specific judge / clear all votes

We need to have some discussion about exactly how this should be implemented.

Right now, Gavel doesn't really have a good way of supporting judging for different categories - it just produces an overall ranking.

One way of working around this is to get an overall ranking and then choose category winners based on the highest ranked submission in each category. But maybe there are better ways of doing this.

We may be implementing a Node.js version. Subscribe to this issue if you're interested in contributing to that!

This would require having actual styling in the first place… so this depends on #6 being finished first.

As a precaution in case judges aren't getting the email with the voting link, it would be nice to make the link accessible from the admin page so admins can give it to them through other channels like Slack or writing it on a piece of paper (hopefully we never have to do that).

Gavel doesn't let you remove judges or projects once they've started (if a project has been judged a nonzero number of times / if a judge has completed a judging). It shows a pretty opaque error message (about a foreign key constraint). It would be nice to show a more understandable error message (or better yet, implement recursive deletion and provide a prompt that asks if people want to do this).

Labeling this as a bug, even though it's not exactly an implementation bug (it's intentional), but it's kinda a UX bug.

pre-commit hook would be great to have to check for code styling and adhere to standard pythonic code guidelines

Ex. http://pre-commit.com/

It could be neat if Gavel handled project submissions (via a user-facing page).

Related to #7, #8, and #36

This would be awesome. Suggested by @dawsonbotsford

Gavel has been used mostly at hackathons. It would be neat to figure out other situations where such a system would be useful and then make Gavel more general so it can be used in more situations.

HackMIT used this script cause we didn't get around to implementing this functionality, but it shouldn't be too hard to do from within the app.

Even if Gavel didn't do this fully automatically, it could be made easier than it currently is.

Feedback from Uncommon Hacks:

Their one comment was that if it could be clearer when judging was determined - instead of it saying CLOSED it could say JUDGING OVER

Right now, there's a CSV import system, but it would be nice if there were a cleaner alternative way of doing it.

This is a design problem, so discuss here before writing any code.

Yeah, there's a lot of work to be done. Right now, there's basically no styling, so it looks pretty ugly.

Help would be very much appreciated!

This is pretty involved and involves substantial changes, so post here before you start working if you're interested in working on the issue.

The templates need some cleaning up so that they pass all validator checks.

It should be pretty easy making everything HTML5 compliant and cleaning up stuff like <input ... /> by changing it to <input ...>.

It could be neat to add (opt-out) automatic reporting of certain statistics.

This would probably involve having to maintain a server somewhere, though.

Right now, there's no hosted version of Gavel. If someone wants to use the system, they have to run their own server (or use Heroku) - this requires some level of technical sophistication.

It could be neat to modify the system to allow for a hosted version (and then we could have a default hosted version that people could use if they don't want to run their own).

This would involve adding user (organizer) accounts and stuff (and then supporting multiple events running at the same time), setting up a system where people can create events at eventname.<domain> or <domain>/event, and a bunch of other changes like this.

This would probably be a pretty substantial project.

Yeah, right now basically everything is in gavel.py. It would be good to make it more modular for the sake of maintainability.

This is pretty involved and involves substantial changes, so post here before you start working if you're interested in working on the issue.

Should we remove the CONTRIBUTING.md file in favor of GitHub's contributors feature?

Yeah… haven't had time to do that yet, but I thought it would be good to open-source this project now. If people want to contribute to this effort, that would be awesome, otherwise I'll probably get to it… eventually…

Factoring out the "auto-reload" from issue #52

Currently, seeing updated data in the admin panel requires a manual reload of the page. It would be nice to make the admin panel live. A lazy way to do that would be to add a meta refresh, but that could be intrusive (it's bad if the page reloads while someone is in the middle of reading something / doing something). A better way to do that would probably be to fetch data using javascript and update the current page without reloading the whole thing.

As referenced in Gitter, Freetail Hackers (org that runs HackTX) found the terms "annotators" and "items" a bit confusing when we first started looking at Gavel. It would be nice to have the frontend call them "judges" and "projects" instead.

We need a troubleshooting guide.

One thing that would be nice to include in there is email troubleshooting. Apparently, many people have had trouble with it in the past, and 1a4d362 is going to make troubleshooting email a little more tricky (because errors won't show up in the UI, you'll have to check the celery log output).

Gavel needs a reasonable number of judges - it would be nice to have the readme mention some guidelines on how many judges are needed (or rather, how many votes are needed, and about how much time is required per project, so that organizers can figure out how many judges they need).

It would be nice to add instructions on how to get started with development

Feedback from HackBeanpot

For setting up Gavel on the admin panel it went pretty smoothly. It would have been nice to add a note on the admin panel (or on the error page) that you could use quotes to enter descriptions with commas. I completely forgot I could do that with CSVs until after the event. A CSV import feature would be nice. Also an auto reload would also be nice to have.

Title says it all. The README contains enough information for people to get started, but it would be nice if it were more thorough.

Right now, we use an epsilon-greedy method of selecting which project a judge sees next. It could be good to modify it so that each project is judged some minimum number of times before having judges move around to maximize expected information gain.

It would also be good to implement stuff so that there aren't multiple judges overwhelming a project at once.

Making this customizable would be awesome.

This is pretty involved and involves substantial changes, so post here before you start working if you're interested in working on the issue.

Gavel has multiple components - the ranking algorithm is just one part of it, but the score collection / admin panel is another important part of the system. It would be neat if these were decoupled enough so that the system could support different judging schemes.

Right now, we only support pairwise comparison as a judging model with Crowd-BT as the algorithm, but maybe it would be cool to support other algorithms (e.g. Elo) and maybe even different judging models (e.g. normalization). It would be nice to write accompanying documentation that describes the tradeoffs of the different models and algorithms (e.g. with very few judges, Crowd-BT might be hard, and absolute scores + normalization may be an okay approach).

This would likely be a very substantial change to Gavel.

It could be nice to have a splash page (something that's more substantial than the readme). We could also have a detailed user guide there with detailed setup instructions, etc.

Having improved documentation could help with users not knowing how to use software. For example, I've gotten feedback where people have said that they don't know how to deal with projects that have moved locations - we have thought about this, and we support changing locations, but I guess people don't always get around to reading https://github.com/anishathalye/gavel#admin-panel-features

Setting IGNORE_CONFIG_FILE=false doesn't work properly because of this line: https://github.com/anishathalye/gavel/blob/master/gavel/settings.py#L11

Maybe this would be a nice feature to add.

Currently, HackMIT has the organizers visit projects that have been skipped lots of times, and if they can't be found, we manually disable them. But maybe making this automatic would be okay, adding some kind of auto_disable_threshold or something.

Yeah, this is broken right now. It's cause the system keeps track of all decisions (why not store more data), and so if we delete a project, it would be bad, cause the decision would point to a deleted project.

It's possible to delete projects before they're judged, though.

The right way to fix this would be to implement some sort of "deactivated" flag for projects and not have them show up anywhere (but still live in the database).

This is somewhat involved, so post here before you start working if you're interested in working on the issue.

Make it up to date for v1.0

It could be cool to support import from Devpost. Does Devpost have an API? I haven't looked into this yet.

Right now, some things are hard-coded, but it would be nice to set up some kind of config system thing to make stuff configurable (e.g. parameters in crowd_bt.py).

This is pretty involved and involves substantial changes, so post here before you start working if you're interested in working on the issue.

It would be good to eventually have tests and CI.

Currently, I think this fails at runtime (upon a web request), but it would be nice to fail fast at start time.

@lengstrom

Currently, the app is just designed for judges, and hackers don't interact with it at all.

It could be cool to have some sort of hacker view.

It looks like some other people have thought about this. See hemangandhi@efdec39.

I think all the dependencies support Python 3, so there's no reason we can't upgrade.

We talked a little bit about this in #10, and some things were fixed. There are still some things that could be explored / improved, though.

For example, we could try to minimize the distance that judges have to walk. At many events, the expo is arranged in a grid, with locations labeled like "B4". I'm sure we could apply some heuristic to try to reduce Euclidean distance that judges walk (while maintaining a good balance with maximizing information gain and all that stuff).

I deployed Gavel to Heroku on a hobby dyno, and I'm seeing a ton of crashes when voting on items. We had around 170 items and 10 annotators spamming the app simultaneously, and the app would show an Internal Server Error quite regularly after voting.

I checked the Heroku logs and this is the error that keeps repeating:

2016-11-07T17:45:24.292366+00:00 app[web.1]: [2016-11-07 17:45:24,289] ERROR in app: Exception on /vote [POST]
2016-11-07T17:45:24.292378+00:00 app[web.1]: Traceback (most recent call last):
2016-11-07T17:45:24.292380+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2016-11-07T17:45:24.292382+00:00 app[web.1]: context)
2016-11-07T17:45:24.292383+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2016-11-07T17:45:24.292384+00:00 app[web.1]: cursor.execute(statement, parameters)
2016-11-07T17:45:24.292385+00:00 app[web.1]: psycopg2.extensions.TransactionRollbackError: could not serialize access due to concurrent update
2016-11-07T17:45:24.292386+00:00 app[web.1]:
2016-11-07T17:45:24.292387+00:00 app[web.1]:
2016-11-07T17:45:24.292387+00:00 app[web.1]: The above exception was the direct cause of the following exception:
2016-11-07T17:45:24.292388+00:00 app[web.1]:
2016-11-07T17:45:24.292389+00:00 app[web.1]: Traceback (most recent call last):
2016-11-07T17:45:24.292390+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/flask/app.py", line 1988, in wsgi_app
2016-11-07T17:45:24.292391+00:00 app[web.1]: response = self.full_dispatch_request()
2016-11-07T17:45:24.292392+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/flask/app.py", line 1641, in full_dispatch_request
2016-11-07T17:45:24.292392+00:00 app[web.1]: rv = self.handle_user_exception(e)
2016-11-07T17:45:24.292393+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/flask/app.py", line 1544, in handle_user_exception
2016-11-07T17:45:24.292394+00:00 app[web.1]: reraise(exc_type, exc_value, tb)
2016-11-07T17:45:24.292420+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise
2016-11-07T17:45:24.292422+00:00 app[web.1]: raise value
2016-11-07T17:45:24.292422+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/flask/app.py", line 1639, in full_dispatch_request
2016-11-07T17:45:24.292423+00:00 app[web.1]: rv = self.dispatch_request()
2016-11-07T17:45:24.292424+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/flask/app.py", line 1625, in dispatch_request
2016-11-07T17:45:24.292425+00:00 app[web.1]: return self.view_functionsrule.endpoint
2016-11-07T17:45:24.292426+00:00 app[web.1]: File "/app/gavel/controllers/judge.py", line 25, in decorated
2016-11-07T17:45:24.292427+00:00 app[web.1]: return f(*args, **kwargs)
2016-11-07T17:45:24.292428+00:00 app[web.1]: File "/app/gavel/controllers/judge.py", line 33, in decorated
2016-11-07T17:45:24.292429+00:00 app[web.1]: annotator = get_current_annotator()
2016-11-07T17:45:24.292429+00:00 app[web.1]: File "/app/gavel/controllers/judge.py", line 150, in get_current_annotator
2016-11-07T17:45:24.292430+00:00 app[web.1]: return Annotator.by_id(session.get(ANNOTATOR_ID, None))
2016-11-07T17:45:24.292430+00:00 app[web.1]: File "/app/gavel/models/annotator.py", line 59, in by_id
2016-11-07T17:45:24.292431+00:00 app[web.1]: annotator = cls.query.with_for_update().get(uid)
2016-11-07T17:45:24.292432+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 831, in get
2016-11-07T17:45:24.292432+00:00 app[web.1]: return self._get_impl(ident, loading.load_on_ident)
2016-11-07T17:45:24.292433+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 864, in _get_impl
2016-11-07T17:45:24.292433+00:00 app[web.1]: return fallback_fn(self, key)
2016-11-07T17:45:24.292434+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/sqlalchemy/orm/loading.py", line 219, in load_on_ident
2016-11-07T17:45:24.292434+00:00 app[web.1]: return q.one()
2016-11-07T17:45:24.292435+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 2718, in one
2016-11-07T17:45:24.292436+00:00 app[web.1]: ret = list(self)
2016-11-07T17:45:24.292436+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 2761, in iter
2016-11-07T17:45:24.292437+00:00 app[web.1]: return self._execute_and_instances(context)
2016-11-07T17:45:24.292438+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/sqlalchemy/orm/query.py", line 2776, in _execute_and_instances
2016-11-07T17:45:24.292439+00:00 app[web.1]: result = conn.execute(querycontext.statement, self._params)
2016-11-07T17:45:24.292439+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 914, in execute
2016-11-07T17:45:24.292440+00:00 app[web.1]: return meth(self, multiparams, params)
2016-11-07T17:45:24.292441+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2016-11-07T17:45:24.292441+00:00 app[web.1]: return connection._execute_clauseelement(self, multiparams, params)
2016-11-07T17:45:24.292442+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2016-11-07T17:45:24.292443+00:00 app[web.1]: compiled_sql, distilled_params
2016-11-07T17:45:24.292443+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2016-11-07T17:45:24.292444+00:00 app[web.1]: context)
2016-11-07T17:45:24.292445+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
2016-11-07T17:45:24.292445+00:00 app[web.1]: exc_info
2016-11-07T17:45:24.292446+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
2016-11-07T17:45:24.292447+00:00 app[web.1]: reraise(type(exception), exception, tb=exc_tb, cause=cause)
2016-11-07T17:45:24.292448+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 185, in reraise
2016-11-07T17:45:24.292448+00:00 app[web.1]: raise value.with_traceback(tb)
2016-11-07T17:45:24.292449+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2016-11-07T17:45:24.292450+00:00 app[web.1]: context)
2016-11-07T17:45:24.292450+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2016-11-07T17:45:24.292457+00:00 app[web.1]: cursor.execute(statement, parameters)
2016-11-07T17:45:24.292459+00:00 app[web.1]: sqlalchemy.exc.OperationalError: (psycopg2.extensions.TransactionRollbackError) could not serialize access due to concurrent update
2016-11-07T17:45:24.292548+00:00 app[web.1]: [SQL: 'SELECT annotator.id AS annotator_id, annotator.name AS annotator_name, annotator.email AS annotator_email, annotator.active AS annotator_active, annotator.read_welcome AS annotator_read_welcome, annotator.description AS annotator_description, annotator.secret AS annotator_secret, annotator.next_id AS annotator_next_id, annotator.updated AS annotator_updated, annotator.prev_id AS annotator_prev_id, annotator.alpha AS annotator_alpha, annotator.beta AS annotator_beta \nFROM annotator \nWHERE annotator.id = %(param_1)s FOR UPDATE'] [parameters: {'param_1': 6}]

Could this be an issue with limits in the hobby dyno or something else? Would be nice to know that this won't be an issue on game day in three weeks :D

Feedback from Junction organizers -- the UI for the item name/location/description editing isn't that great / isn't really easily discoverable. We should fix that.

There's a workaround: not adding way too many people at once (doing it in batches).

Not sure what's the best way to fix this.

This is kinda related to #6, in the sense that making the UI easier should make it more intuitive to understand.

But besides that, it would also be nice to have some nice explicit instructions for judges built into the app (instead of having organizers having to explain everything in great detail / having auxiliary instructions).

Thanks to @zacharyliu for reporting this issue!