chihacknight / civic-json-worker Goto Github PK
View Code? Open in Web Editor NEWWorker script that pulls metadata about opengov projects
License: MIT License
Worker script that pulls metadata about opengov projects
License: MIT License
Hey guys, I've cloned this and want to hook it up for Oakland but I'm a bit of a newbie. For the civic worker app, it seems like I need to get an Amazon EC2 account/space where I can store the Flask app and the output JSON as a resource.
Sound like the right approach?
It would seem that there are duplicates still in the machine. See: chihacknight/chihacknight.org#21
Dunno why I didn't think of this before. Huh. Related to chihacknight/chihacknight.org#21
As fun as it is to have my github commit history augmented by a robot committing on my behalf every four hours, I am constantly reminded that this is happening because in one hour out of four, my github user account is over its API limit. (This will also effect https://github.com/open-city/chicago-river-sewage because my account is being used to push the scraped data to github every so often.)
Not only is this slightly annoying for me, it also means that the actual work that this worker is trying to accomplish is almost certainly not completing. There may be a more efficient way to leverage the API that doesn't require that many calls so maybe that's really what we ought to do. In the meantime, making a dummy user would be appreciated.
@derekeder Any thoughts?
Double check that when a user submits a project, it's not already in the listing. See: chihacknight/chihacknight.org#21
Let's create a JSON file that shows a list of people with total number of contributions of the projects we track.
Just added a repo for which I am the owner and sole contributor (real community effort, I know but I was testing the add project endpoint, OK?). Even after running the update manually, it didn't update my user totals on the people page. This might mean that the owner counts are not getting added in there...
Need to consolidate this junk more so that it's easier to maintain
75 projects listed here: https://s3.amazonaws.com/civic-json/projects.json
But only 59 show up here: https://s3.amazonaws.com/civic-json/project_details.json
As of right now, hitting /update-projects/ is emptying out the detail listing. I think this may have to do with github responding with a non-200 status code when we attempt to fetch the info about repos. I have a theory (which I will test in the morning) that this might be hitting a rate limit (they tend to be rather restrictive about non-authenticated requests).
A better approach here would be to not dump out the file before rebuilding it (duh). Another thing to do would be to actually authenticate the request so that we aren't rate limited.
Anyways, research is needed
Entered "https://github.com/smartchicago/wikichicago"
Returned: Error submitting your project! Check that the URL is correct.
I also tried submitting "https://github.com/GovInTrenches/ctahardship" and received the same error.
This civic tech project tracking API is generically useful infrastructure. Instead of having people deploy their own versions of this API, which has proven difficult, we should just modify this service to work for multiple civic hacking groups and cities.
Our API is already up and running. Civic hacking groups would just need to submit their repo urls to the endpoint from their websites. The system would hit the Github API and get additional repo details, just like it's doing for Chicago. Brigades could then request a list of their projects to display on their own project pages.
Here's what would need to happen, as far as I can tell. (@evz, please jump in):
will require making another API call to contributors_url
"contributors": [
{
"login": "hunterowens",
"avatar_url": "https://2.gravatar.com/avatar/ff16180e6d9715d5ae17526cb83d7cfb?d=https%3A%2F%2Fidenticons.github.com%2Fc3663e10b721e5d713a55aebf8621901.png",
"html_url": "https://github.com/hunterowens",
"contributions": 74
},
{
"login": "evz",
"avatar_url": "https://1.gravatar.com/avatar/1d0c5faee140af87d7d6967bc946ecc6?d=https%3A%2F%2Fidenticons.github.com%2F44e80db9ed8527f429c969e804432b0f.png",
"html_url": "https://github.com/evz",
"contributions": 55
},
{
"login": "fishmanadam",
"avatar_url": "https://2.gravatar.com/avatar/d1529c1f159f49be15cf5fde0366b599?d=https%3A%2F%2Fidenticons.github.com%2Fdb534c76db466f7764d7e745900be88d.png",
"html_url": "https://github.com/fishmanadam",
"contributions": 7
},
{
"login": "jtwalsh0",
"avatar_url": "https://0.gravatar.com/avatar/05e1d117da7a62657f617bd553f4ed83?d=https%3A%2F%2Fidenticons.github.com%2F31d750398f38453e376cf435deb5fea9.png",
"html_url": "https://github.com/jtwalsh0",
"contributions": 2
},
{
"login": "matthewgee",
"avatar_url": "https://1.gravatar.com/avatar/1dbb7a239d59f1effc696f6424050a5a?d=https%3A%2F%2Fidenticons.github.com%2F9bf88927e24ab526ed285da976feecf0.png",
"html_url": "https://github.com/matthewgee",
"contributions": 1
}
]
Another JSON file for tracking organizations and total number of repositories (based on what we're tracking).
Originally reported in the open-gov-hack-night issue tracker under issue 14, searching for "invalid email address" in the People search of open-gov-hack-night will link to a real Github user:
It's an account provided by Github to indicate when there is an invalid author email address for a commit. In this case, it was found on 93 commits of look-at-cook; the email address in question was devnull@localhost
.
Since this tracks a publicly curated list of projects, it's out of our control to make sure all future project contributors are also valid. Instead, we should filter out this user account from the list of contributors.
add a project
$.post( 'http://civic-json-app.herokuapp.com/add-project/', data={'project_url': "https://github.com/my-name/my-project/"}, function(resp){ console.log(resp) });
delete a project
$.post( 'http://civic-json-app.herokuapp.com/delete-project/', data={'project_url': "https://github.com/my-name/my-project/"}, function(resp){ console.log(resp) });
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.