turingschool-projects / lookingfor Goto Github PK

As a user viewing a company show page, we should be able to see the company name, basic company information and a list of all jobs related to that company.

Acceptance Criteria:

• Company information should be shown
• All jobs related to company should be listed
  • jobs should be ordered by most recent
  • No more than 10 jobs should show automatically on the page (pagination)
• Company should have a list of technologies that have been tagged across all job postings.
• Link to company show is added to the index page for jobs

Technologies set on jobs only match direct matches

• i.e. if the raw_technologies on a job has 'angular' and there is an 'angular' technology in db - there's a match... but if the raw_technologies has angular.js it won't be tagged as relating to angular.

Acceptance Criteria:

• Tests are added and matcher is made more robust for edge cases
• Existing records in the database are updated

Concern:

• Adding more matching may add more incorrect matching - something to be wary of and balance

Resources:
This one off rake task might help with the regex matching

Acceptance Criteria:

• Create a Basecamp account/project
• Share with entire team
• Update README

Size: 3

We Work Remote features remote jobs postings. They often have job postings that Stack Overflow doesn't have, so we need a scraper to pull in the job postings.

Acceptance Criteria:

• Job Fetch Service pulls in jobs from We Work Remote's RSS Feed
• Service is tested
• Service has an associated rake task to run it.

Resources:
https://weworkremotely.com/categories/2-programming/jobs.rss

If you run Job.pluck(:location).uniq you'll find some odd locations. We should investigate how these locations are being created by looking at the job titles and beefing up testing.

Acceptance Criteria:

• Run through bizarre locations - add testing for edge cases
• Beef up regex to handle as many edge cases as possible OR switch to NokoGiri (see Erinna's notes)
• Write one off rake task to correct existing records that are borked
  • www.regex101.com is super helpful

• Update basecamp
• Deploy Changes
• Confirm New Relic up and running

Authentic Jobs has an API that may have jobs that we can use. We would like to have a scraper to pull in the job postings.

Acceptance Criteria:

• Evaluate if Authentic Jobs is a viable service to scrape
• If it is...
  • Job Fetch Service pulls in jobs from Authentic Jobs' RSS Feed
  • Service is tested
  • Service has an associated rake task to run it.
• If it isn't
  • Document why in the Basecamp for the project

Resources:
https://authenticjobs.com/api/

Concerns:
Many APIs rate limit usage. How do we make sure our application doesn't hit those rate limits?

Concerns:

• Extreme privacy concerns - can we maintain them?

Size: 1

Acceptance Criteria:

• LookingFor is deployed on a concurrent web server

Resources:

• https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server

Either external or internal?
Links to GlassJobs
Other ways to beef up company information

Information on the interview process

Acceptance Criteria:

• List of jobs can be filtered by technology
• Filtering can be triggered either by clicking the full list of technology
  • or by clicking the technology tags on individual jobs

Acceptance Criteria:

• Look at information that's being collected
• Document potential implementations of collecting trends
  • Document what data needs to be used
  • Document what data is present vs not being collected vs not being normalized
• On a branch, try manipulating the data
• If seems feasible
  • Write up stories around implementing
• If not - add documentation around your findings to BaseCamp

Size: 3

Currently the Stack Overflow service tests only cover a small section of the service - and don't test with real data.

Acceptance Criteria:

• Test coverage on the Stack Overflow service that brings in real sample data

Resources:
Check out the Sinatra app that handles remote requests in /spec/support
https://robots.thoughtbot.com/how-to-stub-external-services-in-tests

In order to evaluate how our application is performing, we'll need to expose it to heavy load. To do this, we will be responsible for implementing a load-testing script which exercises the production application.

Your load script can be provided as a separate project/repository with its own structure and set of dependencies, or included in the existing application as a rake task or other executable script.

The goal of this script is to exercise as many of the application's endpoints as possible. The evaluation rubric for this coverage will be based on the required user paths listed below:

Acceptance Criteria:

• Load testing script exercises all endpoints
• Load Testing Lesson Plan

Size: 3

Acceptance Criteria:

• User can see jobs ordered by date
• User sees tags for technologies related to the job
• User sees company name related to the job
• Pagination

With all the records we're handling now, it won't be feasible to simply render a list of every item that's for sale. On any page that includes a "list" of items (Index, Tags, etc), add a Pagination interface that allows users to:

Click through a numbered list of pages
See which page they're currently on
Go to the first page
Go to the last page
Go to the next page
Go to the previous page

Size: 3

Docker is so hot right now.
But do we need to use container technology for this project?

ACs:

• Research using Docker with this project
• Create a branch to run lookingfor in a Docker container
• Demo in a team stand up the pros and cons for using Docker for this application

Resources:

https://docs.docker.com/compose/rails/
http://www.zdnet.com/article/what-is-docker-and-why-is-it-so-darn-popular/
https://www.ctl.io/developers/blog/post/what-is-docker-and-when-to-use-it/

Size: 8

Note: Required #37 be fixed prior to work

Users should be able to query job postings by their location. ie if they are looking in North Carolina, they should be able to scope the job postings by this location.

Unsure how to implement this as location is not always accurate or uniform.

Fix could be beefing up uniformity of location on jobs OR some kind of elastic search across the contents of title and description OR kick off a background worker to do this string search and output a csv 😨

https://devcenter.heroku.com/articles/upgrading-heroku-postgres-databases

Size: 3

Right now we can run our job fetcher by typing heroku run rake job_fetch:stackoverflow - which is cool... but we're programmers and we should probably automate this.

Acceptance Criteria:

• One task runs all job fetch services
• That task is run every day at least once automatically

Resources:
https://devcenter.heroku.com/articles/scheduler

Note: Relates to #38

Users should be able to search for companies

Acceptance Criteria:

• User is able to begin inputting a company name from the top navigation of any page on LookingFor
• When user submits search, user sees a listing of fuzzy matched companies
  • match should not be case sensitive - ThougHtbot should return Thoughtbot record
  • match should be fuzzy:
    • Thought should return Thoughtbot

Reach Goal:
The ideal implementation for this feature would remove the requirement to submit the search and instead have fuzzy matches appear in a drop down below the search field on key release.

Size: 5

Acceptance Criteria:
Review the documentation from Amber in BaseCamp
Create and document creating a Docker container for the Rails app
Document the deployment of the Rails app (on either Digital Ocean or Heroku)
Document multi-app deployment using Docker

Reach goal:
Actually deploying both the React Client and Rails App together

Size: 3

React is so hot right now.
But do we need it for LookingFor?
And how, of the many ways we could integrate React, would we want to do it?

ACs:

• Research using React with this project and Document the following things:
  • Pros and Cons of using a front end js framework for this project
  • Options we could use to integrate React (react-gem, separate front end app, etc)
  • Changes to the API or pieces of the API that would need to be written
  • Pros and cons of using React over other solutions
• Demo in a team stand up the above research

Resources:

https://github.com/reactjs/react-rails
https://medium.com/@alexfedoseev/isomorphic-react-with-rails-part-i-440754e82a59#.rukej9jwn

Would like to have the ids for each job and technology returned in the JSON response. Right now, the endpoint returns only the company id. Would like something like this:

^That is the fake data we are using right now. Thanks!

Size: 5

In addition to handling heavy request load from users, our applications will need to handle database load against a db with large numbers of records. To do this, we'll need to seed the various tables in our DBs with lots of records.
Writing your seed script won't be too difficult, but running it can take some time, so it's recommended that you get started on this part early. Additionally, Ryan Bates' Populator Gem may be worth a look.

Acceptance Criteria:

• Create a 'seed' script that creates the following
• 500,000+ "Job" records
• 200,000+ "Company" records
• 300+ "Technology" records

Choice 1

• Create and deploy a staging server (if one does not already exist)
• Update the README with the staging server location
• Set up New Relic on the staging server (if not already set up in #7)
• Run the seed script and document the application's performance against the load

Choice 2

• Have a flag in the database or a timestamp that differentiates fake records from real ones
• Write said script
• Run the script during non-peak times
• Have another script or rake task that cleans up the fake records

Would like an endpoint to show individual jobs so we can access it using the id (api/v1/jobs/2). Thanks!

CORS - cross origin resource sharing.

We would like to hit the heroku API endpoint with our app, but we are unable to access it currently.

Chelsea set this up previously with a 'rack-cors' gem and David Whitaker just implemented this on his app, so they would be good resources.

Acceptance Criteria:

• Create docs folder
• Add user profiles as discussed in the kick off

Relates to #65

Information on the interview process

• GlassDoor
• Comments in Slack (slackbot or hook)
• Internal interface to add from Turing

Acceptance Criteria:

• Research what Ruby Gems are already in place
• Research set up of new elasticsearch database and copying over information from existing database, as well as new, incoming data.
• Document new dependencies introduced in adding Elasticsearch
• Document how data would be stored
• Document differences in databases and how Elasticsearch database is queried through curl requests
• On branch, try to seed ElasticSearch DB with existing data
• Write up issues for the next steps

New Relic free account may not give us the information we need for database load testing, so we may need to either bring in another service (Skylight?) or pay for an account for a month or so to get the data we need.

Acceptance:

Breakdown what New Relic provides with both types of accounts compared to what we need
Compare with other services, including what time would be needed to reimplement those services
Provide a recommendation on which service would be best for database load testing and potentially future projects related to response time, etc.

Dependencies: #9

Acceptance Criteria:

• There's a logo for the Github organization

Some jobs coming in from stack overflow don't seem to have any technologies attached.

As a part of this project, the illustrious Pat Wey created a gem to encapsulate the We Work Remote functionality.

We should consider using this gem.

Pros:

• Pat has to maintain the gem
• Theoretically, the gem will stay up to date with the We Work Remote service

Cons:

• Pat has to maintain the gem
• If the gem does not stay up to date, we will depend on the open source owner of the project to merge any pull requests we need - otherwise we'll have to ditch using the gem.

Acceptance Criteria:
Either gem is used and internal service is scrapped or decision is made to keep internal service

Resources:
https://rubygems.org/gems/we_work_remotely_jobs

Concerns:

• Will students actually use this
• Are students okay with this data being public - and to what level.
• Likely requires login functionality

• Set up email to use for major accounts
• Document details in BaseCamp

Angellist has an API that may have jobs that we can use. We would like to have a scraper to pull in the job postings.

Acceptance Criteria:

• Evaluate if Angel List is a viable service to scrape
• If it is...
  • Job Fetch Service pulls in jobs from Angel List Remote's RSS Feed
  • Service is tested
  • Service has an associated rake task to run it.
• If it isn't
  • Document why in the Basecamp for the project

Resources:
https://angel.co/api/spec/jobs

Concerns:
Many APIs rate limit usage. How do we make sure our application doesn't hit those rate limits?

http://stackoverflow.com/jobs/feed?tags=postgresql

Acceptance Criteria:

• API endpoint provides serves json of jobs
• User can see jobs ordered by date
• User sees technologies connected to each job
• User sees company name and id related to the job
• Pagination
• API endpoint is tested

With all the records we're handling now, it won't be feasible to simply send a list of all jobs from the endpoint. The API will need to be scoped to having pages of results

Resources:
http://stackoverflow.com/questions/30229898/rails-api-gem-pagination

We may be able to use some additional APIs to scrape. Some potential sources might be LinkedIn, Github, Dice. These services may not, however, allow us to use them or be feasible to use.

Acceptance Criteria:

Evaluate above services for their viability.

If any are...
Write up an issue with acceptance criteria

If any aren't...
Document why in the Basecamp for the project

Concerns:

• Many APIs rate limit usage. What are the rate limits on the service
• Terms of Service may not allow our use case

We want to make sure our tests are always in a passing state without having to run them.

Acceptance Criteria:

• Github is hooked up to a service that runs tests on every pull request, verifying that they are passing
• Update the README with instructions and a link to where the tests are being run
• Update basecamp with account information for logging in

Resources:
https://travis-ci.org/
http://circleci.com
http://codeship.com
https://drone.io/
http://wercker.com/

Concerns:
Should be a stable service that's easy to use and integrate. Should also be free.

Searching for Go jobs is difficult in Authentic Jobs and StackOverflow since they are essentially searching the title and description for the word 'go'

Acceptance Criteria:

• Look at records tagged for Go
  • Find any records that seem to be matching incorrectly
    • If incorrect match found, add a test case and beef up regex or other matching used in the specific service that found that job

Size: 5

APIs change frequently. Very frequently.

Our app should notify us in some way if we start creating Jobs with missing locations, for example. Or if a StackOverflow scrape returns no jobs. While these things may not be 'errors' (we don't want the app to fall over) we should have a window into them.

Acceptance Criteria

• Maintainers of the LookingFor application have an interface to see if:
  • Job Fetcher Services start setting nil values in job fields
  • APIs that Job Fetcher services hit start going down frequently
  • Job Fetcher Services start returning no hits

Resources:
These notices could live in a page on the LookingFor site that track metrics - they could use a SlackBot to talk to a room in slack to update metrics - they could send an email to maintainers with weekly metrics, etc.

As a user viewing a job show page, we should be able to see the company name (with link to company show if exists), basic job information.

Acceptance Criteria:

• Company name is shown
• Job information should be shown
• List of other jobs tied to the company can be seen
• No more than 10 jobs should show automatically on the page (pagination)
  List of technologies tags should be shown
  Link to job show is added to the index page for jobs

Size: 3

The WeWorkRemotely service tests don't test with real data!

Acceptance Criteria:

• Test coverage on the WeWorkRemotely service that brings in real sample data

Resources:
Check out the Sinatra app that handles remote requests in /spec/support
https://robots.thoughtbot.com/how-to-stub-external-services-in-tests

Size: 1

New Relic

Acceptance Criteria

• Application is hooked up to New Relic for monitoring
• Instructions are documented in the README
• A stand up presentation of how to use New Relic should be given

The project will be expected to comfortably handle several hundreds of requests per minute. We will use New Relic to judge the following:

4: Average response time below 100ms with 600+ RPM
3: Average response time below 200ms with 400+ RPM
2: Average resposne time below 400ms with 300+ RPM
1: Average response time above 400ms, or unable to handle 300 RPM

Needs more info

User Can Sign Up and Get a Regular Email Summary

Concerns:
Should the user sign up for an account OR just add an email that gets emailed by the service.
How frequently should the email go out?
What service would we use to send the emails?
If we don't use a service, how do we handle unsubscribe.
Can we make it a spoken word reading call?

Would be good to have this data normalize - Relates to #37

Size: 5

Acceptance Criteria:

• Application integrates a production error tracking service which notifies the team whenever an exception occurs in production.

Resources:

• https://airbrake.io/
• Some open source examples include:
  • https://github.com/Sharagoz/rails_exception_handler
  • http://smartinez87.github.io/exception_notification/
  • http://errbit.github.io/errbit/.