Read our contributing guidelines.
Use RVM to install the version of Ruby specified in .ruby-version.
Use NVM to install the version of NodeJS specified in the .nvmrc.
Docker can be used to: 1) run just the required services (MySQL, Elasticsearch, etc.) while running the search-gov application in your local machine, and/or 2) run the entire search-gov application in a Docker container. Please refer to searchgov-services for detailed instructions on centralized configuration for the services.
When running in a Docker container (option 2 above), the search-gov application is configured to run on port 3100. Required dependencies - (Ruby, NodeJS, Package Manager, Packages, Gems, JavaScript dependencies) - are installed using Docker. However, other data or configuration may need to be setup manually, which can be done in the running container using bash.
Using bash to perform any operations on search-gov application running in Docker container, below command needs to be run in search-services.
$ docker compose run search-gov bash
For example, to setup DB in Docker:
$ docker compose run search-gov bash
$ bin/rails db:setup
The Elasticsearch service provided by searchgov-services is configured to run on the default port, 9200. To use a different host (with or without port) or set of hosts, set the ES_HOSTS environment variable. For example, use following command to run the specs using Elasticsearch running on localhost:9207:
ES_HOSTS=localhost:9207 bundle exec rspec spec
Verify that Elasticsearch 7.17.x is running on the expected port (port 9200 by default):
$ curl localhost:9200
{
"name" : "002410188f61",
"cluster_name" : "es7-docker-cluster",
"cluster_uuid" : "l3cAhBd4Sqa3B4SkpUilPQ",
"version" : {
"number" : "7.17.7",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "78dcaaa8cee33438b91eca7f5c7f56a70fec9e80",
"build_date" : "2022-10-17T15:29:54.167373105Z",
"build_snapshot" : false,
"lucene_version" : "8.11.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}We recommend using Homebrew for local package installation on a Mac.
Use the package manager of your choice to install the following packages:
- C++ compiler - required by the cld3 gem, which we use for language detection
- Google's protocol buffers - also required by the cld3 gem
- Java Runtime Environment
- ImageMagick - required by the Paperclip gem, used for image attachments
- MySQL client - required by the mysql2 gem
- V8 - required by the libv8 gem
Example of installation on Mac using Homebrew:
$ brew install gcc
$ brew install protobuf
$ brew install java
$ brew install imagemagick
$ brew install [email protected]
$ brew install v8
Example of installation on Linux:
$ apt-get install protobuf-compiler
$ apt-get install libprotobuf-dev
$ apt-get install imagemagick
$ apt-get install default-jre
$ apt-get install default-mysql-client
Use Bundler 2.3.8 to install the required gems:
$ gem install bundler -v 2.3.8
$ bundle install
Refer to the wiki to troubleshoot gem installation errors.
Use Yarn to install the required JavaScript dependencies:
$ npm install --global yarn
$ yarn install
The app does its best to avoid interacting with most remote services during the test phase through heavy use of the VCR gem.
Run this command to get a valid secrets.yml file that will work for running existing specs:
$ cp config/secrets.yml.dev config/secrets.yml
If you find that you need to run specs that interact with a remote service, you'll need to put valid credentials into your secrets.yml file.
Anything listed in the secret_keys entry of that file will automatically be masked by VCR in newly-recorded cassettes.
You can create the USASearch-related indexes like this:
$ rake usasearch:elasticsearch:create_indexes
You can index all the records from ActiveRecord-backed indexes like this:
$ rake usasearch:elasticsearch:index_all[FeaturedCollection+BoostedContent]
If you want it to run in parallel using Resque workers, call it like this:
$ rake usasearch:elasticsearch:resque_index_all[FeaturedCollection+BoostedContent]
Note that indexing everything uses whatever index/mapping/setting is in place. If you need to change the Elasticsearch schema first, you can 'recreate' or 'migrate' the index:
Recreate an index (for development/test environments)
recreate_index task should only be used in development or test environments, as it deletes and then recreates the index from scratch:
$ rake usasearch:elasticsearch:recreate_index[FeaturedCollection]
Migrate an index (safe for production use)
In production, if you are changing a schema and want to migrate the index without having it be unavailable while the new index is being populated, do this:
$ rake usasearch:elasticsearch:migrate[FeaturedCollection]
Same thing, but using Resque to index in parallel:
$ rake usasearch:elasticsearch:resque_migrate[FeaturedCollection]
Create and set up your development and test databases:
$ rails db:setup
$ rails db:test:prepare
Make sure the unit tests, functional and integration tests run:
# Run the RSpec tests
$ rspec spec/
# Run the Cucumber integration tests
$ cucumber features/
# Run the JavaScript tests
$ yarn test
Optionally, to only run Cucumber accessibility tests:
$ cucumber features/ --tags @a11y
The above will call the axe step defined in features/support/hooks.rb for any scenario tagged with the @a11y tag (but not @a11y_wip as these are expected to fail).
We require 100% code coverage. After running the tests (both RSpec & Cucumber), open coverage/index.html in your favorite browser to view the report. You can click around on the files that have < 100% coverage to see what lines weren't exercised.
We use CircleCI for continuous integration. Build artifacts, such as logs, are available in the 'Artifacts' tab of each CircleCI build.
We use Rubocop for static code analysis. Settings specific to search-gov are configured via .rubocop.yml. Settings that can be shared among all Search.gov repos should be configured via the searchgov_style gem.
To run test searches, you will need a working Bing API key. You can request one from Bing, or ask a friendly coworker.
- Add the Bing
web_subscription_idtoconfig/secrets.yml:
bing_v7:
web_subscription_id: *****- Start your local development environment:
bin/dev
- Test searches should return results:
Web results
News results
- http://localhost:3000/search?affiliate=test_affiliate&query=news
- http://localhost:3000/search/news?affiliate=test_affiliate&channel=1&query=news
Video results
- http://localhost:3000/search?affiliate=test_affiliate&query=video
- http://localhost:3000/search/news?affiliate=test_affiliate&channel=3&query=video
Login.gov is used for authentication.
To create a new local admin account we will need to:
- Create an account on Login's sandbox environment.
- Get the Login sandbox private key from a team member.
- Add an admin user to your local app.
Create an account on Login's sandbox environment. This will need to be a valid email address that you can get emails at. You'll receive a validation email to set a password and secondary authentication method.
Ask your team members for the current config/logindotgov.pem file. This private key will let your local app complete the handshake with the Login sandbox servers. After adding the PEM file, start or restart your local Rails server.
Open the rails console, add a new user with the matching email.
u = User.where(email: '[email protected]').first_or_initialize
u.assign_attributes( contact_name: 'admin',
first_name: 'search',
last_name: 'admin',
default_affiliate: Affiliate.find_by_name('usagov'),
is_affiliate: true,
organization_name: 'GSA',
)
u.approval_status = 'approved'
u.is_affiliate_admin = true
u.save!
You should now be able to login to your local instance of search.gov.
Your user account should have admin privileges set. Now go here and poke around.
Several long-running tasks have been moved to the background for processing via Resque.
-
Visit the resque-web sinatra app at http://localhost:3000/admin/resque to inspect queues, workers, etc.
-
In your admin center, create a type-ahead suggestion (SAYT) "delete me". Now create a SAYT filter on the word "delete".
-
Look in the Resque web queue to see the job enqueued.
-
Start a Resque worker to run the job:
$ QUEUE=* VERBOSE=true rake environment resque:work -
You should see log lines indicating that a Resque worker has processed a
ApplySaytFiltersjob:
resque-workers_1 | *** Running before_fork hooks with [(Job{primary_low} | ApplySaytFilters | [])]
At this point, you should see the queue empty in Resque web, and the suggestion "delete me" should be gone from the sayt_suggestions table.
Each Resque job runs in the context of a queue named 'primary' with priorities assigned at job creation time using the resque-priority Gem. We have queues named :primary_low, :primary, and :primary_high. When creating a new background job model, consider the priorities of the existing jobs to determine where your jobs should go. Things like fetching and indexing all Odie documents will take days, and should run as low priority. But fetching and indexing a single URL uploaded by an affiliate should be high priority. When in doubt, just use Resque.enqueue() instead of Resque.enqueue_with_priority() to put it on the normal priority queue.
(Note: newer jobs inherit from ActiveJob, using the resque queue adapter. We are in the process of migrating the older jobs to ActiveJob.)
We use the resque-scheduler gem to schedule delayed jobs. Use ActiveJob's :wait or :wait_until options to enqueue delayed jobs, or schedule them in config/resque_schedule.yml.
Example:
-
In the Rails console, schedule a delayed job:
> SitemapMonitorJob.set(wait: 5.minutes).perform_later -
Run the resque-scheduler rake task:
$ rake resque:scheduler -
Check the 'Delayed' tab in Resque web to see your job.
Precompile assets
bin/rails assets:precompile
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent