crec_scrape is a respository that provides code to collect and tidy the daily editions of the U.S. Congressional Records from https://www.govinfo.gov/. Daily editions include text spoken on the floor of the Congress divided into four units: House of Representative, Senate, Daily Digest, and Extension of Remarks. By making our code available to R users, we hope to contribute to the accessibility of congressional debates and to the open science community.

The respository is the result of a group project within the SPOSM course of the Humboldt-University of Berlin and has the following structure:

• code: contains code to scrape, tidy and visualize the data
• output: store processed datasets
• plots: contains different plots that have been produced to visualize data for a presentation

This project is tested using R version 3.6.0 (2019-04-26), Platform: x86_64-w64-mingw32/x64 (64-bit), Running under: Windows 10 x64 (build 18362)

The general form to start the scraping, tidying, and visualization process of U.S. Congressional Records is to clone the respository and to open the file master.R from the code-directory. Here, you need to set up in a first step the working environment by installing required packages.

In order to start the scraping process, the second step of the script requires you to enter the start_date and end_date of the time period you want to download the Congressional Records for. Then, by sourcing 1_scraper.R the actual scraping process will be started. Thereby, the desired Congressional Records are downloaded as html-files and are stored in the directory output locally on your computer.

From there, the tidying process can start. By sourcing 2_clean_html_files.R, the script parses the text from each html-file and produces a dataset called my_data. The dataset my_data is structured as follows: vol, no, and date identify the daily edition of the Congressional Records. unit identifies the four main sections of each edition, namely House, Senate, Daily Digest, and Extensions of Remarks. pages, start_page, end_page identify associated page numbers for each unit in the edition and text contains the associated text. For further details, please have a look at the data output section. As the daily records are issued only when the congress is in session, a merging command introduce NA’s for dates when congress is not in session (e.g. weekends).

After creating the dataset my_data, the file 3_data_visualization.R need to be sourced as it contains several plotting functions. Our script proposes the following ideas to visualize the text, e.g.

• plotting the number of characters over time by unit
• plotting the top 10 most frequent words of a certain unit in a certain period
• creating wordclouds of a certain unit in a certain period
• plotting the frequency of selected keywords over time of a certain unit
• plotting the use of negative and positive words over time by unit

Based on the data output we want to encourage the community to build analysis on congressional debates, to visualize their findings and to share them with us. Pull request are therefore highly welcomed. Please read CONTRIBUTING.md for details on our code of conduct.