voteview / webvoteview Goto Github PK

Add parties page listing all parties with date range in which they had members in the Congress and max number of members also color code used for that party.

Currently, "first elected" shows when the person was first elected to congress. Jeff expressed that for the congressional seniority sort, it should instead use when the person was first elected to their current chamber. This would probably require rejiggering either the old or new db schemas.

The patch that updates all the early Republicans to be, as they are, Democrat-Republicans, does not update the party vote count field, so we can't get party stats on any Democrat-Republican votes. The patch should be edited to fix this as well.

The states_all.json file is too large. It is ~900KB (225KB gzipped) for 71 state boundaries and some metadata, while individual state-congress files are ~101KB (28KB gzipped) for 50 state boundaries.

Can we get this down to, say, 200KB as a target? How is this file produced?

The thumbnail photos in the list are sometimes distorted even if they're correct on the actual member page.

In the 104th senate, for instance:

I can imagine these rollcall pages being handouts in a class. Can we have a pdf or print button?

As the code grows it's a good idea to test it so we ensure that new functionality didn't break old one. I can take care of this.

I'm planning to use pytest but if you have another suggestion I'm all ears :)

The parties at a glance page loads slowly.

Loading Code
There are three things I want to do to try to improve speed:

1. Instead of requesting 10 different party files, request one grand file.
2. Right now, I make the data sparse once in the export script, and further on rendering. I did this so I could easily test changes without rexporting. By doing both passes in the export script, the file size will reduce.
3. Right now, I include senate, house, and combined ideologies and we only use combined.

Other causes of speed issues
DC having to make all the dimensions and plot them.

I just checked the page and realize that the amount of images could be problematic (some pages have more than 200). It could be a good idea to optimize member images for the web.

I run jpegoptim and looks like we can save ~50% of the size using lossless compression. I have this in a branch, if you want I can create a PR.

Fix cutting lines and form ellipse of feasible coordinates

We use a Google Fonts webfont for our site. This is by far the slowest thing in our pageloads (around 300ms on a first load). It also injects Google Analytics code onto our site for some reason. We should investigate localfont.com and other options to see about local-hosting the fonts so that users still get the benefit of the fonts but we don't have to wait on Google Fonts. We should also see if this is actually something where our server would be any faster or if the fonts are just large enough that it's a throughput issue.

Members without nominate data do not load the ideology histogram, and much of the other javascript loading fails. The failure is silent though, so it's not immediately clear to me where the actual error is.

The party control map loads a JSON containing each unique state-border configuration over the history over the US (total 71 states). For example, Massachusetts was originally what today we call Massachusetts and Maine. So the map contains present-day Maine, and the old Northern Part of Massachusetts. We handle the party control map by selectively dropping the opacity to states that are out of the time range to 0.

When we mouse over a state, a tooltip fires. Unfortunately, DC.js thinks the tooltip is associated with whatever the top layer of the map is, even if that layer has opacity 0. For example, move your mouse over West Virginia before Virginia was split, and you'll get no tooltip, because DC.js thinks you're mousing over a non-existent West Virginia.

This might be an intractable problem with DC, but I wanted to log it here.

Not sure if this is a particular data problem or an assumption in the code about valid data that is false in at least one case.

Here is an example of such a member page (maybe the only one):
http://128.97.229.160/person/4854

We should have a page that allows us to list up to n (8? 10?) votes, which are presented as columns, and we list all the members who voted on those votes as rows. This would allow people to quickly compare how members voted on the votes.

• Load votes from stash
• Sort table by any vote
• Sort table by ideology
• Subset members by party
• Subset members by checked / unchecked
• Export filtered table as CSV
• Plot vote cut lines on single graph
• Create scorecard by letting users select which vote direction is the high score.
• Manually select votes through dropdowns (this might be a UI mess)

User story:
I search for a bunch of key votes from the last congress, compare across them, select 2020 presidential hopefuls, subset to just those voters, and export.

User story:
I want to replicate an interest group score using our votes

http://voteview.polisci.ucla.edu/rollcall/H1141180

A portion of our code requires updates each congress to ensure things do not break. We should inventory this code, look at impacts to the site if we fail to update each congress (i.e. will it break features or just display poorly), decide if can we automate the transition or is it necessarily manual, note which files are affected, etc. I don't think we'll forget to do the 2017 transition to the 115th, but by the time the next one rolls around, we might be less involved.

• Vote maps rely on the existence of a json file for each congress. If a map is not found for a given congress, the map will not display but the vote page will otherwise work. Jeff can speak to if it is possible to automate this further. We could also make a cron job to automatically copy this session's maps to the next session--I don't know what kind of errors this could lead to if we did. Maybe do this pre-emptively until 2020? We could also write error handling code to fall back on the most recent good map in the event that no maps exist. Edit: The vote map will now try to load a map from one congress ago if the current congress' map is missing.
• The maps used for the party control map rely on state boundaries that have "expiry date". Jeff has set the expiry dates to 2020. If the states expire, then in 2021 our map will just be blank. This requires updating the /static/json/states_all.json file. Does it make sense to maybe set them to 2050 (i.e. push the problem until we are dead or retired)? Do we expect any states to secede and/or split in two? Admission of new states isn't a problem--first, because we're not likely to passively forget when this happens. Second, because the error would only be the new state not appearing, nothing would happen to the map as a whole. I would recommend pushing the date at least a little bit past 2020.
• A variety of graphs rely on a hardcoded maximum congress number for setting x-axes. If we do not change this, data from current congresses will not show up on those pages. I think it probably makes sense to hardcode the maximum congress number in exactly one place, have all of the graphs read the maximum from that one place, and then if we're confident change that one place from hardcoded to dynamic depending on the calendar or we could create a cron job to dynamically pull the maximum congress number from the vote databases. Thoughts?

I have no idea if this is exhaustive. We should grep all code looking for 113, 114, or 115 (to account for off-by-one), as well as 2014, 2015, 2016, 2017, 2020, 2025, 2030. Most of this is my code but I'd appreciate it if someone else could hop in to help work through this.

Sorting member votes breaks the date column, because of the way I display dates (suppress redundant dates).

There are several where we're showing information about legislators. We'll probably want to discuss what pieces of information should be shown separately for each ICPSR entry, and which should be shown only once for each person. I'm opening this issue to track any conclusions we come to on this.

In places where we show ICPSRs separately, it may be helpful explain that such 'members' are a statistical construction, and perhaps an explanation of the reasons for why we scale party-persons individually.

The slope/intercept are calculated directly from mid/spread. It seems redundant to keep them both in the db - i'm likely to forget to update them somewhere and we'll get inconsistency. Would it be ok to make this routine a view thing instead of including it in the data model?

def add_endpoints(self):
    """Add attributes to nomimate attribute that aid in
    drawing cutting lines
    """
    self['nominate']['slope'] = None
    self['nominate']['intercept'] = None
    if (self['nominate']['spread'][0] == 0 and
            float(self['nominate']['spread'][1]) == 0 and
            float(self['nominate']['mid'][0]) == 0 and
            float(self['nominate']['mid'][1]) == 0):
        self['nominate']['x'] = [0, 0]
        self['nominate']['y'] = [0, 0]
    elif abs(float(self['nominate']['spread'][1])) < 1e-16:
        self['nominate']['x'] = [float(self['nominate']['mid'][0]),
                                 float(self['nominate']['mid'][0])]
        self['nominate']['y'] = [-10, 10]
        slope = 1000
        intercept = -slope * (float(self['nominate']['mid'][0])
                              + float(self['nominate']['mid'][1]))
        self['nominate']['slope'] = slope
        self['nominate']['intercept'] = intercept
    else:
        slope = -float(self['nominate']['spread'][0] /
                       (float(self['nominate']['spread'][1]) *
                        self.dimweight * self.dimweight))
        intercept = (-slope * float(self['nominate']['mid'][0])
                     + float(self['nominate']['mid'][1]))
        self['nominate']['x'] = [10, -10]
        self['nominate']['y'] = [intercept + slope * xx for xx in
                                 self['nominate']['x']]
        self['nominate']['slope'] = slope
        self['nominate']['intercept'] = intercept

We now depend on jpegoptim on the dev server to pre-compress our images. Ubuntu's default version is 3 years old, and we should upgrade to 1.4.2 or higher for better performance and addition of a flag to preserve permissions (currently I need to re-chmod every file after the compression script runs).

On both the search_list and the individual votes the "passed" or "failed" label is simply determined by whether the yea_count is greater or less than the nay_count. This isn't quite correct.

Some rollcalls have "majority_requirement" which denotes the needed support to pass but not all votes have this. It could also be possible to use the vote_question as a rough guideline for this given we create a dictionary of questions -> needed votes. Again, though, vote_question is not on old rollcalls.

1. Go to congress page
2. Select a region of voters, observe as they are filtered from the list
3. Now sort by anything, observe as everyone else reappears.

Currently, when you do a full text search and a word score is returned, the date sorting does not work. Instead we should have the option to sort by word score, newest, and oldest. the word score option should only display when a full text search has been completed.

This is an open issue to track our db schema change.

voteview_rollcalls

• synthID to date_chamber_rollnumber
• result to vote_counts
• votes: {id, v, p} to votes: {icpsr, cast_code, prob}
• Add prob to votes @lukesonnet
• resultparty to party_vote_counts -- and keys change from party labels to party numbers
• code to codes
• nay to nay_count
• yea to yea_count
• pre and classified fields missing from nominate @lukesonnet
• keyvote to key_flags
• support to percent_support
• shortdescription to short_description
• New description waterfall: vote_desc, vote_document_text, description, short_description, vote_question, question
• Fix indices @adamboche

voteview_members

• stateAbbr to state_abbrev
• bioName to bioname (lowercase)
• party to party_code
• districtCode to district_code
• bio to biography
• stateName, cqLabel, startdate, lastmeans, occupancy, state, partyName, geo, enddate, dimweight, stateCode ALL REMOVED -- verify OK
• Add born and died @adamboche
• Crosswalk social fields @aaronrudkin
• wikiStatus to wiki_status
• Nominate: oneDimNominate to dim1; geoMeanProbability to geo_mean_probability; numberOfVotes to number_of_votes; twoDimNominate to dim2; logLikelihood to log_likelihood; numberOfErrors to number_of_errors.

Reimport voteview_parties

Web code check

• Search votes rewired
• Download votes rewired
• Vote view page rewired
• Member bio page rewired
• Congress page rewired
• Parties pages rewired
• All code pointing to test_voteview
• Drop voteview db and renamed test_voteview to voteview
• All code pointing to voteview
• Deployed to production server

It should be possible to search on a member's page for a subset of their votes. This search should also include a "load" button to load your current stash.

We should have a page where you click on a district (i.e. Massachusetts 1st), see:
a) Everyone who represented the district historically
b) A nominate map of where those people were ideologically
c) Maybe some sort of map to show how the boundaries change over time.

I just ran across this map code, posting it here in case it's helpful for anything.

http://bl.ocks.org/rveciana/fe6b452c853146e674dd6dd09c1cc6e3

Jeff expressed a desire to make a page where you can enter your own location, we use a geolocation IP (Google Maps) to get a lat-long pair for your location, then we use our maps to figure out what districts your location was in historically, so you can see who would have represented you through history.

I think this probably goes well with #17.

If you want to release this software it may be a good idea to include a LICENSE file

Cannot figure out why the tooltip isn't working. Identical code to other tooltips that do work on that page and other pages. Maybe something in the javascript or css that I couldnt find out.

No idea why this is the case. Something between the code that builds the reduced vote objects and the code that sorts/groups breaks.

We should have a small, square logo next to our voteview.com header text. This could double as a favicon (the current favicon is an American flag).

This issue can serve as a place to brainstorm logo ideas. One option is a wordmark (i.e. a symbol built around "vv" in some way). I find these fairly messy. Another option is something that plays with the notion of cut-lines or scatter-plots or graphing in some way. Maybe something like plot axes with diverging red and blue lines to give a nod to the polarization literature? The cool thing about plot axes as a logo component is that whatever you put on them suggests motion, which is good.

DC.js has a nice wordmark. I don't know if the weird diagonal shading makes any sense, but the core elements of axes (suggesting graphing) and the dc wordmark are nice:
https://dc-js.github.io/dc.js/

Ideally this should be something that is small, simple, relatively flat (i.e. no shading), not too many colours, easy to vectorize, colour-blind friendly, colours that work fine with the site's UCLA blue scheme, works in greyscale and coloured, square, visible at low resolution and high resolution.

Consider this person's biography (Charles Sumner):
http://voteview.polisci.ucla.edu/person/9083

Sumner's biography has a variety of interesting facts in it, many of which relate to other things we have pages for. For example, he's "one of the founders of the Free Soil Party in 1848". He was "removed as chairman of the Committee on Foreign Relations in 1871 as a result of differences with President Ulysses S. Grant over policy in Santo Domingo". He was "assaulted in the Senate chamber by Representative Preston Brooks of South Carolina on May 22, 1856".

Ideally, Sumner's biography would link to the Free Soil Party, Ulysses S. Grant, and Preston Brooks.

Doing this live is fairly complex but doing it offline would be fairly feasible, and because biographies are static for a fairly long time, I think worth doing.

The link to sort does not update to reflect the new chamber. Update this in congress.js.

I have assigned Erik to give us one paragraph summary party descriptions that hit major notes (period active, historical context--who did they come from, who did they become when they collapsed, key issues or salience, regions active, notable individuals associated with party, etc.)

I will use this issue to track his progress. Further, we should cross-link as much as we can from the descriptions so that someone clicking on the Whigs, for example, will be able to click through to the Republicans.

• Code for displaying descriptions
• Get descriptions
• Code for cross-linking descriptions

http://128.97.229.160/rollcall/RH1141202

Woopsies, floating off the end of the page to the right.

For the purposes of colour commentary and education, it would probably be a good idea to get some photos associated with key votes. It is easy to imagine someone coming to the page, searching "ObamaCare", clicking on the vote, and wanting some context.

This is obviously a manual task, incredibly open ended, and could soak up just an enormous amount of resources, so I think maybe the better thing to do would be to complete #15 and then subset our list to the most important votes, develop infrastructure to detect and display the photos, and then farm out the work at a later period to research assistants.

Also, while the biographical photos don't largely have fair use issues, substantive photos quite possibly would, so it'd be important to train research assistants to find stuff with no rights issues.

The footer sticks to the bottom of pages that you have to scroll to, but does not stick to the bottom of pages that take up less room than the window on your browser. Compare the parties page to the data page, for example.

I tried to fix this in base.tpl some time this summer but I couldn't make it work.

The presidents do not have bios, because they are not in the congressional guide. We should source short bios somewhere and scrape them (excluding, perhaps, the current president?) to solve this.

Everything on the data page should be classified based on whether or not we wish to continue producing it or not. Everything we plan to continue producing needs to be linked instead of broken.

UNCLASSIFIED

• DW-NOMINATE Program With Examples
• DW-NOMINATE Probabilities for Legislator Choices: 1st to 112th Congresses (huge file, not sure if we should maintain? People can get this through the main interface if they really want to)

PRODUCE EVERY UPDATE

• Percent Voting on the Winning Side by Member -- Houses/Senates 1 - 112 (memberCSV)
• Party Unity Scores by Democrat and Republican Members: 1857 - 2012 (memberCSV)
• Common Space DW-NOMINATE Scores 1st to 112th Congresses (memberCSV, rollcallCSV)
• Cutting Line Angle Files for the House and Senate 1st to 112th Congresses (rollcallCSV)
• Party and Chamber Medians, 1 - 112 Congresses (DW-NOMINATE Scores) (partiesCSV)
• Party Unity Scores by Congress: 1879 - 2012 (partiesCSV)
• Political Polarization Measures: 1879 - 2012 (either as a polarization Rmd or a congress-chamber level csv?)
• ORD file for present congress

PRODUCE EVERY SESSION OF CONGRESS

• DW-NOMINATE Scores 1st to 112th Congresses (fixed after the end of the last congress, only updated every two years)
• One-Congress-at-a-Time DW-NOMINATE (Nokken-Poole) data for the House and Senate 1st to 112th Congresses
• ORD files for past congresses

MAINTAIN ONLY AS LEGACY KEITH FILES

• Rank Orderings all Houses and Senates -- 1st to 112th Congresses
• Clausen, Peltzman, and Issue codes for 1st to 113th Congresses

Static text files that will never change should be put in /static/data/. Legacy files should be put in /static/data/legacy/. Ongoing files should output to /static/data/current/. I will write a cronjob to automagically back up the current folders each week.

The template itself is /var/www/voteview/views/data.tpl. Feel free to edit it and commit changes.

Please use comments to either suggest placement for these files, place them, or note when you've created a script to output them.

The code to display vote probabilities on user pages is complete, but we have no vote probabilities. Currently votes are a list of 2-tuples under the key "votes"; each 2-tuple consists of the keys "id", and "v". My display code expects a third key, "p" containing a 0-100 float corresponding to the probability of the person making that vote.

Adam expressed willingness to take this on but was not sure where to get the data for this from.

(Note, this description refers to the old schema; the keys are renamed in the new schema, but the main thing is just getting them in the database).

Currently we have photos of every member of congress going back to 1985 and all but 2-3 back to 1980. This exhausts the bioguide, as well as Wikipedia for anyone who has a wiki page that's the same as their name.

Jeff says he thinks we have further photos from a past incarnation of WVV, but those files have not as of yet been located.

This issue should track our progress locating further member photos. Perhaps this could be done by hiring some undergrads and having them work. We have a script to generate which members we're missing--we could just give them a CSV that has five columns per row: ICPSR, name, state, district, photo URL. I can automate the code side of ingesting and converting whatever photos they find.

I think it makes sense to try to fill the missing photos moving backwards, both because recent members are more "important" in some respects, and because I think more recent members are going to be easier to find.

Currently we have 8754/12288 photos. You can calculate the numerator by running ls -altr /var/www/voteview/static/img/bios/ | wc -l and subtracting 3. You can calculate the denominator by running db.voteview_members.distinct('icpsr').length in mongo.

• Done back to 1970
• Done back to 1960
• Done back to 1950
• Hit 10,000 ICPSR-photos

I have written code to read keyvote indicators from the database (old schema, not new schema). This will not be useful if no one marks anything as a key vote. As a result, someone should write a script as part of our overall scraping workflow to assign key vote status to certain votes.

In the old schema, assigning a key vote status is done by adding a "keyvote" key to voteview_rollcalls, which is a list containing strings. Example:
"keyvote": ["cq", "gov", "vv"]

You can also use this issue to suggest new keyvote values, and I will add the mappings to the display code.

Searching the main page with an invalid field does not pass back the error message cleanly.

I assigned colour schemes to parties earlier in development. The colour schemes are: red, orange, yellow, green, teal, blue, purple, pinkpurple, grey, brown

You can see which colour each party is in based on the little left border strips in the party list, or by clicking into the party page and observing the scheme used:
http://voteview.polisci.ucla.edu/parties/all

I don't have the substantive knowledge to know if my colour choices were at all appropriate. The idea was to ensure that concurrent major parties would use different colours. Beyond that, I didn't really have any priorities.

This issue is to allow people to review my colour choices and suggest if anything should be in a different colour category.

In addition, we should consider reallocating people off yellow; yellow is not recommended for use on a white background. It's the only palette that doesn't come from ColorBrewer. It's generally not recommended. It's bad on low contrast screens, it goes bad with f.lux, etc.

I'll close the issue if no one has any input. In addition, if Erik finds anything out about historical party colours in his research, I'll set them accordingly.

On the main page, open the advanced search. Wait for a suggestion to pop into the text box. If there is a suggestion in the box when you click a checkbox or enter a number into an input field it sends the suggestion as if it were the user query.

To nullify this we could:

• have no suggestions when the advanced pane is open
• clear the search bar when facet is clicked if nothing has been manually entered in to box
• turn off suggestions all together, or display them somewhere besides in the search bar itself
• don't allow the field to be submitted if the search bar has never been in focus

Add party page for each party that plots features of that party over time and across its members (perhaps also a map showing where its members came from)

Search fails on unicode input. Try searching "gutiérrez".

Edit member pages such that listed votes show choice made, probability of choice, and agreement (or not with party)