expersso / gesis Goto Github PK

View Code? Open in Web Editor NEW

15.0 7.0 1.0 40 KB

R Client for GESIS Data Catalogue (DBK)

R 100.00%

webservice-client r-package rstats research-data gesis

gesis's Introduction

Gesis

Introduction

The GESIS Data Catalogue offers a repository of approximately 5,000 datasets.

To install the package from github:

# install.packages("devtools")
devtools::install_github("expersso/gesis")

library(gesis)

A simple example

We start by listing all available groups of studies:

groups <- get_study_groups()
head(groups, 10)

##    group_no                                        value
## 1      0001 International Social Survey Programme (ISSP)
## 2      0002                     EB - Flash Eurobarometer
## 3      0003                               Travel Surveys
## 4      0004                            Time Budget Study
## 5      0005       EB - Central and Eastern Eurobarometer
## 6      0006       EB - Candidate Countries Eurobarometer
## 7      0007                                       ALLBUS
## 8      0008      EB - Standard and Special Eurobarometer
## 9      0009                  European Values Study (EVS)
## 10     0010                               Politbarometer

We see that the Eurobarometer has study group number 0008 Let's looks at all available Eurobarometer waves:

eurobars <- get_datasets("0008")
head(eurobars)

##    doi                           title
## 1 0078 Attitudes towards Europe (1962)
## 2 0626 European Communities Study 1970
## 3 0627 European Communities Study 1971
## 4 0628 European Communities Study 1973
## 5 0986  Eurobarometer 2 (Oct-Nov 1974)
## 6 0987      Eurobarometer 3 (May 1975)

We would now like to download the first three studies. We first need to log in to the Gesis website and then pass the DOIs (unique data set identifiers) to download_dataset:

# username and password stored as environment 
# variables "GESIS_USER" and "GESIS_PASS"
gesis_session <- login()

if(!dir.exists("downloads")) dir.create("downloads")
download_dataset(s = gesis_session, doi = eurobars$doi[1:3], 
                 path = "downloads", filetype = ".dta")

## Downloading DOI: 0078

## Downloading DOI: 0626

## Downloading DOI: 0627

(files <- list.files("downloads", full.names = TRUE))

## [1] "downloads/ZA0078_v1-0-1.dta" "downloads/ZA0626_v1-0-1.dta"
## [3] "downloads/ZA0627_v1-0-1.dta"

We can also download the codebooks for the same studies:

download_codebook(eurobars$doi[1:3], path = "downloads")

## Downloading codebook for DOI: 0078

## Downloading codebook for DOI: 0626

## Downloading codebook for DOI: 0627

Using the haven package we can now read the data sets:

library(haven)
df <- read_dta(files[1])
dim(df)

## [1] 4774  175

Disclaimer: the gesis package is neither affiliated with, nor endorsed by, the Leibniz Institute for the Social Sciences. I have been unable to find any indication that programmatic access to the website is disallowed under its terms of use (indeed, its guidelines appear to encourage it). That said, I would discourage users from using the gesis package to put undue pressure on their servers by initiating unnecessary (or unnecessarily large) batch downloads.

gesis's People

Contributors

Stargazers

Watchers

Forkers

fsolt

gesis's Issues

argument name doi can be misleading

download_codebook and download_dataset both have an argument called doi and the description of that argument is "The unique identifier(s) for the data set(s)". This may be misleading/confusing for users as the number that is needed here is the study number (i.e., the number that comes after "ZA" in the dataset title in the DBK). For example, the DOI for ZA4587 is 10.4232/1.13048 (as specified in the bibliographic citation for that dataset), but the "doi" required by the gesis package functions is "4587". Hence, to avoid potential confusion it might help to change the name and description of the argument. One suggestion that I can think of could be argument doi = study (or studynr) + Description = "Study number(s) for the data set(s). The number that comes after "ZA" in the DBK study title".
For consistency, this would also require changing the column name created by the get_datasets function from doi to study or studynr or whatever else is used as an argument for download_codebook and download_dataset.

Add ability to batch download codebooks

Suggested by @fsolt (#1).

Should probably rename browse_codebook to e.g. get_codebook, then allow options such as {"browse", "download", "url"} to give user flexibility.

GESIS identifiers should always be 4-digits long

Kinda related to #12.

The download_dataset function will fail if the "DOI" (the GESIS ID) is 3-digits long instead of 4-digits long, as e.g. with old Eurobarometer survey waves.

Example:

gesis::download_dataset(s = GESIS_SESSION, doi = 990, path = ".", filetype = ".dta")

This will fail unless 990 is converted to 0990 first. In fact, the same thing happens if the "DOI" / GESIS number provided by gesis is used to access the studies online:

https://dbk.gesis.org/dbksearch/SDesc2.asp?db=E&no=990 will 404
https://dbk.gesis.org/dbksearch/SDesc2.asp?db=E&no=0990 is the correct URL

A simple fix should be:

doi <- dplyr::if_else(doi < 1000, str_c("0", doi), as.character(doi))

I guess the issue was introduced by converting "DOIs" / GESIS IDs to integers.

Please let me know if you can reproduce the error, and I'll submit a PR.

I do not know what other functions are affected.

Provide explicit way to deal with stray zip files

Better than just warning the user of this in the help file.

Add additional mime type to setup_gesis. Probably also handle this in download_dataset as suggested by @fsolt (#1).

Resubmit to CRAN?

Hello,

I guess it's not a priority, given how easy it is to install packages from GitHub, but I was saddened to see that CRAN had removed your excellent package. Resubmit one day, perhaps?

(I confess I do not know if packages can be resubmitted after removal).

All the best,

François

Consider including a convenience wrapper

Suggested by @fsolt (#1).

Would wrap up existing functions, vectorize the downloading of data sets, and close the Selenium server. Important to avoid duplicating code.

download_codebook throws an error message

First of all, I wanted to thank you for creating this great package!
I am planning on using it in a workshop I will teach and noticed that download_codebook gives the error message "Error in is.url(url) : length(url) == 1 is not TRUE"
Could it be that this is because the particular study whose codebook I wanted to download has multiple files whose names include "_cdb"?
If you want to reproduce the error, the study number is "4587".

Add GitHub topics

webservice-client, r-package, rstats, research-data for example ;-)

download_dataset function fails: "Error in html_form(s)[[1]] : subscript out of bounds"

download_dataset(
   s = gesis_session,
   # example Eurobarometer waves
   doi = c("1036", "1037", "1320"),
   path = ".",
   filetype = ".dta"
)

Result:

Error in html_form(s)[[1]] : subscript out of bounds

Add Chrome support

Latest version of Firefox appears out of sync with (R)Selenium

dbk.gesis.org is dead

dbk.gesis.org is dead. Will there be an update to the package making it work again through search.gesis.org?

download_codebook fails for files without "_cdb"

Thanks for writing this package, it's very useful!

I have noticed that not all dois have a codebook section. So, there's an error when the download_codebook() function does not find one on the Gesis website. For example download_codebook(doi = c("5876", "5913", "5689")) evaluates the first doi and then stops because doi "5913" does not have a codebook on the Gesis website (although "5689" does and should be downloaded).

However, it would be great to have some documentation/codebook for the corresponding data. One solution would be to download the questionnaire for those cases and display a warning that a questionnaire was downloaded instead of a codebook. Another would be to find/construct a codebook from another online source (like ICPSR).