Error in open.connection(x, "rb") about ngramr HOT 4 CLOSED

Hi Aurelie

I think that the problem you are running into is that the Google rate limits the requests to the ngram viewer: if you hit it with too many requests it will block your IP address's access for a period of time (if you visit https://books.google.com/ngrams after your error you'll find it says "Please try again later").

I will look into providing a more informative error message, but the most constructive suggestion I would have is to break your work into chunks. Since you managed to get through to 75, you could try looping 1:50, saving the results and then 51:100 and combining what you get.

Alternatively, you can download the full ngram sets here: https://storage.googleapis.com/books/ngrams/books/datasetsv3.html. However, you'll have to work through them yourself as I haven't written any code to help with that!

Sean.

from ngramr.

Aurelie,

I have had a closer look and here is a quick and dirty fix, adding some periods of "sleep" to avoid hitting the rate limit:

google_ngram_tmp <- NULL

for (ixs in chunk(1:nrow(Ngram_data), 50)){
  for (i in ixs) 
  {
    # launch Google Ngram query and extract the results that are displayed with viewer
    google_ngram_tmp <- bind_rows(google_ngram_tmp, 
                                  ngram(Ngram_data$Xgram_pattern[i], 
                                      corpus = "fre_2019", 
                                      year_start = 1800,
                                      year_end = 2019,
                                      smoothing = 0,
                                      case_ins = TRUE,
                                      aggregate = TRUE)
    )
    
    # returns the Ngram index at each iteration to keep track of the computation
    print (paste("Ngram index", i, sep = ": ") )
  }
  print("Sleeping for a while...")
  Sys.sleep(120)
}

class(google_ngram_tmp) <- c("ngram", class(google_ngram_tmp))

This could be improved (e.g. there's no need to sleep after the last loop), but it does the job.

Separately I noticed that, despite all of this hard work, most of these phrases will give no results (I only found results for four of the phrases). It's worth noting that there are no ngrams in Google's collection that are more than 5 words long, and some of these phrases may simply not appear. Did you mean to use Xgram_dependencies instead?

from ngramr.

Hi Sean,

thank you very much for your fast reply.

This was really helpful, because I was looking towards an R problem, and did not even think about a webpage rate limit.

As you suggested, I added a sleeping period at the end of each loop, and it works perfectly (I had to add a longer period of 600sec though) 👍

To answer your question, yes, I am aware that most of the Xgram_pattern entries are either too long or do not appear in the google database. This is actually part of my research question, which aims at comparing pattern vs. dependencies. So of course, I also plan to run a similar piece of code with Xgram_dependencies, which will be as easy as pie thanks to your code :-)

BTW, I should also mention that the using your library allows me to run some fancy analysis I gave up on because I did not have enough computer storage to load the entire ngram dataset.... Thanks again for this nice piece of code!!

Cheers

• Aurelie

from ngramr.

Happy to help Aurelie - I'm glad the package is proving useful.

from ngramr.