alexgand / springer_free_books Goto Github PK

View Code? Open in Web Editor NEW

1.6K 49.0 366.0 164 KB

Python script to download all Springer books released for free during the 2020 COVID-19 quarantine

License: GNU General Public License v3.0

Python 93.91% Dockerfile 1.16% Batchfile 4.94%

springer_free_books's Introduction

The script is not working anymore, due to the reCAPTCHA step Springer added.

Python script to download all Springer books released for free during the 2020 COVID-19 quarantine

Books available

It used to have 409 english books (it took about 4 hours to complete the download ~ 14 GB, both PDF and EPUB, on my machine), but Springer has since removed some of them from the open access.

Usage

Note: If you want just one or two specific books, get the excel file and manually download those or use the script to help you download (see Download book selectively sub-section).

By default the script stores the books in ./downloads subfolder according to the subject ("English Package Name" column of the excel file).

Download all English books (PDF and EPUB)

Use the following command to download all PDF and EPUB books to the default download folder ./downloads

python3 main.py

To download them to a subfolder of your choice, say books

python3 main.py -f ./books

You can download to an absolute path, say C:/ebooks/springer/

python3 main.py -f C:/ebooks/springer/

To download books in separate chapters (PDF only)

python3 main.py --dl_chapters

Download all German books (PDF and EPUB)

To download the German books use

python3 main.py --lang de

This language flag can be combined with any of the above and below mentioned options. If not specified, --lang en, i.e. English, is the default.

Download all books of specific format

To download all PDF books only, run

python3 main.py --pdf

or all EPUB books only

python3 main.py --epub

Download only specific book categories

You can download selective books by categories. Make sure category name with space(s) is enclosed in double quotation marks, like so

python3 main.py -c Engineering "CoMpuTer sciEnCe"

The category name is case-insensitive.

Download book selectively

You can download books by indices (refer to the excel file for the titles)

python3 main.py -i 2 345 70 90 0 213

Verbose mode

If you need to know what you are downloading, enable the verbose mode with -v option

python3 main.py -v -c "Behavioral science and psychology" "chaos AD" UFO "Life science" -i 2 34 400

You will get the following summary before the downloading starts

                                          Book Title               English Package Name
2                                  All of Statistics         Mathematics and Statistics
34                            Essential Astrophysics              Physics and Astronomy
37       International Perspectives on Psychotherapy  Behavioral Science and Psychology
40                         Applied Behavior Analysis  Behavioral Science and Psychology
70                          Psychology of Perception  Behavioral Science and Psychology
95             Handbook of Consumer Finance Research  Behavioral Science and Psychology
311       Clinical Methods in Medical Family Therapy  Behavioral Science and Psychology
313                           Motivation and Action   Behavioral Science and Psychology
381                          Perceptual Organization  Behavioral Science and Psychology
396  Evidence-Based Practice in Clinical Social Work  Behavioral Science and Psychology
397                 Foundations of Behavioral Health  Behavioral Science and Psychology
398                      Social Psychology in Action  Behavioral Science and Psychology
400             A Course in Rasch Measurement Theory                          Education

13 titles ready to be downloaded...
The following invalid book categories will be ignored:
 1. chaos AD
 2. UFO
 3. Life science

Running in a virtual environment (Bash commands):

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python3 main.py

Virtual environment on Windows (auto-detects Python version):

Just double-click the run_VirtualEnv.bat file from Windows Explorer, or type-in its name at the command-line and press ENTER. Do install Python first, either from python.org or from the Microsoft Store for Windows 10.

Virtual environment on Windows (Python 3.x):

python -m venv .venv
.venv\Scripts\activate.bat
pip install -r requirements.txt
python main.py

Virtual environment on Windows (Python 2.7.x):

python -m virtualenv .venv
.venv\Scripts\activate.bat
pip install -r requirements.txt
python main.py

Sources:

Thanks Springer!

Docker container to do all the work

Considering you cloned the repo in a folder, as:

cd \home\[user]\workspace
git clone https://github.com/alexgand/springer_free_books.git
cd springer_free_books
mkdir downloads

You can run the scripts like that:

docker build . -t springer-image
docker run --rm -v [local_download_folder]:/app/downloads springer-image

and the downloads will be at [local_download_folder].

Troubleshooting

On macOS, if the script produces a [SSL: CERTIFICATE_VERIFY_FAILED] error, then security certificates may need to be installed. Instructions for doing so can be found here. If that doesn't work, see alternate instructions here.

springer_free_books's People

Contributors

Stargazers

Watchers

Forkers

jangirshubham cyriltw behyeefatt diveshkr-code viidix vikash-kothary ngoubou seyedrezamirkhani rahigeramifar parthagar py9mrg pjcrosbie torrilira nmimoto ledudu nrsyed hunor87 jackmitcheltree jacqhen siddharthgolecha marlonchalegre restrepo ro91 thuh2305 justinallennz aserun c6p luismiguelmoreno susmuffin alexgv99 sharondenisse jqueguiner hathagat 24093 franzforstmayr pjungermann parampavar jmanforce mingumckevin jerjohn15 jrt2030 bhadet alexnguyen96 daleang demogorcus titannotsogreat jborza fdnk vspur 51g53gv haint morad1998 muradalshammari chaosad rizkyarlin shin-pete marcell666 robertsisaacp valrcs a2393439531 spiritsree orbitalz shaunstanislauslau anielshri darshanmahadev melloa proloy2018 vsujeesh claytonbarrozo yanfulin beautyhasnoshape prassorathia aswani-eedara debsen077 luannaqrl 15520529 shubha777 shah-aman kzeimpekis keerthidasari9 tocococa fperrotta broomva neagaze fullstackinfo nicolo-felicioni ytiam navaneeth20 bfrentz dmbrannon thearihant bardo1 abulho gassantos vamosraghava dikako tanthanadon birbilis noskid210021 pirasteh

springer_free_books's Issues

Problem with temporary directory in Windows OS

Inside helper.py the subfolder "./tmp" is not welcomed in my Windows 10
I have replaced the line path = create_path('tmp')
with path = create_path(os.path.join(book_path, 'tmp'))

I have added also a shutil.rmtree(os.path.join(book_path, 'tmp'))
at the end of the _download_book(url, book_path) function

I made the workaraound without inspecting the whole code. But it is working fine now in my Windows machine. I thinks it is also compatible with other OS's as I have used os.path.join without the os.path.sep, but I have not tested it.

Could you please elaborate more on readme section on "HOw to download perticular subject book download"

And also can i download books from perticular subject like machich learning or data science in one script?

error while trying to download

File "main.py", line 4, in
import requests
ModuleNotFoundError: No module named 'requests

If both pdf and epub are available, download only pdf

I noticed that there are several books which have both pdf and epub available. The current script downloads both, creating a lot of duplicate files. It would be nice to only download pdf if epub and pdf both are available.

Download doesn't complete

Mine is stopping at 63% (255/407).

Traceback (most recent call last):
  File "main.py", line 64, in <module>
    myfile = requests.get(new_url, allow_redirects=True)
  File "C:\Users\...\venv\lib\site-packages\requests\api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\...\venv\lib\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\...\venv\lib\site-packages\requests\sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\...\venv\lib\site-packages\requests\sessions.py", line 683, in send
    r.content
  File "C:\Users\...\venv\lib\site-packages\requests\models.py", line 829, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
MemoryError

Any guidance here?

possible bug in macOS

Trying to download all the books. After having installed the requirements, the process completes (in only ~1 minute) but there is not any book in the respective folders. macOS limitation? Is the code only prepared for windows?

Workaround just FYI

To circumvent redownloading aborted downloads you can just add:

import os
import curlify as cy

and replace the two open(new_folder+final, 'wb').write(myfile.content) statements with:

os.popen("{}{}\"{}\"{}".format(cy.to_curl(myfile.request), " -o ", new_folder+final, " -C -").replace("'", "\""))

Assumes a working curl binary nonetheless on your system.

object of type 'float' has no len()

I tried running the simple python3 main.py after installing the dependencies but I got the following error:

[~]:python3 main.py

387 titles ready to be downloaded...
Traceback (most recent call last):
File "main.py", line 95, in <module>
download_books(books, folder, patches)
File "/mnt/c/Users/bounc/Desktop/springer_free_books/helper.py", line 126, in download_books
longest_name = books[CATEGORY].map(len).max()
File "/home/mlindn/.local/lib/python3.8/site-packages/pandas/core/series.py", line 3630, in map
new_values = super()._map_values(arg, na_action=na_action)
File "/home/mlindn/.local/lib/python3.8/site-packages/pandas/core/base.py", line 1145, in _map_values
new_values = map_f(values, mapper)
File "pandas/_libs/lib.pyx", line 2329, in pandas._libs.lib.map_infer
TypeError: object of type 'float' has no len()

I am running this on Ubuntu 20.04 WSL

Use specific User-Agent to bypass Google captcha

Just tried this script (https://github.com/tempname1024/springer-dl) and it works well except some minor problems. I think it uses a specific UA with which the captcha won't appear.

HTTP Error 404 Not Found but I was able to manually download spreadsheet

~/repos/springer_free_books$ python3 main.py
Traceback (most recent call last):
File "main.py", line 37, in
books = pd.read_excel(table_url)
File "/home/ddarden/.local/lib/python3.8/site-packages/pandas/util/_decorators.py", line 188, in wrapper
return func(*args, **kwargs)
File "/home/ddarden/.local/lib/python3.8/site-packages/pandas/util/_decorators.py", line 188, in wrapper
return func(*args, **kwargs)
File "/home/ddarden/.local/lib/python3.8/site-packages/pandas/io/excel.py", line 350, in read_excel
io = ExcelFile(io, engine=engine)
File "/home/ddarden/.local/lib/python3.8/site-packages/pandas/io/excel.py", line 653, in init
self._reader = self._enginesengine
File "/home/ddarden/.local/lib/python3.8/site-packages/pandas/io/excel.py", line 402, in init
filepath_or_buffer = _urlopen(filepath_or_buffer)
File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.8/urllib/request.py", line 531, in open
response = meth(req, response)
File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
response = self.parent.error(
File "/usr/lib/python3.8/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]

Hi. Thanks a lot for this useful script. I tried to set it up on my macOS Catalina. I used the virtual environment by following your steps. The requirements have been installed. When I'm running the main.py using python3 main.py, I got the ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] like the error log below. How to fix it?

Python version

% python
Python 3.7.3 (default, Nov 15 2019, 04:04:52)
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin

Error log

Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1317, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/http/client.py", line 956, in send
    self.connect()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/http/client.py", line 1392, in connect
    server_hostname=server_hostname)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/ssl.py", line 412, in wrap_socket
    session=session
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/ssl.py", line 853, in _create
    self.do_handshake()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 38, in <module>
    books = pd.read_excel(table_url)
  File "/Volumes/Transcend2TB/Books/springer_free_books/.venv/lib/python3.7/site-packages/pandas/io/excel/_base.py", line 304, in read_excel
    io = ExcelFile(io, engine=engine)
  File "/Volumes/Transcend2TB/Books/springer_free_books/.venv/lib/python3.7/site-packages/pandas/io/excel/_base.py", line 824, in __init__
    self._reader = self._engines[engine](self._io)
  File "/Volumes/Transcend2TB/Books/springer_free_books/.venv/lib/python3.7/site-packages/pandas/io/excel/_xlrd.py", line 21, in __init__
    super().__init__(filepath_or_buffer)
  File "/Volumes/Transcend2TB/Books/springer_free_books/.venv/lib/python3.7/site-packages/pandas/io/excel/_base.py", line 342, in __init__
    filepath_or_buffer = BytesIO(urlopen(filepath_or_buffer).read())
  File "/Volumes/Transcend2TB/Books/springer_free_books/.venv/lib/python3.7/site-packages/pandas/io/common.py", line 141, in urlopen
    return urllib.request.urlopen(*args, **kwargs)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1360, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1319, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)>

Add the ability to download specific categories (column English Package Name)

It would be nice if you could run it like

python3 main.py --category "Mathematics and Statistics" --category "Physics and Astronomy"

and get only those categories downloaded.

Missing titles

The script throws an error on missing titles. An example:

FileNotFoundError: [Errno 2] No such file or directory:
'C:\Users\david\Desktop\springer_free_books-master\
springer_free_books-master\downloads\
Literature, Cultural and Media Studies\
An Anthology of London in Literature,
1558-1914 - Geoffrey G. Hiller, Peter L. Groves,
Alan F. Dilnot, 1st ed. 2019 - 978-3-030-05609-4.pdf'

The missing titles:
"Optimization of Process Flowsheets through Metaheuristic Techniques "

"Foundations of Analytical Chemistry"

"Clinical Methods in Medical Family Therapy"

"An Anthology of London in Literature, 1558-1914"

to fix it, remove the titles from the list one by one:
books = books[books["Book Title"]!=]

I think a try except would be better though.
try:
download_book(new_url, output_file)
except:
continue

Add LICENSE

You should add a license since multiple people are contributing to this project. Have a look at https://choosealicense.com/ if you're not familiar with the different open source licenses.

Personally I use https://choosealicense.com/licenses/agpl-3.0/ for all my projects.

AttributeError: 'Int64Index' object has no attribute 'array'

After ran python3 main.py -f ./books, I got the following error.

Traceback (most recent call last):
  File "main.py", line 70, in <module>
    books.index = [i + 2 for i in books.index.array]        # Recorrect indices
AttributeError: 'Int64Index' object has no attribute 'array'

python 3.7.5
ubuntu 19.10

Somebody please mirror and make a torrent

Somebody please try to collect all the files you can and make and seed a torrent of it, especially since some were removed. Seems dumb to rely on the script to keep working when the source is hostile (removing stuff, for example). Should have a backup for if one day it appears to have all disappeared from source.

Torrents are an amazing technology when they aren't being purposely neutered (private torrents) or opposed for near-sighted or misguided or otherwise stupid reasons

Problem with run_VirtualEnv.bat

Python 3.7
Win10 64bit

TypeError: object of type 'float' has no len()

HTTP Error 404: Not Found

The link to the Excel file seems to be broken. When you visit the link in main.py you get: {"projectVersion":"2.245.0-54832206c2d4e90345d71a4427e7542e623e43bf-2020-04-23_08:48:48.0010-local-1","requestUri":"https://resource-cms.springernature.com/springer-cms/rest/v1/content/17858272/data/v4","message":"com.springer.cms.service.ContentNotFoundException: No Content found for Version: 4 with Content-Id: coremedia:///cap/content/17858272","responseCode":404}

Check file after downloading and redownload if corrupt?

I used the script to download 14GB worth of files and more than 60% of them turned to be corrupt files to due to incomplete downloads.
Would be great if somehow we could check that the downloaded file is not corrupt.
If corrupt: reinitiate download.

excel file indices missing - can't download specific books

I want to download the epubs for a list of 38 books but there's no indices and row number has no relation to book downloaded.. please help

Duplicate downloading the pdf files as epub

When downloading everything with python main.py command (using python 3), noticed that the epub and pdf files are exactly the same.

Seems like the issue is similar to #36, although root cause might be different?

Nothing is happening after creating the first folder

Hi,
I tried running the script as is and the progress bar was stuck at 0%.

While debugging I noticed that requests.get() is not finishing. It keeps on running without finishing or time out. Not sure what can be done in this scenario.

Possible Introduction of reCaptcha by Springer?

Hi all,

I am not reporting an issue about the code, but just to share about what I encountered yesterday. While I was amending and testing the code, I was greeted with reCAPTCHA page. When I tried downloading manually, it indicated that the server was overloaded. However, the reCAPTCHA disappeared after about an hour or so later.

I believe Springer are aware that their free e-books are being downloaded by bots and has already put in place the reCAPTCHA, waiting to appear when overloaded with suspicious downloads. If we don't play nice, I reckon they might permanently enforce the reCAPTCHA downloading policy. Once that happens, then that will be the end to this automated download.

Please share the archive

Hello. Can anyone share a full archive of books on cloud or via torrent, with partial download capability? Currently, i cannot use this script for do this.

GUI? Available if you want one.

Awesome program!

I've added a GUI onto your code and replaced tqdm with a graphical progress meter that includes stats and a mechanism to cancel the operation. It only changed 7 lines in the code.

A GUI provides a number of benefits at little / no cost

Setting the location to save without modifying the source code
A progress meter with stats
- Shows overall progress
- Shows time remaining
- Ability to "cancel" the download in a safer manner than a Control C or killing the process
Popup error messages so that you can run the entire program without a console
Can distribute as an EXE file / py file that runs without a console

I forked your project and made the changes to my version. I didn't want to do a pull request as you may be fine with running on the command line, or maybe want to enable both modes by adding a "-gui" parameter when the program is started. It's your project so don't want to be presumptuous about any changes I made. If you're not interested, no problem, I'll just rename my version so there's no confusion.

GUI Screenshots

Startup

You can copy and paste the folder location into the window or use the Browse button.

After the download starts, the progress meter is shown that shows the current book as well as stats about time spent so far and time remaining (based on how long it's taken so far)

If you click "cancel" or close the progress meter window, then you'll be shown another popu-up window so that you understand why the download has stopped and the program exited

Possible confusing situation with "not responsive" message.

Because the GUI was added on top of code that blocks for long periods of time while downloading, Windows will sometimes display a "not responsive" message then the user tries to cancel the operation.

The problem could be solved by threading the downloading portion so that it doesn't simply pend for long periods of time, but the added complexity isn't likely worth it. A simple message "Your program hasn't locked up, it just looks busy during downloading" would explain the situation to users if desired.

raspian: numpy fails with libf77blas.so.3: cannot open shared object file: No such file or directory

sudo apt-get install libatlas-base-dev

fixes this

Import Error: No module names requests

I am getting following error while running main.py on version 2.7.15

λ python main.py
Traceback (most recent call last):
File "main.py", line 4, in
import requests
ImportError: No module named requests

I tried docker way as well and getting following error
λ docker run --rm -v E:/Tech/eBooks-PDFs-RefCards/springer_free_books:/app/downloads springer-image
/usr/bin/env: ‘python\r’: No such file or directory

Unable to install requirements

Python version: 3.5.2
OS: Ubuntu 16.04.5 LTS

After activating the venv, and launching the command "pip install -r requirements.txt"

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-c8xdz0d6/pandas/
You are using pip version 8.1.1, however version 20.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
`
No idea what I am doing wrong. Any help would be much appreciated. Thanks in advance

in windows : import pandas report error ImportError: DLL load failed while importing aggregations:找不到指定模块

today I down the latest code to run in windows,when I run "python3 main.py" directly,it dosen't work.(I forgot the error message..sorry,but it about pandas).
Then I use it in windows Virtual environment ,all package download completed, run the cmd as readme.md describe bellow:
python -m venv .venv
.venv\Scripts\activate.bat
pip install -r requirements.txt
python main.py

and then it report an error: DLL load failed while importing aggregations:找不到指定模块. and stop work.

I searched from internet and found the reason,pandas new version 1.0.2 and 1.0.3 forgot to import vs2015 runtime in it.
you have tow ways to fix it,
one is install vs2015 runtime in windows youself.
another is to rollback to pandas 1.0.1 ,cmd :"pip install pandas==1.0.1".
I used the second way to fix it.

Problem with EPUBs

There seems to be a problem downloading EPUBs.

Link to the downloaded books

The script is not easy to run to completion if downloading all books. Finally was able to finish downloading on my fifth attempt. So to save people some time I'm sharing a link to the downloaded books.

Currently missing the last 13 titles on the excel file.

https://drive.google.com/drive/folders/1JC15m__PbPaowQ7k2zS1-Us72yvROCQs?usp=sharing

Syntax error

How do I overcome this error?

I have changed my download folder

No other changes done.

Disparity between book counts / total sizes

Hi! Great project!

After getting dependencies installed, the download went without errors. But I'm wondering about the result...

https://link.springer.com/search?facet-content-type=%22Book%22&package=mat-covid19_textbooks&%23038;facet-language=%22En%22&%23038;sortOrder=newestFirst&%23038;showAll=true says there are 473 books, 407 of which are in English.

The readme says there are "409 english books (14 GB, both PDF and EPUB)"

My download directory says there are 409 objects, totaling 6.5 GB. Digging into directories shows both PDFs and EPUBs.

Why is the size so much lower than in the readme?
If there are 407 books, how does that become only 409 files, when most books seem to be represented by 2 files each (PDF and EPUB)? (I would have expected more like 814 files, unless only a handful of books were available in both formats.)

memory error using 32 bit python in windows

Downloading book not in list

How do I download not in list from https://link.springer.com/books/a/1 ?

Is there a way to download?

I just found all the download links

The script is too slow, only 6 books took 4hours!!
i just found all the direct links you may need
You can copy the links to software like "IDM". It will save you a lot of time.

Where are the downloaded files?

Hi, thanks a lot for this program ... but I cannot find the downloaded files. I tried various versions of

docker build . -t springer-image
docker run --rm -v [local_download_folder]:/app/downloads springer-image

Downloaded files are not visible.

Hello,

I am not able to see the download content in my local directory. Is it download somewhere else?
I clone this repo and created the downloads folder in springer_free_books folder.
I used following command to download the content:

sudo docker build . -t springer-image
sudo docker run --rm -v downloads springer-image

After completion of 100% I am not able to see any of the book in the local folder which pointed in second command. Please help me to get the content?

Incomplete downloaded book considered valid after restarting script

For some reason if a user hit Ctrl+C or Ctrl+Break, the incomplete downloaded book would be considered a valid book and would not be replaced on restarting the script.

Also on restart, if a PDF file is present (but incomplete, due to Ctrl+C or Ctrl+Break), its EPUB will be skipped.

Error: probably not a valid book

Hello, do you have any ideas why it's not downloading any of the books? I have all the dependencies and the python code seems to be running, but ot doesn't download anything right now, skipping every book.

Does it mean that Springer recovered all the books from their site?

Thanks!

Download fails around 4%, "File name too long" & NameError: name 'time' is not defined

Python version: 3.7.2
OS: Ubuntu 16.04.5 LTS

I finally managed to get the script going, however after some downloads (aroud 4%) the script throws up a bunch of errors and exits, leaving the download half-way. Re-running the script results in the same issue repeatedly.

Any help in resolving this issue will be much appreciated. Thanks in advance.

`[Errno 36] File name too long: './downloads/Medicine/Primer on the Rheumatic Diseases - John H. Klippel, John H. Stone, L eslie J. Crofford, Patience H. Whi>

Problem downloading: Primer on the Rheumatic Diseases, so skipping it.
Traceback (most recent call last):
File "/usr/lib/python3.7/shutil.py", line 566, in move
os.rename(src, real_dst)
OSError: [Errno 36] File name too long: './tmp/-temp_file-.bak' -> './downloads/Medicine/Primer on the Rheumatic Diseases - John H. Klippel, John H. Sto>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/xuxo/dl/videos/_descargados_a_ordenar/Springer_Books/springer_free_books/helper.py", line 113, in download_selected_books
download_book(request, output_file, patch)
File "/home/xuxo/dl/videos/_descargados_a_ordenar/Springer_Books/springer_free_books/helper.py", line 90, in download_book
_download_book(new_url, output_file)
File "/home/xuxo/dl/videos/_descargados_a_ordenar/Springer_Books/springer_free_books/helper.py", line 83, in _download_book
shutil.move(tmp_file, book_path)
File "/usr/lib/python3.7/shutil.py", line 580, in move
copy_function(src, real_dst)
File "/usr/lib/python3.7/shutil.py", line 266, in copy2
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/usr/lib/python3.7/shutil.py", line 121, in copyfile
with open(dst, 'wb') as fdst:
OSError: [Errno 36] File name too long: './downloads/Medicine/Primer on the Rheumatic Diseases - John H. Klippel, John H. Stone, L eslie J. Crofford, Patien>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "main.py", line 73, in
download_selected_books(books, folder, patches)
File "/home/xuxo/dl/videos/_descargados_a_ordenar/Springer_Books/springer_free_books/helper.py", line 119, in download_selected_books
time.sleep(30)
NameError: name 'time' is not defined

ModuleNotFoundError: No module named 'requests'

When I click on run in Python with the main.py, I can't start the download

Problem with Docker/Raspberry Pi

I ran out of storage space on my MacBook, so I tried running this script on my Raspberry Pi instead. I was using Docker on my Raspberry Pi, So I decided to choose a way to create a Docker image by Dockerfile. However, this method failed.

After reviewing the log, I realized that some packages defined in requirements.txt had to be compiled to work in my environment. However, the base image applied to the Dockerfile is a "slim" version that lacks some elements needed for compiles. That's the cause of the error. So I solved the problem by changing the base image from python:3.8-slim-buster to python:latest. The Docker image created in this way took a long time to create, but it worked perfectly on my Raspberry Pi.

However, this simple solution has a little problem. The size of the pulled image becomes quite large. Therefore, this solution may be inconvenient for some people. This script is used to download a large amount of data, but it would still be better to have less download capacity. So I want to discuss this.

My environment:

Raspberry Pi 4 Model B 4GB (aarch64)
Ubuntu Linux 20.04 LTS (Focal Fossa) 64-bit
Docker 19.03.8

KeyError: 'content-length'

In some occassions (eg. in my case indices 295 and 331), the script abandons with "KeyError: 'content-length'". I printed the request.headers and it appears that specific key was not present.

Full stack trace
Traceback (most recent call last):
File "C:\Users...\OneDrive\Documenten\7. Source\Repos\springer_free_books\main.py", line 88, in
download_books(books, folder, patches)
File "C:\Users...\OneDrive\Documenten\7. Source\Repos\springer_free_books\helper.py", line 137, in download_books
download_book(request, output_file, patch)
File "C:\Users...\OneDrive\Documenten\7. Source\Repos\springer_free_books\helper.py", line 91, in download_book
file_size = int(req.headers['Content-Length'])
File "C:\Users...\AppData\Roaming\Python\Python37\site-packages\requests\structures.py", line 52, in getitem
return self._store[key.lower()][1]
KeyError: 'content-length'

Can't get the requirements to work

I'm not normally using python at all.
Aparently pip can't find any of the requirements.
(The machine is online via IPv6. I tested by also giving the machine some legacy IPv4. No improvement.)

How do I tell it where to look for these requirements?
(The requirements listed also don't exist as Debian packages to install them manually, without pip.)

Example:

/tmp# python3 -m venv .venv
/tmp# . .venv/bin/activate
(.venv) 
/tmp# pip install -r requirements.txt
Collecting certifi==2020.4.5.1 (from -r requirements.txt (line 1))
  Could not find a version that satisfies the requirement certifi==2020.4.5.1 (from -r requirements.txt (line 1)) (from versions: )
No matching distribution found for certifi==2020.4.5.1 (from -r requirements.txt (line 1))

[Feature Request] Springer's 1000 open-access books

Earlier this month Springer published their 1000th open access book:
https://www.springernature.com/gp/researchers/campaigns/celebrating-1000-open-access-books

A full CSV file, similar to the one released in the Covid-19 collection is available here:
https://link.springer.com/search/csv?facet-content-type=%22Book%22&package=openaccess

Having looked at download_books() from helper.py and the column headers of the CSV file from above, I think a simple rename of the headers should be enough. URLs are similar. However:

Electronic ISBN and English Package Name columns are not available. This will alter the folder structure
Further, Edition is limited to just publication years.

Is this something that is straightforward and can be integrated easily?

Running in Windows

I had to modify the commands to run in Windows to this:

python -m venv .venv
cd .venv/Scripts
activate.bat
cd ../..
pip install -r requirements.txt
python main.py

ERROR: Could not open requirements file

i cannot install requirements.txt.
im using win10, phyton 3.8

C:\Program Files\Python38>python3 -m venv .venv

C:\Program Files\Python38>.venv/bin/activate
'.venv' is not recognized as an internal or external command,
operable program or batch file.

C:\Program Files\Python38>.venv\Scripts\activate.bat

(.venv) C:\Program Files\Python38>pip install -r requirements.txt
ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

New.

Sorry I am new to Python and I do not understand how to download the files, could somebody or the author please help me, I would greatly appreciate it.

alexgand / springer_free_books Goto Github PK

springer_free_books's Introduction

The script is not working anymore, due to the reCAPTCHA step Springer added.

Python script to download all Springer books released for free during the 2020 COVID-19 quarantine

Books available

Usage

Download all English books (PDF and EPUB)

Download all German books (PDF and EPUB)

Download all books of specific format

Download only specific book categories

Download book selectively

Verbose mode

Running in a virtual environment (Bash commands):

Virtual environment on Windows (auto-detects Python version):

Virtual environment on Windows (Python 3.x):

Virtual environment on Windows (Python 2.7.x):

Sources:

Docker container to do all the work

Troubleshooting

springer_free_books's People

Contributors

Stargazers

Watchers

Forkers

springer_free_books's Issues

GUI Screenshots

Recommend Projects

Recommend Topics

Recommend Org