[QUESTION] - It should only follow, if a redirect happens about magpidownloader HOT 11 CLOSED

For example,

#!/bin/bash

download_pdf(){
        tmp=tmpfile.$$
        wget $URL -O $tmp
        if [ $? -ne 0 ]; then
                echo "Download error. check URL"
                echo " URL="$URL
                return
        fi
        file_type=(`file $tmp`)
        if [ ${file_type[1]} == "PDF" ]; then
                mv -f $tmp "${TITLE}.pdf"
        else
                pdf_url=`cat $tmp | grep c-link | sed 's/^.*href=\"//' | sed 's/\?.*$//'`
                wget $pdf_url -O $tmp
                if [ $? -ne 0 ]; then
                        echo "Download error. check Page Structure."
                fi
                mv -f $tmp "${TITLE}.pdf"
        fi
        rm -f $tmp

}

URL="https://magpi.raspberrypi.org/books/camera-guide/pdf"
TITLE="Raspberry Pi Camera Guide"

download_pdf

URL=https://www.raspberrypi.org/magpi-issues/Essentials_AIY_Projects_Voice_v1.pdf
TITLE="Essentials AIY Project Voice Ver1"

download_pdf

exit 0

This code has the issue that it always is downloaded.

from magpidownloader.

#!/bin/bash

download_pdf(){
         if [[ $URL =~ ^.+\.pdf$ ]]; then
                # text end is .pdf
                wget -N $URL
        else
                pdf_url=`curl -sf $URL | grep \"c-link\" | sed 's/^.*href=\"//' | sed 's/\?.*$//'`
                wget -N $pdf_url
        fi
}

URL="https://magpi.raspberrypi.org/books/camera-guide/pdf"
TITLE="Raspberry Pi Camera Guide"

download_pdf

URL=https://www.raspberrypi.org/magpi-issues/Essentials_AIY_Projects_Voice_v1.pdf
TITLE="Essentials AIY Project Voice Ver1"

download_pdf

exit 0

from magpidownloader.

If $page_url is the link which is not forwarded, I think

pdf_url = $page_url

Am I misunderstanding?

from magpidownloader.

First of all: thank you a lot, for answering so fast, I really appreciate it.

Maybe I asked wrong:

let's say take the special issue list:
https://github.com/joergi/MagPiDownloader/blob/master/linux_mac/magpi-special-issue-downloader.sh
Let's say, if I don't want change the url to the new standard.
so let's say, I still want to use:
https://www.raspberrypi.org/magpi-issues/Essentials_Games_v1.pdf
and the new one looks like:
https://magpi.raspberrypi.org/books/camera-guide/pdf

And I want to use the script you provided for both versions.

printf -v page_url "https://magpi.raspberrypi.org/issues/%02d/pdf" $i
pdf_url=`curl -sf $page_url | grep c-link | sed 's/^.*href=\"//' | sed 's/\?.*$//'`
wget -N $pdf_url -P $OUTDIR

for the first url I only would need pdf_url=curl -sf $page_url but for the 2nd url I would like to use the complete line.
is there a easy way to write it? I hope you understand what I want.

The story behind it:
I'm making at the moment a downloader script -> https://github.com/joergi/downloader/blob/master/linux_mac/downloader.sh
this should cover everything, all the magpi issues, but also some downloaders with simple pdf links, where `grep c-link | sed 's/^.href="//' | sed 's/?.$//'`` this is not needed.
I have at the moment 4 repos which works similar to the MagPi, and 2 who have a list of normal direct pdf links.
But I want just push the links to the downloader and the downloader will take care of it....

I hope this helps you more

from magpidownloader.

or in other words: how can I check if it's forwarded, as in the MagPi Downloader?

from magpidownloader.

I think

1. Download the page contents.
2. Check it
   2.1 if it is PDF, then end
   2.2 if it is HTML, then search c-link tag and, get new_url and download it.

from magpidownloader.

ok, no idea yet, how to, but I will find out...
but if you already have already the code in your head, let me know ;-)

from magpidownloader.

hmmm the code itself looks good. Thanks a lot.
But this means, it will always try it to download....
Maybe I will just make 2 versions, a downloader to follow the urls (as for MagPi issues) and a normal one without it.

from magpidownloader.

Always download means that wget withouth -O option can check the timestamp between local and Web site.
But wget with -O option gets the contents always.

from magpidownloader.

Wouldn't be a more simpler way, just a check if the URL ends with a ".pdf" or similar. If not, try the follow one.
Should work for most of the cases I get.

from magpidownloader.

that's lovely.
I will close it, but I will definitely use this for the downloader.
thanks a lot @keni-n 👍

from magpidownloader.