dd5ht / image_downloader Goto Github PK

This solution is super fast, handles errors and malformated entries without a problem and ignores duplicates.

#! /usr/bin/env bash
file="$1"
while read p; do
  wget -nv -N "$p"
done <$file

Usage: $ ./download.sh samplefile.txt

Usage: $ ./image_downloader.py samplefile.txt

Testing: $ pytest

For manual testing a testserver can be used like this: $ python3 -m http.server 60451 --bind 127.0.0.1

This type of program can easily be parallelized by splitting the input file into multiple smaller files and iterating over each separately.