jhnwr / basic-html-scraper Goto Github PK

hi sorry for bothering you just starting with this scraper. everything works fine until i add

urls = get_product_links(1)
for url in urls:
print(parse_product(url))

after that ill get

Traceback (most recent call last):
File "/Users/scrap.py", line 29, in
print(parse_product(url))
File "/Users/scrap.py", line 17, in parse_product
r = s.get(url)
File "/Users/scrap/venv/lib/python3.8/site-packages/requests/sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
File "/Users/scrap/venv/lib/python3.8/site-packages/requests/sessions.py", line 573, in request
prep = self.prepare_request(req)
File "/Users/scrap/venv/lib/python3.8/site-packages/requests/sessions.py", line 484, in prepare_request
p.prepare(
File "/Users/scrap/venv/lib/python3.8/site-packages/requests/models.py", line 368, in prepare
self.prepare_url(url, params)

i know its easy for you but i really dont see it :(

Question:

Any idea how to implement the scraper with a dynamic list?

Example:
Members in this website
https://connect.medteq.ca/en/directory