eed85 / webii Goto Github PK

View Code? Open in Web Editor NEW

This project forked from eed-solutions/webii

0.0 0.0 0.0 47 KB

Python 100.00%

webii's People

Contributors

webii's Issues

import vars module

WebII/web.py

Line 21 in c8e086d

#import global variables

How to import other Python files? - Stackoverflow

code to test here:

WebII/_test_.py

Line 1 in c8e086d

 sys.path.append(os.path.abspath("C:/Users/EricBrahmann/EED-Solutions by Eric Brahmann/Ideal Dental - Dateien Code/webII/vars_module")) 

Download all html files and save them to disk

Produkte auslesen Level 1

Produkte auslesen Level 0

Produkte 3 auslesen (webscarp downloaded htmls)

see #4

Clean Strings

\n & a lots of spaces are exported within Dataframe Produkt_1

Kategorien auslesen Level 2

Besonderheit - Einige Level 1 Kategorien haben unterkategorien, andere nicht
Entweder beim webscrapping abfangen oder per pandas im Nachgang bereinigen

Produkte auslesen Level 2 (download all htmls)

after #3 has been solved, all products shall be scrapped

Kategorien auslesen - 0

WebII/web.py

Lines 66 to 80 in b4dd4ab

 for item in all_cat0: 

 I+=1 

 print(item.Text) 

 sl0 = item.find_all("a") 

 print ('found ' +str(len(sl0)) + ' sublinks') 

 J = 0 

 for l0 in sl0: 

 has_link = l0.has_attr('href') 

 print('Hat link',has_link) 

 if has_link: 

 J+=1 

 print (l0.text,l0['href']) 

 print('cat read')

Encoding UTF-8

Beautifulsoup does not show Umlaute correctly
first try to fix this see
see #10

Kategorien auslesen Level 1

Set WD to OneDrive4B

use html saved in Sharepoint

WebII/web.py

Line 52 in 3a17ae1

r = requests.get(url);cr = cr+1

improve "def encode_html_and_save"

only encode html for specifric product level
Idea : pass regex object for checking filename

Print UTF-8 Code

On local disc, html is stored as utf-8 and python has a Problem handling it.

possible solutions with code in python

stackoverflow

Hope this will not encounter, when running on real website ...

	for item in all_cat0:
	I+=1
	print(item.Text)
	sl0 = item.find_all("a")
	print ('found ' +str(len(sl0)) + ' sublinks')
	J = 0
	for l0 in sl0:

	has_link = l0.has_attr('href')
	print('Hat link',has_link)
	if has_link:
	J+=1
	print (l0.text,l0['href'])

	print('cat read')

eed85 / webii Goto Github PK

webii's People

Contributors

webii's Issues

Recommend Projects

Recommend Topics

Recommend Org