taetscher / hbs Goto Github PK

It appears that the Handball Federation switched from TORHÜTER to TORHÜTER*IN. This might very well be the reason why player progress statistics are not produced anymore.
In particular: look at the string-splitting, maybe splitting the data 3 spaces later (or subsequent splitting integers +3 spaces) fixes the issue already.

GitPython, since recently gives the following error when trying to push to origin:

commit message: statistician @ 17.04.2021 14:56:57
Error while pushing to remote repository (in git_push): 
'utf-8' codec can't decode byte 0xfc in position 79: invalid start byte

-> Try to find another library that can automate the git flow, as manual pushing works just fine. Therefore the problem seems to be related to GitPython, not the code in HBS

If the program runs for a number of weeks, at some point a ```CANNOT ALLCOATE MEMORY`` error shows up. This needs to be adressed, to keep the maintenance of HBS low.

the goal would be to have these pages work just as well on mobile as they are on desktop.

-> font sizes seem to be a big part of what needs to be improved in order to achieve this

Progress:

• Font sizes are mobile ready
• About page is mr
• ips is mr
• ts is mr
• landing page is mr

Recently, the raspi got the following error message when scraping:

Message: timeout
(Session info: chrome=72.0.3626.121)
(Driver info: chromedriver=72.0.3626.121,platform=Linux 4.19.66-v7+ armv7l)

figure out why this is happening and fix the issue

https://www.handball.ch/de/matchcenter/spiele/428240
Error while getting game progression or writing game progression:
Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="live"]/div[2]/div[3]"}
(Session info: chrome=72.0.3626.121)
(Driver info: chromedriver=72.0.3626.121,platform=Linux 4.19.66-v7+ armv7l)

display a tooltip (successes/attempts) when the user hovers over a certain point on the graph.

see if that is possible with the line, or if for each player, a point should be added

if points are added, see here to try and omit lines where they should not be (player has not had actions, in the data this is shown as "")

.defined(function(d) { return d.y; }) // Omit empty values.

https://stackoverflow.com/a/25972625/13389221

on the 10.04.2021, the scraper scraped the first 25 minutes of a game in progress.

This cannot happen, because of the way the scraper checks for already scraped games.

when statistician is running on raspi, it would be nice to see timestamps next to the printed stuff in the console, to get a feel for how things are going at the minute

While in custom selection mode, the hover function does not always work.

Try d3.raise() on selected area graphs to see if this could fix the problem

currently, regardless of anything having changed or not, player progress statistics are always regenerated.
this is useless in cases where nothing changed, so check for that and leave it out whenever possible

Obviously the SHV pages can change, as they have in the past.

Whenever the SHV makes changes to relevant pages, XPaths can change, which can mess up the scraper.
In order to make maintenance that much easier, it could be worth to make a repository of all used xpaths (with meaninful names)

-> this could be integrated into options.py as a dict

Example:

  (Session info: chrome=72.0.3626.121)
  (Driver info: chromedriver=72.0.3626.121,platform=Linux 4.19.66-v7+ armv7l)

This can occur, so far i have no idea why.
One way for dealing with it would be to remove the sys.exit() call on errors and keep the scraper running (as the above exception does not always happen twice in a row)

currently, each iteration of both scrapers seem to add the list of teams from options.py with each iteration.

make it so this only happens once

It is currently difficult to distingiush between players that have, lets say 0 technical faults and players that simply have not played a game (both will get a value of 0).

-> adjust the scraper so it records if a value is 0 or is None (or nan), so the stats represent reality better

probably necessary to adjust the scraper (or adjust in conversion of csv files), as the gh-pages page gets csvs where df.fillna(0) is being called during scraping
--> Adjustments probably needed in mergeStats{Outfield/Goalie} in playerProgressScraper.py