bernhardauer / austrian-parliament-data-processing Goto Github PK

• check legal things (running websites in austria, licensing of data etc.)
• add footer
  • impressum
  • gpdr
  • barrierefreiheitserklärung
  • cookie notice
  • some info about this site / about me maybe?
• add "production" environments and edit pipelines if neccessary ...
• scraper laufen lassen
• add default routing
• - [ ] add legal mandatory pages
• add license for data
• add favicon
• [ ]

Tasks

• general tasks

  • fix: do not show empty chat-bubbles with time only
  • desktop: smaller width of chat bubbles (better reading xp)
  • add link to original doc/video and cc remark (check if cc remark is even neccessary...)
  • website title/page title: GP, meetingNr, Topic, speaker Name
  • url schema change: instead of ascending number use speaker name and optional numbering (for multiple speeches by same person

• chat enhancements

  • speaker chat on left side only
  • other chats on right side only
  • fixed color for main speaker (primary color?)
  • fixed color for president of parliament
  • different colors for remaining speakers (but same colors for same speaker on current page)
  • applause activity with emoji
  • shout activity with emoji
  • general/unknown info as info-box
  • further description of speaker shown as small gray text (same as time)

• postboned
  - [ ] breadcrumb navigation

error: Error: self-signed certificate
at TLSSocket.onConnectSecure (node:_tls_wrap:1600:34)
at TLSSocket.emit (node:events:517:28)
at TLSSocket._finishInit (node:_tls_wrap:1017:8)
at ssl.onhandshakedone (node:_tls_wrap:803:12) {
code: 'DEPTH_ZERO_SELF_SIGNED_CERT',
response: undefined

• - finishing Layout
• - good wording/descriptive texts (good german language quality please)
• - add hints/info symbols with further description
• - graphics: choose consisten color scheme
• - add link to debatte/topic or corresponding parliament web page --> moved to other issue
• - api should return long names instead of abreviations
• - check Requirements for TOP-search
• fix null bugs while parsing speechMetaData
• - enhance filter
  • add date range to GP and date to session --> moved to other issue
  • add option "all" to dropdowns
• check site on mobile
  • big enough text size (easily readable)
  • no zoom in when clicking on input field
  • auto-completer should open downwards and input field should move to top of screen on click
  • scroll to diagramm when click on "draw diagram" button
• add loading indicators
• indicate if there is no data for current filter
• fix layout for big diagram legend
• order GPs by roman numerals
• fix bug: clearing of filter is not working properly

fullstack Features
- [ ] searchable speech interruptions (zwischenrufe)
- [ ] speech interruptions visualizations

Scraper
- [ ] store raw html in DB and re-use if applicable

• #48
• add unit tests for speech parser
• #47
• #57
• #61
  - [ ] manually update data set

Backend
- [ ] fix technical debts in backend/refactor inconsistencies

Frontend

• #39
• #40
• #41

- [ ] #54 blocked, because svelteKit2 feature is needed for proper implementation
- [ ] add homepage
- [ ] add project history page
- [ ] add contributors page
- [ ] optimize page loading times postboned, no priority right now

• #52

DevOps

• migrate from windows to linux AZ app service
• logging infrastructure considerations
• fully automate data scraping

Documentation

• add repo description
• add readme with general info & contribution / setup help

- [ ] downloadable DB snapshot for easy self import to local db
- [ ] youtube-video: short introduction to project

Tasks

• page title
  • GP, meetingNr
  • header
  • website title
• add link to original page and CC hint
• add speech info summorization (as emojis)
  - [ ] add link to official website "overview" page

Postboned/further ideas
- [ ] gpt-4 summorization --> postboned
- [ ] breadcrumb --> postboned

fullstack Features

• searchable speech interruptions (zwischenrufe)
• speech interruptions visualizations
• feedback button

Scraper

• Update scrapy
• store raw html in DB and re-use if applicable
• phytify project / refactor scraper (DI, structured logging, ....)

Backend

• upgrade to .net8
• fix technical debts in backend/refactor inconsistencies

Frontend

• update to svelte 5
• Migrate to svelteKit 2 https://kit.svelte.dev/docs/migrating-to-sveltekit-2
• update tailwind/daisyUI 4
• SEO optimizations
• optimize page loading times
• add homepage
• add project history page
• add contributors page
• scroll up button

DevOps

Documentation

• downloadable DB snapshot for easy self import to local db
• youtube-video: short introduction to project

** todo**

• HTML vs Text parsing considerations

• fix time parsing (time with seconds is not working)

• do not parse (Entschließungs-) anträge etc. as speeches

• do not parse name titles

• fix missing speeches if name titles are wrong

• fix missing speeches if topNr is missing

(see also #47)

no url-encoded urls please.
seo-friendly: lowercase & words seperated by dashes
eg: https://parli-info.org/wortmeldung/XXVII/217/Lebenshaltungs-%20und%20Wohnkosten-Ausgleichs-Gesetz-LWA-G/11
should be: https://parli-info.org/wortmeldung/XXVII/217/lebenshaltungs-und-wohnkosten-ausgleichs-gesetz-lwa-g/11

tasks

• consolidate: all urls german (license, imprint, ...); consider german umlaute (äö...)
• proper url encoding (dash, lowercase, umlaute escaped, ....)

see handwritten notes

• detailed info log msg in spiders
• scrapeScript config log msg
  - [ ] metadata in DB

• speech page: distinguish between main speaker, president and others
• add queryable type / subtype to speech entities
• remove prefixes/postfixes (titles,...) from speaker names
• simple logging
  - [ ] backend: change type from enum to string
• remove links to speeches which are not parsable at the moment (to prevent 404s)
• add sneak-peak of speech on speech overview page
• store which carousel is open in URL
• add title to speech page and speech overview page
• add breadcrumb to speech page
  - [ ] speech page: add emojis for special parsable infos (applause, shouting, ...)

see handwritten notes pls

scraper enhancements to support fully automated scraping

Fix parsing errors regarding names/titles of persons. e.g:

• buy domain
• create YT-channel
  • intro video (small tutorial how to use the site)
• add email

see handwritten notes

Data Filter is set with custom values
When users navigates to other page with same data filter option, those same custom user option should be kept

https://learn.microsoft.com/en-us/azure/azure-monitor/app/javascript-sdk?tabs=npmpackage

Tasks

• rewrite speech info parser as state machine
• check potential wrong entities (see embedded json exported by my custom mongodb aggregation pipeline)
• add tests

Known Bugs
- [ ] parsing of names with Mag [.] / MMag [.] / Ing (potential all titles) are not working --> not needed anymore (see #47 (comment))

• Parsing of unknown names is not good enough (see json); e.g.: "reicht Vizekanzler Kogler Hand"
  • named entity recognition or classic algorithm? Names are probably highlighted somehow in the raw HTML source code ....

generall discussion
Raw text parsing only? Or html enriched parsing? --> since when is proper hmtl encoded doc available?

austrianParliamentaryDataScraping.speechesMetaData-potentialWrongEntityList.json