mastaal / parlhist Goto Github PK

parlhist is a Python application intended to enable more empirical and statistical academic studies of (Dutch) parliamentary minutes and documents

License: European Union Public License 1.2

Python 97.79% Shell 2.21%

open-data the-netherlands staten-generaal parliamentary-data handelingen kamerstukken

parlhist's People

Contributors

Stargazers

Watchers

parlhist's Issues

Add support for Staten-Generaal Digitaal data (1814/1815 - 1994-31-12)

Older parliamentary history is not available via the regular data collection, but is digitalized in the "Staten-Generaal Digitaal" (sometimes abbreviated to sgd) dataset. Support for this involves some extra work, as this dataset is completely based on scans of paper documents.

Web interface for making simple queries

Add support for Kamerstuk attachments

Automatically match debates in TK and EK if they are about the same dossier / legislative proposal

Directly use API for all crawling

Create experiment template

Based on this template, users (who can already program in Python) should be able to easily create new experiments.

The template should store its inputs and results in an appropriate fashion in the database (also see #8)
The template should make effective use of parallelization (also see #9)

Add documentation

Support crawling complete dossier with more than 1000 results

parlhist/parlhistnl/crawler/kamerdossier.py

Lines 21 to 43 in e59edcd

 def get_kamerstukken_in_kamerstukdossier(dossiernummer: str) -> list[tuple[str, str]]: 

 """Using the SRU api, get all kamerstukken in a kamerstukdossier 

  Returns strings in the same formatting as denoted in other metadata, e.g. '35899;7', '35925-VII;31' or '35979;F' 

  """ 

 # overheid SRU documentation: 

 # https://data.overheid.nl/sites/default/files/dataset/d0cca537-44ea-48cf-9880-fa21e1a7058f/resources/Handleiding%2BSRU%2B2.0.pdf 

 base_query = f"https://repository.overheid.nl/sru?query=(w.dossiernummer=={dossiernummer})&maximumRecords=1000&startRecord=1" 

 try: 

 query_response = get_url_or_error(base_query) 

 except CrawlerException as exc: 

 logger.error("Received exception %s when trying to query the kamerstukken in dossier %s", exc, dossiernummer) 

 raise CrawlerException(f"Received exception when trying to query the kamerstukken in dossier {dossiernummer}") from exc 

 xml: ET.Element = ET.fromstring(query_response.text) 

 number_of_records = int(xml.find("{http://docs.oasis-open.org/ns/search-ws/sruResponse}numberOfRecords").text) 

 if number_of_records > 1000: 

 logger.critical("More than 1000 results in a kamerdossier!") 

 # TODO, implement this case

mastaal / parlhist Goto Github PK

parlhist's People

Contributors

Stargazers

Watchers

parlhist's Issues

Add support for Staten-Generaal Digitaal data (1814/1815 - 1994-31-12)

Web interface for making simple queries

Add support for Kamerstuk attachments

Automatically match debates in TK and EK if they are about the same dossier / legislative proposal

Directly use API for all crawling

Create experiment template

Add documentation

Support crawling complete dossier with more than 1000 results

Automatically match Handeling items relating to the same debate

Store experiment results in the database

Add efficient parallelization when indexing new data

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	def get_kamerstukken_in_kamerstukdossier(dossiernummer: str) -> list[tuple[str, str]]:
	"""Using the SRU api, get all kamerstukken in a kamerstukdossier

	Returns strings in the same formatting as denoted in other metadata, e.g. '35899;7', '35925-VII;31' or '35979;F'
	"""

	# overheid SRU documentation:
	# https://data.overheid.nl/sites/default/files/dataset/d0cca537-44ea-48cf-9880-fa21e1a7058f/resources/Handleiding%2BSRU%2B2.0.pdf
	base_query = f"https://repository.overheid.nl/sru?query=(w.dossiernummer=={dossiernummer})&maximumRecords=1000&startRecord=1"

	try:
	query_response = get_url_or_error(base_query)
	except CrawlerException as exc:
	logger.error("Received exception %s when trying to query the kamerstukken in dossier %s", exc, dossiernummer)
	raise CrawlerException(f"Received exception when trying to query the kamerstukken in dossier {dossiernummer}") from exc

	xml: ET.Element = ET.fromstring(query_response.text)

	number_of_records = int(xml.find("{http://docs.oasis-open.org/ns/search-ws/sruResponse}numberOfRecords").text)

	if number_of_records > 1000:
	logger.critical("More than 1000 results in a kamerdossier!")
	# TODO, implement this case