cynkra / munch Goto Github PK

Was possible in the original code, we should be able to adapt here. Leads to somewhat more matching tables.

Not sure, but maybe we could use {dplyr} instead of {plyr} and {kimisc}?

• Set of source municipalities identical for all source years
• Set of target municipalities identical for all target years

We're looking only for internal consistency here -- within the mapping tables. Consistency with external data is important too, but a different issue.

becocmes -> becomes

with a graph widget or a sankey? Could be useful for #30.

needs to be activated

• When loading the SwissCommunes package check if data is up to date.

• Ask Peter Steiner for area as well as population data for years before 2012.
  (for 2012 and later we can get it from the BFS)

Provide helpers to conveniently import matching tables into a database.

• Merge two data sets based on commune IDs
• Aggregate using a function (default: sum)
• Make sure that factor levels match

https://www.bfs.admin.ch/bfs/de/home/dienstleistungen/forschung/api/api-gemeinde.assetdetail.15224054.html

Also check how zazuko is doing it. Old way: zazuko/fso-lod@e8f08de, maybe this has improved by now?

or of all mutation events in a time range. Could make use of #35.

Useful for cross-checking with the official SFSO communication.

Generates a lookup table for the mapping of the municipality id's which were involved in the time series between year_from and year_to.

mID_to can be a vector of mID's.

output columns: mID_from, name_from, mID_to, name_to, year_from, population, area

zazuko/fso-lod@ad15875#r31813718

• Function that automatically downloads the data every year?
• exists an API for the dataset?

Link: https://www.bfs.admin.ch/bfs/de/home/statistiken/regionalstatistik/regionale-portraets-kennzahlen/gemeinden.html

In an rmarkdown document, indicating for which cantons for which years we need to expect problems.

See script/analyze.R for a start.

Only relevant if we ever need to match data prior to 2006.

https://doodles.mountainmath.ca/blog/2020/11/10/tongfen/

Solves this problem for arbitrary geometries.

Apparently, US and Canada have "correspondence files". Is this similar to our historicized municipality list?

We should strive towards working with the dates as provided by the data, and not combine to years. (I remember adding an issue but can't find it now.)

• https://de.wikipedia.org/wiki/Berom%C3%BCnster#Fusionen
• https://de.wikipedia.org/wiki/Castel_San_Pietro_TI
• https://de.wikipedia.org/wiki/Breggia_TI
• https://de.wikipedia.org/wiki/Caneggio
• https://de.wikipedia.org/wiki/Buch_am_Irchel
• https://de.wikipedia.org/wiki/Neftenbach

Now the parameter municipalityIds is gone.

Could be optimized.

write_all_mapping_tables() determines the latest target year by:

MAX_YEAR <- lubridate::year(
    max(swc_get_municipality_mutations()$mAbolitionDate, na.rm = TRUE)
  ) + 1L

which means in case the last abolition happened during the current year, the target year will be 1 year in the future.

I would suggest

lubridate::year(
    max(swc_get_municipality_mutations()$mAdmissionDate, na.rm = TRUE)
)

Gets the municipalities-inventory of a certain year.

Discussion:

• should it be delivered with the SwissHistMunData package or is it calculated on-the-fly?

Could this be character encoding problems? Switching to testthat 3rd edition should show the difference.

https://github.com/cynkra/munch/pull/44/checks?check_run_id=2236574313

#44 (comment)

Via #22.

I changed cantons to get_cantons(), need to clean up.

• New pull request when data changes, see upgrade.yaml in RSQLite
• Update data page automatically (doesn't happen now for some reason)
• Data page should link to files directly on GitHub, requires #22

Allows to aggregate time series to the specified year_to municipality data.

Options for summarize_fun: sum, mean, median, min, max, ...
Options for weight: NULL, population, area

Output: tibble with columns: year, value

Discussion: Add weight to output? (how much did each municipality contribute to the output value)
e.g. additional column weight_info:
"3 municipalities: 20%, 30%, 50%"

It downloads the data for 2019-01-01 hardcoded. We have new functions for the intended purpose.

and other legacy code, convert to dplyr.

Needs #31.

Would be useful to have for testing if the deduction of the inventory from the manipulations achieves the correct result.

• Add note to README
• Archive

in inst directory, recreated by the pkgdown.yaml workflow. The data.Rmd vignette should link to those files.

Rationale: putting stuff into ../docs no longer works when rendering a pkgdown vignette.

As it turns out, our mapping tables include the only land-area which is not part of a municipality ("Staatswald Galm", BFSNr: 2391) and five "Kommunanzen", which belong to more than one municipality. Two of the latter still exist.
I don't think that's necessarily wrong, it's just something to be aware of.
Possibly we should mention it in the documentation.

E.g. in the downloaded BFS data (load_bfs_mun_list()) these areas do not appear among the municipalities.

More details in https://github.com/cynkra/munch/blob/f-31-tests/tests/testthat/test-external-consistency.R

• The data should be on GitHub, but not part of the repository
• A function should load the data into a cache or update an existing cache, with user consent (cf. renv::consent())
• If no consent given -- no caching, only in temporary directory
• We can perhaps use httr::HEAD() to check if the cache is up to date

• Remove legacy functions
• Decide on naming
• Decide if we need #11 and #12

should be possible for each year

Let's start with a few known output tests for select municipalities, and also error behavior.

Read from GitHub and cache locally, update on demand, issue a HEAD() request to check if data has changed.

Running swc_get_mun_merges(2019, canton = "AG") into the console does not return a data.frame or a tibble, conversely to what happens with:

• tibble::as_tibble(swc_get_mun_merges(2019, canton = "AG"))
• as.data.frame(swc_get_mun_merges(2019, canton = "AG"))

This behaviour is not aligned with those provided by other functions, such as swc_get_mutations() or swc_get_merger_mapping_table()

In particular to list

development:
  mode: auto

as well as the pkgdown website URL.

Can we add Tobias as contributor since we want to work together on this project, or should we fork the repo?

• camel-case to snake-case

Function to get all historical municipality id's between year_from and year_to, for a municipality id valid in year_to.

is rather terse right now, we need much more. With examples and perhaps a walkthrough.

This way I can push my docs- branch @krlmlr

• Replace everywhere
• GitHub homepage
• Create stub for old pkgdown content
• Replace swc_() prefix
• Clean up README