Problemas a serem resolvidos no pacote

bdc

A toolkit for standardizing, integrating, and cleaning biodiversity data

Overview

Handle biodiversity data from several different sources is not an easy task. Here, we present the Biodiversity Data Cleaning (bdc), an R package to address quality issues and improve the fitness-for-use of biodiversity datasets. bdc contains functions to harmonize and integrate data from different sources following common standards and protocols, and implements various tests and tools to flag, document, clean, and correct taxonomic, spatial, and temporal data.

Compared to other available R packages, the main strengths of the bdc package are that it brings together available tools – and a series of new ones – to assess the quality of different dimensions of biodiversity data into a single and flexible toolkit. The functions can be applied to a multitude of taxonomic groups, datasets (including regional or local repositories), countries, or worldwide.

Structure of bdc

The bdc toolkit is organized in thematic modules related to different biodiversity dimensions.

⚠️ The modules illustrated, and functions within, were linked to form a proposed reproducible workflow (see vignettes). However, all functions can also be executed independently.

1. Merge databases

Standardization and integration of different datasets into a standard database.

bdc_standardize_datasets() Standardization and integration of different datasets into a new dataset with column names following Darwin Core terminology

2. Pre-filter

Flagging and removal of invalid or non-interpretable information, followed by data amendments (e.g., correct transposed coordinates and standardize country names).

bdc_scientificName_empty() Identification of records lacking names or with names not interpretable
bdc_coordinates_empty() Identification of records lacking information on latitude or longitude
bdc_coordinates_outOfRange() Identification of records with out-of-range coordinates (latitude > 90 or -90; longitude >180 or -180)
bdc_basisOfRecords_notStandard() Identification of records from doubtful sources (e.g., fossil or machine observation) impossible to interpret and not compatible with Darwin Core recommended vocabulary
bdc_country_from_coordinates() Derive country name from valid geographic coordinates
bdc_country_standardized() Standardization of country names and retrieve country code
bdc_coordinates_transposed() Identification of records with potentially transposed latitude and longitude
bdc_coordinates_country_inconsistent() Identification of coordinates in other countries or far from a specified distance from the coast of a reference country (i.e., in the ocean)
bdc_coordinates_from_locality() Identification of records lacking coordinates but with a detailed description of the locality associate with records from which coordinates can be derived

3. Taxonomy

Cleaning, parsing, and harmonization of scientific names against multiple taxonomic references.

bdc_clean_names() Name-checking routines to clean and split a taxonomic name into its binomial and authority components
bdc_query_names_taxadb() Harmonization of scientific names by correcting spelling errors and converting nomenclatural synonyms to currently accepted names.
bdc_filter_out_names() Function used to filter out records according to their taxonomic status present in the column “notes”. For example, to filter only valid accepted names categorized as “accepted”

4. Space

Flagging of erroneous, suspicious, and low-precision geographic coordinates.

bdc_coordinates_precision() Identification of records with a coordinate precision below a specified number of decimal places
clean_coordinates() (From CoordinateCleaner package and part of the data-cleaning workflow). Identification of potentially problematic geographic coordinates based on geographic gazetteers and metadata. Include tests for flagging records: around country capitals or country or province centroids, duplicated, with equal coordinates, around biodiversity institutions, within urban areas, plain zeros in the coordinates, and suspect geographic outliers

5. Time

Flagging and, whenever possible, correction of inconsistent collection date.

bdc_eventDate_empty() Identification of records lacking information on event date (i.e., when a record was collected or observed)
bdc_year_outOfRange() Identification of records with illegitimate or potentially imprecise collecting year. The year provided can be out-of-range (e.g., in the future) or collected before a specified year supplied by the user (e.g., 1900)
bdc_year_from_eventDate() This function extracts four-digit year from unambiguously interpretable collecting dates

Other functions

Aim to facilitate the documentation, visualization, and interpretation of results of data quality tests the package contains functions for documenting the results of the data-cleaning tests, including functions for saving i) records needing further inspection, ii) figures, and iii) data-quality reports.

bdc_create_report() Creation of data-quality reports documenting the results of data-quality tests and the taxonomic harmonization process
bdc_create_figures() Creation of figures (i.e., bar plots and maps) reporting the results of data-quality tests
bdc_filter_out_flags() Removal of columns containing the results of data quality tests (i.e., column starting with “.”) or other columns specified
bdc_quickmap() Creation of a map of points using ggplot2. Helpful in inspecting the results of data-cleaning tests
bdc_summary_col() This function creates or updates the column summarizing the results of data quality tests (i.e., the column “.summary”)

Installation

Gnparser installation

Previously to bdc installation is necessary to install GNparser. First, download the binary file of gnparser for your operational system. For example, download the file using R as follow:

download.file(url = "file_link", 
              destfile = "destination_path")

The downloaded file has extensions .zip or .gz.

Mac OS

Extract the binary file gnparser from .zip or .gz files and move it to the folder ~/Library/Application Support/. Move the file manually or using R:

# Extract gnparser file
untar("~/Downloads/gnparser-v1.9.1-linux.tar.gz")

# Move to the path
file.copy("./gnparser", "~/Library/Application Support/")

Linux

Extract the binary file gnparser from .zip or .gz files and move it to the folder ~/bin. Move the file manually or using R:

# Extract gnparser file
untar("~/Downloads/gnparser-v1.9.1-linux.tar.gz")

# Move to the path
file.copy("./gnparser",  "~/bin")

Windows

In Windows, extract the binary file gnparser from .zip. Then, move gnparser file to the folder Appdata. To find the Appdata path, run this in R:

# Unzip the downloaded file
unzip(gnparser.zip, exdir = "destination_path/gnparser")

# Find the AppData path
AppData_path <- Sys.getenv("AppData")

# Copy gnparser to AppData

file.copy("destination_path/gnparser", AppData_path, recursive = TRUE)

bdc installation

After installing Gnparser, you can install bdc from CRAN:

install.packages("bdc")
library(taxadb)

or the development version from GitHub using:

install.packages("remotes")
remotes::install_github("brunobrr/bdc")

Load the package with:

library(bdc)

Package website

See bdc package website (https://brunobrr.github.io/bdc/) for detailed explanation on each module.

Getting help

If you encounter a clear bug, please file an issue here. For questions or suggestion, please send us a email ([email protected]).

Citation

Ribeiro, BR; Velazco, SJE; Guidoni-Martins, K; Tessarolo, G; Jardim, Lucas; Bachman, SP; Loyola, R (2022). bdc: A toolkit for standardizing, integrating, and cleaning biodiversity data. Methods in Ecology and Evolution. doi.org/10.1111/2041-210X.13868

	# Currently available databases and versions
	switch(
	EXPR = db,
	itis = {
	db_version <- 2022
	},
	ncbi = {
	db_version <- 2022
	},
	col = {
	db_version <- 2022
	},
	gbif = {
	db_version <- 2022
	},
	iucn = {
	db_version <- 2019 ### TODO: Change to 2022
	},
	ott = {
	db_version <- 2021
	},
	fb = {
	db_version <- 2019
	},
	tpl = {
	db_version <- 2019
	}
	)

brunobrr / bdc Goto Github PK

bdc's Introduction

bdc

A toolkit for standardizing, integrating, and cleaning biodiversity data

Overview

Structure of bdc

1. Merge databases

2. Pre-filter

3. Taxonomy

4. Space

5. Time

Installation

Gnparser installation

Mac OS

Linux

Windows

bdc installation

Package website

Getting help

Citation

bdc's People

Contributors

Stargazers

Watchers

Forkers

bdc's Issues

point1 = Finland, point2 = Sweden, point3 = Norway, point4 = Estonia)

decimalLatitude decimalLongitude country 1 62.58727 30.81352 Estonia 2 66.62444 21.21965 Norway 3 60.91266 10.65224 Sweden 4 59.16613 25.88388 Finland

1. Error in utils::download.file... after running bdc_clean_names()

2. Error: rapi_startup: Failed to open database... after running bdc_query_names_taxadb()

raw data

preprocessed data

TODO

Descrição

Recommend Projects

Recommend Topics

Recommend Org

decimalLatitude decimalLongitude country
1 62.58727 30.81352 Estonia
2 66.62444 21.21965 Norway
3 60.91266 10.65224 Sweden
4 59.16613 25.88388 Finland

1. `Error in utils::download.file...` after running `bdc_clean_names()`

2. `Error: rapi_startup: Failed to open database...` after running `bdc_query_names_taxadb()`