nceas / datamgmt Goto Github PK
View Code? Open in Web Editor NEWData management utilities for curating, documenting, and publishing data.
Home Page: https://nceas.github.io/datamgmt/
License: Apache License 2.0
Data management utilities for curating, documenting, and publishing data.
Home Page: https://nceas.github.io/datamgmt/
License: Apache License 2.0
The udunits2
library is required for datamgmt
and requires steps beyond install.packages("udunits2")
. May be worth adding in prompts (The udunits2 library is required for the datamgmt package. Would you like to install the udunits2 library?)
Another idea for a fellows first project / function:
The idea here is that we sometimes use datamgmt::clone_package
or datamgmt::copy_package
to clone or copy a package to another member node, or to restore a previous version. This function does not update the pids in the eml (if new_pids = TRUE
in the case of clone_package
). It would be useful to write a helper function that does this in an efficient manner and add it to copy_package
and conditionally in clone_package
if new_pids = TRUE
.
Potential workflow:
@maier-m I'm cleaning up datamgmt
and this function makes way more sense in EML. Did you ever ask if Karl wanted to add it to the ropensci/EML
package?
If you're interested in the longer story: I used which_in_eml
as a helper in this function NCEAS/arcticdatautils#97, but because datamgmt
already imports arcticdatautils
then arcticdatautils
cannot import any datamgmt
functions because this creates a cyclic package dependency. My hacky solution was to add which_in_eml
to arcticdatautils
so I could use it 🙄, but it makes way more sense in EML
.
High level function for checking whether attribute metadata is correct.
Should be able to input an eml, a single eml attribute list, or a data.frame.
arcticdatautils::pid_to_eml_physical
is one of most useful functions but specific fields only work for uploads to the Arctic Data Center.
For instance the following creates the online distribution info link:
phys@distribution[[1]]@online@url <- new("url", paste0("https://cn.dataone.org/cn/v2/resolve/",
x@identifier))
We should generalize this function to work across all (or alot) of the dataOne member nodes - definitely test ADC to start.
Currently, any time you update an object with a new version (via update_object), you have to save the new PID and then update the resource map.
I'm guessing it's possible to bundle the two together into one function, which updates the resource map whenever an object is updated.
possibly consolidate with update_object
Rewrite compare_eml_to_package_pids
function using eml_get
.
Look into how quickly replacing the bulk of our imported packages with importFrom
tags could be finished
We're generally following tidyverse standards, but we have a few package-specific style prefs so far:
Write script to regularly harvest information about NSF awardees for outreach purposes
An alternative to RT would need to meet the following requirements:
So far, we've discussed:
Should check physical
> dataFormat
> textFormat
> numHeaderLines
to see if there are multiple header lines.
When using create_attributes_table to edit an existing attributes table from an eml, the tables associated with enumerated domains do not appear.
@params – resource map PID
Output will be written to a file and should include resource map PID, and any identifiers (filenames, PIDs of datasets, PIDs of EML, datatable#, etc) that are applicable to errors that occurred.
The goal of this is to get packages ready for consumption by erddap. This will likely be integrated with the metadata quality engine.
Correct data/EML/resource map linkage (given a resource map PID – we need to be able to get current EML & data PIDS)
Data/EML compatibility
Check that all column names in attributes match the column names in the csv
Domains: Check that all attribute types match attribute types in the csv
When any changes are made to a data object, it would be nice to have the physical automatically updated. Perhaps a wrapper that calls publish object or publish update and also builds the new physical.
For textDomain definition
is a required field. Function should copy attributeDefinition into definition by default, or make definition a red box when using the shiny app.
This bug fix should also be applied to eml2
version of this function.
I had several open tickets for issues here, so I am going to consolidate them all into one. I think other issues arose as I was out, too.
Bugs
Improvements
I am closing #90, #91, #92, and #119 since this captures all of them.
The edit_attribute
function only modifies dataTable
objects in EML, when it should be able to modify dataTable
or otherEntity
objects as both can contain attribute metadata. One way to go about this would be replacing the dataTableNumber
with something like object
, and having the user specify the path as an input (for instance object = eml@dataset@dataTable[[2]]
)
This comes up when trying to install a new version of 'arcticdatautils' when 'datamgmt' is already loaded.
Reloading installed arcticdatautils
unloadNamespace("arcticdatautils") not successful, probably because another loaded package depends on it.Forcing unload. If you encounter problems, please restart R.
We might need to remove arcticdatautils
as an import from datamgmt
and switch to importFrom
calls for the specific functions we need.
This would be super helpful for testing the new Package Editor and PROV Editors:
prod <- MNode(...)
test <- MNode(...)
package_copy(package_pid, prod, test)
query_all_versions
as currently constructed only queries metadata. For instance this query returns a blank data frame:
datamgmt::query_all_versions(adc_prod, "resource_map_urn:uuid:a3784690-300e-4041-bd30-5d9b47759903", fields = "submitter")
It would be great to extend this to query resource map and data objects, along with some error handling for blank fields (for instance a resource map will not return a northBoundingCoordinate
).
Some unit tests for these cases would be great too (resource map, data, and catching blank fields)
2/3 functions have unit tests. I was using a broken version - a test for the last function would've avoided this.
https://github.nceas.ucsb.edu/KNB/submissions-bot/blob/master/bot.py
clone_package
copies a parent + it's (optional) children. It does not update the package metadata to reflect the new pids yet.
Filing this issue as a feature request so I don't forget.
shiny_attributes_table: no visible global function definition for
‘packageVersion’
shiny_attributes_table: no visible global function definition for
‘get_unitList’
shiny_attributes_table: no visible global function definition for
‘fluidPage’
shiny_attributes_table: no visible global function definition for ‘br’
shiny_attributes_table: no visible binding for global variable ‘tags’
shiny_attributes_table: no visible global function definition for
‘actionButton’
shiny_attributes_table: no visible global function definition for ‘h5’
shiny_attributes_table: no visible global function definition for
‘rHandsontableOutput’
shiny_attributes_table: no visible global function definition for
‘fluidRow’
shiny_attributes_table: no visible global function definition for
‘column’
shiny_attributes_table : server: no visible global function definition
for ‘reactive’
shiny_attributes_table : server: no visible global function definition
for ‘hot_to_r’
shiny_attributes_table : server: no visible global function definition
for ‘renderRHandsontable’
shiny_attributes_table : server: no visible global function definition
for ‘%>%’
shiny_attributes_table : server: no visible global function definition
for ‘rhandsontable’
shiny_attributes_table : server: no visible global function definition
for ‘hot_table’
shiny_attributes_table : server: no visible global function definition
for ‘hot_col’
shiny_attributes_table : server: no visible global function definition
for ‘hot_cols’
shiny_attributes_table : server: no visible global function definition
for ‘observeEvent’
shiny_attributes_table : server: no visible global function definition
for ‘stopApp’
shiny_attributes_table: no visible global function definition for
‘shinyApp’
Undefined global functions or variables:
%>% actionButton br column fluidPage fluidRow get_unitList green h5
hot_col hot_cols hot_table hot_to_r is new observeEvent
packageVersion pid_to_eml_physical query rHandsontableOutput reactive
red renderRHandsontable rhandsontable shinyApp stopApp tags tail
write.csv write_eml
The EML package eml_validate
function currently does not run some of the checks used in Metacat. Ideally, they would show the same result.
From Slack:
"it would be nice to add the extra schema-validity rules to the eml_validate function so it will show the same results as Metacat" (Matt)
"Once you send it across the network to the Member Node, Metacat runs a suite of custom validation rules on the EML.
It's stuff that XML schema's can't help us enforce, such as the match between a custom unit and its definition (as here) or ids and references." (Bryce)
open question:
"Im not sure what additional schema rules would be needed? right now EML::eml_validate
is basically a wrapper for xml2::xml_validate
which feeds that function the eml schema (EML/xsd/eml-2.1.1/eml.xsd
). what additional checks are needed?" (Mitchell)
This always fails at dataone::getObject()
. Should try to debug or just add a condition to skip if private + xlsx.
Easy checks for reviewing methods/abstracts. Submissions should:
This isn't the highest priority but we should add unit tests for this function eventually.
See https://github.com/NCEAS/arcticdatautils/tree/master/tests for examples
Add summary of all functions in datamgmt/master, possibly to README?
This will help:
Code review for the three functions in R/package_copy.R would be appreciated! Open to any general changes / comments.
https://github.com/NCEAS/datamgmt/blob/master/R/package_copy.R
Add dependencies on an install of the datamgmt package.
We currently need: arcticdatautils, dataone, shiny, rhandsontable, any others?
Right now, the easy (but tedious) way of dealing with Excel messing up dates is to open a file in Excel, save it as a csv, and import into R.
If a function exists on the internet, then please comment here! Otherwise, I solved part of this problem with this frac_to_hm
function:
frac_to_hm <- function(fraction, format = "character") {
#changes time as fraction to hh:mm format
#can either output as character or lubridate
h <- as.integer(fraction * 24)
m <- as.integer((fraction * 24 - h) * 60)
if(format == "lubridate"){
time <- hm(paste(h, m))
}
if(format == "character"){
time <- paste(h, str_pad(m, 2, pad = "0"), sep = ":")
}
return(time)
}
In this case, when Excel converts hh:mm to a fraction (i.e. 0.5), I can re-convert it to the appropriate character string. (doesn't deal with dates though)
The solution to this problem should also be written out in Rmd and get added to our reference guide.
add_creator_id: no visible global function definition for ‘green’
add_creator_id: no visible global function definition for ‘red’
add_creator_id: no visible global function definition for ‘new’
This idea is from @jagoldstein. It would be useful to write an eml_diff
function that compares two emls and returns differences (similar to github). The trick here is not to compare it line by line, but detect which sections exist dynamically, and then compare those.
This could be a good starting project for a fellow that wants to get more familiar with EML and writing functions.
This is because I am not currently coercing values to dateTie. Should remove warning or handle this case.
I just get a blank screen when using create_attributes_table. Sometimes this error pops up in the console:
[No stack trace available]
Error in safeFromJSON(charData, simplifyVector = FALSE) :
Argument 'txt' is not a valid JSON string.```
I received a ticket today that necessitates adding a new attribute to an existing attribute table (the csv file for this package was given an additional column). The options for doing this appear to be 1) rebuild the whole attribute table or 2) edit the xml file in-line. These are both pretty simple (especially 2) for a small table like the one I am working on, but in future cases I think it would be very helpful to have a way to set a new attribute in the eml, e.g.
attribute19<-new(attribute, arguments yada yada yada)
eml@dataset@dataTable@attributeList@attribute[[19]]<-attribute19
forgot to put in a token, and the result when using obsolete_package
without one is:
obsolete_package(mn, "urn:uuid:14d91309-0ecb-4c78-88a4-ff6e691bcc9a", "urn:uuid:17d733ef-cb2b-48e0-8ca7-07aa162c6b70")
Error: arcticdatautils::object_exists(mn, metadata_obs) is not TRUE
which makes it look like there is no object there, but the issue was really the token.
When using qa_package
an object may fail to process because the distribution URL in the physical is missing or differs from the object pid. It would be nice to have a helper function to pinpoint exactly what is happening with the physical. The main possibilities are that the physical is simply missing, the physical was not updated after the object was updated, or the wrong physical was assigned to the object. A qa_physical
function would first check if the physical exists and if so check for these other possibilities to narrow down the issue, producing the appropriate messages to guide action. I'd be happy to work on this if it sounds like a good idea.
Related to @csjx’s idea to provide option for new awardees to populate (some fields) of an EML template of sorts, to get an idea early-on about what’s expected (at least Project level - avoid PI’s copying and pasting NSF abstract to dataset)
Currently using Remotes: ropensci/EML in the Description
file to load the EML package. Change this back to a normal import once a new version of the EML package is released on CRAN
The datamgmt pkgdown site is a good place for newbies to datamgmt to explore a curated list of functions, but is currently very out of date
@dmullen17 perhaps you're interested?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.