g-node / gin-doi Goto Github PK
View Code? Open in Web Editor NEWG-Node DOI Service
License: BSD 3-Clause "New" or "Revised" License
G-Node DOI Service
License: BSD 3-Clause "New" or "Revised" License
The "DOI registration" service and the static "DOI hosting" service both share the same "assets".
Still, when the assets of the project are updated, the assets need to be manually copied from this project to where the hosting of the static DOI pages is taking place.
Making the assets available to the host running this project via docker would make this copy step no longer required and would ensure, that the assets of the "DOI registration" and the static "DOI hosting" service are the same when the project is deployed.
Currently a DOI registration request fails,
This leads to the case that even a differing whitespace in the license content text will fail a registration request.
Since the formatting of license texts can vary while still being valid licenses, this check should be removed.
Add a notification to the admins if a DOI request repository contains submodules.
If there is no grant number in a funding item the entry ends up in the awardNumber tag. It should go into the funderName, which is the required field.
When the funding
information contains no split character but has an entry, set funderName
as default.
Currently it seems to default to awardNumber
.
We can semi-automate the confirmation email message that we send to the requesting user. Currently, I send it manually when the registration is finished and I base it on a template email that includes fields for the name of the user, the title of the dataset, and the DOI.
The service could create a link that encodes the above information (encrypted) in the same way the GOGS service encodes the request it sends to GIN DOI. The link can be included in the email that notifies us of a request. When the link is clicked, it loads a page with a text box pre-filled with the mail template with the fields filled in. We can then review or edit the email content and subject and press a button to have the DOI service send the email message to the user.
Requirements already met:
New requirements:
I think this might be related to the server configuration or the network drive. We've had reports of this being an issue with the gin-cli. Local tests work, so I'll have to reproduce the server environment or safely test on the server somehow.
During the current automatic zip file creation, it can happen that some files are added as read-only. This seems to happen for annexed files that were uploaded using a previous version of the gin client and which remain locked and read only after annex uninit
.
The read-only
status of these files might catch some users off guard, so it would be nice to make sure that all files are added to the zip file as unlocked and read-write
.
See G-Node/gogs#24
Describe the bug
If I click on the button "request doi" in the repository https://gin.g-node.org/mikapfl/read_di_unfccc , I get the error message:
DOI request failed
The DOI file is missing or not valid. See the messages below for specific issues with the provided data.
Also, please see the DOI guide for detailed instructions.
No datacite.yml file found in repository
However, the datacite.yml definitely exists: https://gin.g-node.org/mikapfl/read_di_unfccc/src/main/datacite.yml
The problem might be related (wild guess) to the fact that this repository has no "master" branch and uses a "main" branch instead.
Probably happened when we changed the reference ID parsing.
When no DOI is given in a reference, an empty RelatedIdentifier entry is included. This creates an error at registration.
Could this be filtered?
Following up from #57, we should also consider the case where a dataset has more than two versions. This hasn't come up yet, but there's a chance it will.
In the current system "middle" versions will both have a notice that a new version is available and a link to the previous one at the bottom. This will require the user to follow the links up the chain to the newest version.
It would be preferable to link to the newest version from all older pages and it would also be nice if the latest version listed all previous versions. If we want to do this, we'd have to make some more changes so that the landing page creation function follows the chain of references to link the latest one (if it's old) or list all available versions (if it's the newest).
The following functions do not seem to be used and could be removed if they are not required for legacy purposes or future development
util.go:readBody()
... seems unusedutil.go:makeUUID()
... seems unused; probably superseded by different DOI ID scheme.util.go:ReferenceDescription()
... seems unused and superseded by util.go:FormatReferences()
util.go:ReferenceSource()
... seems unused and superseded by util.go:FormatReferences()
util.go:ReferenceID()
... seems unused and superseded by util.go:FormatReferences()
It happened a couple of times now, that if the download rate of annex content drops below byte/s, the content download stops silently, continuing with the next file. This can lead to files without content which is an issue when the zip file is created, containing only files with the annex object reference instead of the full files.
If another content download is initiated, the content download of these files will continue where the download has stopped before.
As a quick fix, a second round of content download should be added to the download procedure. If files have not been fully downloaded, they should finish during the second round. If all files had been downloaded in the first place, nothing will happen.
As a long term solution content download fails have to be identified; if content is still missing after the second round of downloads, admins should be notified via email to look into the issue.
The xml has empty ORCID fields because in the datacite the IDs are missing. Could that be filtered?
We usually want people to register their DOI as ResourceType 'Dataset' even though other ResourceTypes are valid. To not miss a ResourceType other than 'Dataset' add a corresponding admin warning after the DOI preparation is complete.
Marshalling XML using encoding/xml would be safer and more flexible.
The service should be able to generate the full list of registered datasets, found at https://doi.din.g-node.org, given all the XML metadata files.
Try to match the license string in the datacite.yml with the LICENSE file in the repository based on the standard description and text in the file (or the first line of the file). If it doesn't match, simply show a warning and ask the user to confirm that they actually do match.
The issue creation added in #61 works for the creation of the issue, but adding a comment to report on problems after the initial issue is created doesn't work.
Error message: Failed to create issue or comment on XML repo: [500] {"message":"issue does not exist [id: 0, repo_id: 1338, index: 298]","url":"https://github.com/gogs/docs-api"}
The link is hardcoded to link to https://gin.g-node.org, which is fine for the live system, but we should make it dynamic so that it links to the configured GIN server. This is useful for testing and dev builds.
Before each operation that potentially downloads enough data to fill the target storage, the service should check the available storage.
There are three steps where this would be useful:
Step 1 might be an issue since it's not straightforward to know the size of the repository before cloning. We might have to solve that by adding the functionality to report repository sizes on GIN Web (Gogs).
Step 2 is pretty straightforward. For one, git annex will refuse to download a file if there is not enough local storage. That said, if several files are being downloaded, we might reach near-capacity before the limit is hit, so the service should check for the entire download size before running get-content
. Git annex provides this information via git annex info
(which also supports --json
output).
When we add the repo size reporting functionality to GIN Web, this step could be merged into step 1.
Step 3 essentially means we would require twice the space required for step 2. Assuming no compression (worst case), the zip file would take up the same amount of storage as the repository, so the storage needed to clone & get-content will be doubled when creating the zip file.
Essentially, if we can know ahead of time the storage requirements for a repository, we can safely clone and zip if there is twice as much available space.
NOTE: This becomes tricky when multiple repositories are being registered simultaneously. We should consider having workers report the storage space they're planning to use to the other workers.
Followup to #55
The license validation function assumes the licenses match when it's unable to validate the text against the name. Instead, it should add a warning to the admin notification that a manual check is needed.
gin-doi/cmd/gindoid/validation.go
Lines 94 to 105 in ad4935d
Top navbar for landing pages and upcoming keyword and front page should be unified and embedded from the same template fragment.
Warn if a funding reference is included without a funder name.
The service should keep a record of open requests and block the user from resubmitting a request from the same repository while a request is still open unless the source repository has changed. The record should be a database (or a flat file) with the following information:
When a new request comes in, the workflow would then be:
The datacite.yml content is validated in two different locations: partially in the readRepoYAML
function and partially in the validateDataCiteValues
function. Issues returned from both functions lead to a registration rejection.
Move the validation specific code from readRepoYAML
to validateDataCiteValues
. readRepoYAML
then only returns an error, if the datacite.yml file is not a valid YAML file and the content cannot be unmarshalled. The validateDataCiteValues
function will validate the datacite.yml content and collect and return any and all Datacite specific issues.
When a new version of an existing dataset is linked, we add the information to the XML files of both the new and the old datasets using related identifiers and the reftype "IsNewVersionOf" and "IsOldVersionOf" respectively. The landing page generator should use this information to add:
If a user navigates to https://doi.gin.g-node.org they get the gin-doi request page (initjob.html) but it's empty because no variables were supplied.
The root of the service should redirect to the registered datasets page (https://doid.gin.g-node.org) when all variables are empty. Alternatively, the initjob page could be a separate page and the root could redirect always.
All valid DataCite reftypes should be allowed through the validator. We should mark some that are less appropriate to warn us for manual review though.
We should add a warning for the admins to do a manual review if any reftype is not IsSupplementTo, which is the common way of referencing a manuscript.
It happens quite frequently that license title and link do not match in the datacite.yml which could be caught by adding a validation similar to the license "title" and "expected content" validation.
Requesting a DOI can fail due to a couple of issues:
datacite.yaml
is missing some required fields.datacite.yaml
file contains unsupported values.Each of these cases leads to different error pages, potentially leading the user to submit a DOI request three times before all issues are resolved.
Change the procedure to always check all three of the above described conditions, collect all issues and show them to the user at once, before failing the DOI request.
The main functionality of preparing a repository for registration, as well as sub-parts of it, should be exposed via the command line. This would be useful for:
Each keyword on a landing page links to a keyword page that should list all the published datasets that share that keywords.
Submodules are not included in archive zip or repo. If a repository contains submodules, add a notice to the DOI requestee on the DOI request overview page.
When the datacite.yaml
file contains invalid entries or missing required data, the corresponding messages are displayed to the user as a single string.
Display the messages as list items to make them more convenient to read.
In the datacite.yml, we expect the author ID to be in the form type:id
, e.g., ORCID:0000-0001-2345-6789
. Sometimes a user will enter their ORCID but without the prefix. We should allow this and assume it's an ORCID when the prefix is missing, since that's the most commonly used author ID type. The service should warn us when this happens so we can manually verify the ID.
When a new DOI request is being processed, add the HEAD commit hash to the gin issue and the admin email to verify and document at which stage the gin repository was submitted as a DOI request.
Add the GIN bottle favicon to all pages.
A very large dataset can cause parts of the preparation process to hang indefinitely. The most recent occurrence involved three very large datasets (hundreds of GiB each, the largest being over 800 GiB) being prepared at the same time. All three workers stopped during the git annex uninit
phase. It seems the git-annex
process hanged for all three. Killing the processes let the worker queue continue, but all subsequent zip files (for requests that were already in the queue) did not contain any annexed data.
Currently the information in funder
is split at the comma character. Since this character has been used in the past in the funder name entry, use semi-colon instead.
Keep split on comma as fallback and backwards compatibility.
This functionality can be a small, constantly running worker (goroutine) in gin-doi that periodically checks the status of dataset landing pages. Broken links can occur for mainly two reasons:
In the first case, the badge on the repository and the link in the fork on GIN will be intact. The link to the source repository on the landing page should be updated. This should be done by updating the XML file and regenerating the landing page through the make-html
command. The published metadata should also be updated.
In the second case, the link should be removed. The procedure should be the same: Update the XML file (remove the relatedIdentifier
), regenerate the landing page, and update the published metadata.
A related situation (which hasn't occurred so far) is the owner of the repository making it private. This would also make the DOI fork private. In this case, it might be better to treat it like the second case, but also edit the DOI fork to make it a non-fork (or recreate it).
The repository is cloned for preparation into a directory with the repository name in all lowercase:
gin-doi/cmd/gindoid/datasource.go
Line 73 in 5842539
When preparing the archive, the commands are run without converting the path to lowercase:
gin-doi/cmd/gindoid/dstorage.go
Line 105 in 5842539
If a repository has uppercase characters in the name, this fails.
After submitting a request and before the dataset is finalised, we should have a way to block repeated requests to register the same repository. As it is now, if a user reloads the final request confirmation page, they will trigger a second request as the page loads.
The service should keep a list of ongoing registration jobs and when the user attempts to register a repository while a request is still being processed, they should get an appropriate message.
The util.go function FunderName
is linked via the template.Func
map, but seems to be unused.
There is a reference in template.go
to funding.Funder
which cannot be found in the code. Since funding.Funder
appears right next to funding.AwardNumber
and there is an AwardNumber
function in util.go right next to the FunderName
function, funding.Funder
should probably be funding.FunderName
in template.go
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.