nrcan / gsip Goto Github PK
View Code? Open in Web Editor NEWGroundwater Surface water Interop Experiment
License: MIT License
Groundwater Surface water Interop Experiment
License: MIT License
In data configuration (/data/), an attribute is used to override the media type (if the requested media type is not exactly what is being returned). Except that attribute is call "header", which is misleading because it give the impression any http header could be overridden. Should be "media-type" or "alt-media-type".
In order to demonstrate the harvesting of linkages, there is an immediate need for more relations. Currently, there is one relation encoded in the harvester:
<https://cida-test.er.usgs.gov/chyld-pilot/id/hu/041504081604>
<https://www.opengis.net/def/hy_features/ontology/hyf/lowerCatchment>
<https://geoconnex.ca/id/catchment/02OJ*CA>.
@heryk When there are too many features one on top of the other and one tries to click, the message says "Too many wells" - should be "too many features"
Wondering what the best path is going to be to get HY_Features relations implemented in GSIP. I'd be happy to contribute if @tomkralidis or @denevers can point me to the RTFM and / or the files I can hack on.
Make sure feature doesn't disappear when you select a new one.
Depending of the usage, multiple /info/ representations might be necessary. The challenge and how to negotiate the a particular info for a given /id/. Since /id/ there should be only one /id/ per real world feature in a given governance, the resolution to multiple /info/ cannot rely on the /id/ structure.
One can rely only on
or a combination of those options.
Put the JSON-LD within the tag instead of near the end
Since we don't need to have a fast reasoner and a SPARQL endpoint in the same system, this opens up using quite a large set of possible tools.
There is a long list here: http://owl.cs.manchester.ac.uk/tools/list-of-reasoners/
We need @brodaric and anyone familiar with reasoning technologies to pick the right one.
GSIP content negotiation can override the Accept header by passing a f=
parameter using a list of well known format key (stored in the configuration). right now, it's clunky code in each handler that tests this override before invoking the code.
A cleaner way to do this is to implement a PreMarcher change the Accept header
import java.io.IOException;
import java.util.ArrayList;
import javax.ws.rs.container.ContainerRequestContext;
import javax.ws.rs.container.ContainerRequestFilter;
import javax.ws.rs.container.PreMatching;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.ext.Provider;
@Provider
@PreMatching
public class FormatRequestFilter implements ContainerRequestFilter {
@Override
public void filter(ContainerRequestContext requestContext) throws IOException {
String format = requestContext.getUriInfo().getQueryParameters().getFirst("f");
ArrayList<String> mediaTypes = new ArrayList<>();
if ("application/json".equals(format) || "json".equals(format))
mediaTypes.add(MediaType.APPLICATION_JSON);
if ("text/html".equals(format) || "html".equals(format))
mediaTypes.add(MediaType.TEXT_HTML);
if (mediaTypes.size() > 0)
requestContext.getHeaders().put("Accept", mediaTypes);
}
}
this will override the header before routing the call to the right handler
The property to link a NIR to a DIR should be schemas:subjectOf (from schemas.org)
must replace in configuration, template, instance and javascript code that uses it to identify representations in the landing page
Right now, GSIP use well known extension. OGC API recommend that we use mime-type instead.
We can configure GSIP to accept both.
Assess technologies that can be leveraged on AWS.
on GSIP (see on stage http://w-stf-a128913.nrn.nrcan.gc.ca:8085/gsip/app/index.html) , when a GeoJSON is loaded on the map, it comes in multiple copies (not sure why the number of copies is different)
The script from “https://geoconnex.ca/gsip/info/waterwells/qc.1985_1551_300?f=jsonld&callback=jQuery32103053047573287002_1583436292316” was loaded even though its MIME type (“application/json”) is not a valid JavaScript MIME type. index.html
Default JSON-LD generated by JENA is not conformant with SELFIE pattern, must try these
https://jena.apache.org/documentation/io/rdf-output.html#json-ld
Right now, a single feature is displayed (except Borehole). We should have dcat:Dataset that could be loaded as a whole in the map (or exposed to whoever wants it). À la WFS 3.0
As per https://geoconnex.ca/gsip/info/LOD_Node/CAN_Hydro_LOD_Node.
We would like to demonstrate the harvesting and registration of linkages across nodes. In order to demonstrate this effectively, we need a connected node.
Currently there is a gap between CHyF pilote-project and GSIP when one wants to find the closest hydrometric station to a point or in a catchment for example. One way is to fetch all the watersheds, get the geometry, find which one intersects our point and look for a relation with an hydrometric station.
Some alternatives are available and could help in simplifying the linkage between both application.
We have to find out which solution is best suited for our needs.
In one case, we have a representation that goes to a private ftp server. This u/p is coded in the ftp url (ftp://user:password@,,,). This means that this information is now visible publically on the web (because the proxy pulls the data from the ftp). Probably a bad idea, if it's password protected, the user should have to provide credential (if it has them) when redirected to the ftp site.
Create an access point to dump all triples that are cross nodes
following this patter:
The project could make a good use of an open source licence like some other public projects.
https://github.com/NRCan/geo-deep-learning/blob/master/LICENSE
There is also this thing
https://open.canada.ca/en/open-government-licence-canada
Now that we have more links thanks to Dave, should we add them in our own ?
Or should we use this opportunity to show that nodes does not necessary have symmetric content ?
@heryk : When we click on something on the map ( a catchment), we get a bubble showing relations to other features and some might have multiple pushpin for geojson representation. Is it possible to have a label for each of the pushping when the mouse hovers above it
Because we serialize "format" in the RDF, to be consistent, we should use the same term, therefore use "format="
The info page should know it has been called directly and now from redirection from /id/. We should either block (400 bad request) - or at least have a warning in the page stating that user might be trying to use this page as a persistent identifier.
Establish a common ontology/vocabulary for naming concepts and features. Consider HyFeatures.
This URI returns an empty page
https://geoconnex.ca/id/mapunit/292441/213
it's supposed to be a mapped features
We use
@prefix gw: <http://geosciences.ca/def/groundwater#>.
should be harmonised to one of the options (around http(s)://www.opengis.net/def/{ontology}
) proposed in https://github.com/opengeospatial/GeoSciML/tree/master/documents
Some features in CHyF are transient - they only exist in a particular version of the database. This is the case for small streams that may or may not exist to complete the network (?) or move and split in different segments. Assigning a persistent URI to those feature is an issue because it might not exists in a future iteration. But we still need them because they might be referred to (my a monitoring station, or by another segment).
Maybe one way to address this would be to use HTTP 301 (moved permanently) to tell client that the segment (the real feature) does not exist anymore (this means the system will need to keep track of them for a certain time).
Another solution is that we don't create Non-Information Resource (NIR) for transient features and only allow access to non transient representation through a persistent one. This means we need another kind of NIR -> IR link . The only one we support right now is "seeAlso", linking a NIR to a IR.
Hey @denevers thanks for setting this up! Can we add https://github.com/jkreft-usgs as a collaborator?
for example
https://cida-test.er.usgs.gov/chyld-pilot/info/LOD_Node/US_Hydro_LOD_Node
https://cida-test.er.usgs.gov/chyld-pilot/info/hu/041504040205
don't have any representations because the repository still uses rdfs:seeAlso
instead of schemas:subjectOf
and node information is not correct either
Currently, when a MIR (/info/) is requested from GSIP, it can "know" what NIR we are referring to by replacing /info/ with /id/. This assumption is embedded in GSIP architecture. This prevents GSIP MIR template to be used used with arbitrary PID. A use case proposed by @dblodgett-usgs is to have the NIR managed by a PID that would redirect to GSIP to do the MIR job (pull data from the storage and build the page). Right now it's not possible because GSIP will always assume that the MIR is a representation of a NIR built by replacing /info/ with /id/.
This logic is embedded in Information.java
public String getIdResource(UriInfo uriInfo)
{
// get the /id/ resource and convert to persistant
String persistentUri = Manager.getInstance().getConfiguration().getParameterAsString(PERSISTENT_URI, BASE_URI);
// this one is called by /info/, but the real NIR is /id/
StringBuilder infoUri = new StringBuilder(persistentUri + "/id");
boolean first = true;
for(PathSegment segment: uriInfo.getPathSegments())
{
if (first) {first = false; continue;}
infoUri.append("/"+segment.getPath());
}
//Logger.getAnonymousLogger().log(Level.INFO, "processed URI:" + uriInfo.getPath());
//Logger.getAnonymousLogger().log(Level.INFO,"/id/ URI:" + infoUri);
return infoUri.toString();
//
}
(https://github.com/NRCan/GSIP/blob/master/src/nrcan/lms/gsc/gsip/Information.java)
One solution would be to add a hook to a scripting language (Nashorn) in python or javascript - or a regexp ? that user could configure to provide custom MIR->NIR transformation. Absence of hook would result in current transformation logic (info -> id)
Right now, all HTML landing page uses a single HTML template. It should be possible to configure many templates depending of the URI pattern.
https://geoconnex.ca/data/gwmon/WML2/historical/GIN/prj_27.53517?f=geojson
return
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="https://gin.gw-info.net/service/ngwds/en/data/geojson/monitoring/prj_27.53517">here</a>.</p>
</body></html>
The HTML representation should display schema.org#image automatically. It does not, probably deleted by mistake when converting XSLT based HTML create to FreeMarker
Jena takes a lots of time to compile small RDF model and it might now scale very well when we add more data. This is particularly an issue when using EmbeddedStore. We are using full OWL reasoner, maybe we could settle for something less demanding (we need a limited set of functionnalities like symmetrics, inverseOf, transitivity and inheritance (others ).
Another option is to NOT use RDF and rely on a clever relational database for the whole database and just pull RDF from it to create the landing page.
On startup I am getting the following error:
Failed to load /usr/local/tomcat/webapps/gsip/repos/gsip/ontology.ttl
CHyLD | org.apache.jena.riot.RiotException: [line: 64, col: 1 ]
Triples not terminated by DOT
Could you build a release and push it into a release here: https://github.com/NRCan/GSIP/releases or somewhere else accessible? I'd like to build a docker container and need a built artifact to start from.
Not clear to me how to build and deploy the app. Am I missing that in the docs?
While testing a prototype harvester, we generated HTTP 500 errors. In this special case, we asked for three datasets from the Node (https://geoconnex.ca/gsip/info/LOD_Node/CAN_Hydro_LOD_Node) "simultaneously" (in quick succession). The system eventually recovers (?)
If a /id/ is request in a mime-type that is not one of the 4 mime type supported (or not an hypermedia), and if there is only 1 representation in that format, the conneg can 303 directly to that resource.
eg: /id/ resource is available in .pdf and only one representation if pdf is available, the conneg process will 303 directly to this .pdf. Otherwise, it will return 404, UNLESS the accept header also has one of the aforementioned hypermedia in the preferred list.
Might want to remove (or disable) the [H] button
voir sur le staging:
http://w-stf-a128913.nrn.nrcan.gc.ca/info/swmonitoring/MDDELCC_030415
et
http://w-stf-a128913.nrn.nrcan.gc.ca/gsip/info/LOD_Node/CAN_Hydro_LOD_Node
Les blocs de représentations sont pas très beaux
Information handler uses this ugly code
if (format != null && format.trim().length() > 0)
{
// TODO: use the Configuration instead
if ("rdf".equalsIgnoreCase(format) || "application/rdf+xml".equalsIgnoreCase(format)) of = InfoOutputFormat.ioRDFXML;
if ("ttl".equalsIgnoreCase(format) || "text/turtle".equalsIgnoreCase(format)) of = InfoOutputFormat.ioTURTLE;
if ("xml".equalsIgnoreCase(format) || "text/xml".equalsIgnoreCase(format)) of = InfoOutputFormat.ioXML;
if ("html".equalsIgnoreCase(format) || "htm".equalsIgnoreCase(format) || "text/html".equalsIgnoreCase(format)) of = InfoOutputFormat.ioHTML;
if ("json".equalsIgnoreCase(format) || "jsonld".equalsIgnoreCase(format) || "application/ld+json".equalsIgnoreCase(format)) of = InfoOutputFormat.ioJSONLD;
// otherwise, file not found
if (of == InfoOutputFormat.ioUnknown)
return Response.status(HttpStatus.SC_UNSUPPORTED_MEDIA_TYPE).entity(format +" extension is not supported : use html/ttl/rdf/xml/json").build();
}
instead of using list of mime-types in configuration file.
https://geoconnex.ca/gsip/info/LOD_Node/CAN_Hydro_LOD_Node
Representation: should be plural
https://sgillies.net/2014/05/22/the-geojson-media-type.html
Update (2019-11-13): Since the publication of RFC 7946 GeoJSON has a new official media type, application/geo+json.
Need to reflect the repository name change from GSIP to GeoConnex
eg:
https://geoconnex.ca/data/LOD_Node/CAN_Hydro_LOD_Node?f=json
should return 404 because json is not available (or should it ?). It returns 204 (no content) - not technically false, but not what is expected.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.