nrcan / gsip Goto Github PK

In data configuration (/data/), an attribute is used to override the media type (if the requested media type is not exactly what is being returned). Except that attribute is call "header", which is misleading because it give the impression any http header could be overridden. Should be "media-type" or "alt-media-type".

In order to demonstrate the harvesting of linkages, there is an immediate need for more relations. Currently, there is one relation encoded in the harvester:

<https://cida-test.er.usgs.gov/chyld-pilot/id/hu/041504081604>
<https://www.opengis.net/def/hy_features/ontology/hyf/lowerCatchment>
    <https://geoconnex.ca/id/catchment/02OJ*CA>.

@heryk When there are too many features one on top of the other and one tries to click, the message says "Too many wells" - should be "too many features"

Wondering what the best path is going to be to get HY_Features relations implemented in GSIP. I'd be happy to contribute if @tomkralidis or @denevers can point me to the RTFM and / or the files I can hack on.

Make sure feature doesn't disappear when you select a new one.

Depending of the usage, multiple /info/ representations might be necessary. The challenge and how to negotiate the a particular info for a given /id/. Since /id/ there should be only one /id/ per real world feature in a given governance, the resolution to multiple /info/ cannot rely on the /id/ structure.

One can rely only on

• specialised format in the content negotiation (or subtype of formats, such as text/html; subtype=gin
• profile negotiation (in http header)
• parameters (profile=gin)

or a combination of those options.

Put the JSON-LD within the tag instead of near the end

Since we don't need to have a fast reasoner and a SPARQL endpoint in the same system, this opens up using quite a large set of possible tools.

There is a long list here: http://owl.cs.manchester.ac.uk/tools/list-of-reasoners/
We need @brodaric and anyone familiar with reasoning technologies to pick the right one.

GSIP content negotiation can override the Accept header by passing a f= parameter using a list of well known format key (stored in the configuration). right now, it's clunky code in each handler that tests this override before invoking the code.

A cleaner way to do this is to implement a PreMarcher change the Accept header

import java.io.IOException;
import java.util.ArrayList;

import javax.ws.rs.container.ContainerRequestContext;
import javax.ws.rs.container.ContainerRequestFilter;
import javax.ws.rs.container.PreMatching;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.ext.Provider;

@Provider
@PreMatching
public class FormatRequestFilter implements ContainerRequestFilter {

	@Override
	public void filter(ContainerRequestContext requestContext) throws IOException {
		String format = requestContext.getUriInfo().getQueryParameters().getFirst("f");
		ArrayList<String> mediaTypes = new ArrayList<>();
		if ("application/json".equals(format) || "json".equals(format))
			mediaTypes.add(MediaType.APPLICATION_JSON);
		if ("text/html".equals(format) || "html".equals(format))
			mediaTypes.add(MediaType.TEXT_HTML);
		if (mediaTypes.size() > 0)
			requestContext.getHeaders().put("Accept", mediaTypes);
		
	}

}

this will override the header before routing the call to the right handler

The property to link a NIR to a DIR should be schemas:subjectOf (from schemas.org)

https://schema.org/subjectOf

must replace in configuration, template, instance and javascript code that uses it to identify representations in the landing page

Right now, GSIP use well known extension. OGC API recommend that we use mime-type instead.
We can configure GSIP to accept both.

Assess technologies that can be leveraged on AWS.

• Blazegraph
• Neo4J
• Neptune (Blazegraph on AWS)
• Dgraph

on GSIP (see on stage http://w-stf-a128913.nrn.nrcan.gc.ca:8085/gsip/app/index.html) , when a GeoJSON is loaded on the map, it comes in multiple copies (not sure why the number of copies is different)

The script from “https://geoconnex.ca/gsip/info/waterwells/qc.1985_1551_300?f=jsonld&callback=jQuery32103053047573287002_1583436292316” was loaded even though its MIME type (“application/json”) is not a valid JavaScript MIME type. index.html

Default JSON-LD generated by JENA is not conformant with SELFIE pattern, must try these

https://jena.apache.org/documentation/io/rdf-output.html#json-ld

Right now, a single feature is displayed (except Borehole). We should have dcat:Dataset that could be loaded as a whole in the map (or exposed to whoever wants it). À la WFS 3.0

As per https://geoconnex.ca/gsip/info/LOD_Node/CAN_Hydro_LOD_Node.

We would like to demonstrate the harvesting and registration of linkages across nodes. In order to demonstrate this effectively, we need a connected node.

Currently there is a gap between CHyF pilote-project and GSIP when one wants to find the closest hydrometric station to a point or in a catchment for example. One way is to fetch all the watersheds, get the geometry, find which one intersects our point and look for a relation with an hydrometric station.

Some alternatives are available and could help in simplifying the linkage between both application.

1. Add a geometric component on each item so that it can be searched via a GeoSPARQL query.
2. Load the HUC 12 watersheds in the CHyF model. Its ID would be used to access the LOD API and find the information needed.
3. Build an API on top of GSIP that takes a geometry and returns the ID of the watershed(s) that intersects it.
4. ...

We have to find out which solution is best suited for our needs.

In one case, we have a representation that goes to a private ftp server. This u/p is coded in the ftp url (ftp://user:password@,,,). This means that this information is now visible publically on the web (because the proxy pulls the data from the ftp). Probably a bad idea, if it's password protected, the user should have to provide credential (if it has them) when redirected to the ftp site.

Create an access point to dump all triples that are cross nodes

following this patter:

http://localhost:8080/gsip/node/cross?f=ttl

The project could make a good use of an open source licence like some other public projects.

https://github.com/NRCan/geo-deep-learning/blob/master/LICENSE

There is also this thing
https://open.canada.ca/en/open-government-licence-canada

Now that we have more links thanks to Dave, should we add them in our own ?
Or should we use this opportunity to show that nodes does not necessary have symmetric content ?

@heryk : When we click on something on the map ( a catchment), we get a bubble showing relations to other features and some might have multiple pushpin for geojson representation. Is it possible to have a label for each of the pushping when the mouse hovers above it

Because we serialize "format" in the RDF, to be consistent, we should use the same term, therefore use "format="

The info page should know it has been called directly and now from redirection from /id/. We should either block (400 bad request) - or at least have a warning in the page stating that user might be trying to use this page as a persistent identifier.

Establish a common ontology/vocabulary for naming concepts and features. Consider HyFeatures.

This URI returns an empty page

https://geoconnex.ca/id/mapunit/292441/213

it's supposed to be a mapped features

We use

@prefix gw: <http://geosciences.ca/def/groundwater#>.

should be harmonised to one of the options (around http(s)://www.opengis.net/def/{ontology}) proposed in https://github.com/opengeospatial/GeoSciML/tree/master/documents

Some features in CHyF are transient - they only exist in a particular version of the database. This is the case for small streams that may or may not exist to complete the network (?) or move and split in different segments. Assigning a persistent URI to those feature is an issue because it might not exists in a future iteration. But we still need them because they might be referred to (my a monitoring station, or by another segment).

Maybe one way to address this would be to use HTTP 301 (moved permanently) to tell client that the segment (the real feature) does not exist anymore (this means the system will need to keep track of them for a certain time).

Another solution is that we don't create Non-Information Resource (NIR) for transient features and only allow access to non transient representation through a persistent one. This means we need another kind of NIR -> IR link . The only one we support right now is "seeAlso", linking a NIR to a IR.

Hey @denevers thanks for setting this up! Can we add https://github.com/jkreft-usgs as a collaborator?

We are still using the http beta

http://geo.weather.gc.ca/geomet-beta/features/collections/hydrometric-monthly-mean/items?STATION_NUMBER=02OJ007&f=json

should be

https://geo.weather.gc.ca/geomet/features/collections/hydrometric-monthly-mean/items?STATION_NUMBER=02OJ007&f=json

for example

https://cida-test.er.usgs.gov/chyld-pilot/info/LOD_Node/US_Hydro_LOD_Node
https://cida-test.er.usgs.gov/chyld-pilot/info/hu/041504040205

don't have any representations because the repository still uses rdfs:seeAlso instead of schemas:subjectOf and node information is not correct either

Currently, when a MIR (/info/) is requested from GSIP, it can "know" what NIR we are referring to by replacing /info/ with /id/. This assumption is embedded in GSIP architecture. This prevents GSIP MIR template to be used used with arbitrary PID. A use case proposed by @dblodgett-usgs is to have the NIR managed by a PID that would redirect to GSIP to do the MIR job (pull data from the storage and build the page). Right now it's not possible because GSIP will always assume that the MIR is a representation of a NIR built by replacing /info/ with /id/.

This logic is embedded in Information.java

public  String getIdResource(UriInfo uriInfo)
	{
		// get the /id/ resource and convert to persistant
		String persistentUri = Manager.getInstance().getConfiguration().getParameterAsString(PERSISTENT_URI, BASE_URI);
		// this one is called by /info/, but the real NIR is /id/
		StringBuilder infoUri = new StringBuilder(persistentUri + "/id");
		boolean first = true;
		for(PathSegment segment: uriInfo.getPathSegments())
		{
			if (first) {first = false; continue;}
			infoUri.append("/"+segment.getPath());
		}
		//Logger.getAnonymousLogger().log(Level.INFO, "processed URI:" + uriInfo.getPath());
		//Logger.getAnonymousLogger().log(Level.INFO,"/id/ URI:" + infoUri);
		return infoUri.toString();
		// 
	}

(https://github.com/NRCan/GSIP/blob/master/src/nrcan/lms/gsc/gsip/Information.java)

One solution would be to add a hook to a scripting language (Nashorn) in python or javascript - or a regexp ? that user could configure to provide custom MIR->NIR transformation. Absence of hook would result in current transformation logic (info -> id)

Right now, all HTML landing page uses a single HTML template. It should be possible to configure many templates depending of the URI pattern.

https://geoconnex.ca/data/gwmon/WML2/historical/GIN/prj_27.53517?f=geojson

return

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="https://gin.gw-info.net/service/ngwds/en/data/geojson/monitoring/prj_27.53517">here</a>.</p>
</body></html>

Architecture

The HTML representation should display schema.org#image automatically. It does not, probably deleted by mistake when converting XSLT based HTML create to FreeMarker

Jena takes a lots of time to compile small RDF model and it might now scale very well when we add more data. This is particularly an issue when using EmbeddedStore. We are using full OWL reasoner, maybe we could settle for something less demanding (we need a limited set of functionnalities like symmetrics, inverseOf, transitivity and inheritance (others ).

Another option is to NOT use RDF and rely on a clever relational database for the whole database and just pull RDF from it to create the landing page.

On startup I am getting the following error:

Failed to load /usr/local/tomcat/webapps/gsip/repos/gsip/ontology.ttl
CHyLD | org.apache.jena.riot.RiotException: [line: 64, col: 1 ]
Triples not terminated by DOT

Could you build a release and push it into a release here: https://github.com/NRCan/GSIP/releases or somewhere else accessible? I'd like to build a docker container and need a built artifact to start from.

Not clear to me how to build and deploy the app. Am I missing that in the docs?

While testing a prototype harvester, we generated HTTP 500 errors. In this special case, we asked for three datasets from the Node (https://geoconnex.ca/gsip/info/LOD_Node/CAN_Hydro_LOD_Node) "simultaneously" (in quick succession). The system eventually recovers (?)

If a /id/ is request in a mime-type that is not one of the 4 mime type supported (or not an hypermedia), and if there is only 1 representation in that format, the conneg can 303 directly to that resource.

eg: /id/ resource is available in .pdf and only one representation if pdf is available, the conneg process will 303 directly to this .pdf. Otherwise, it will return 404, UNLESS the accept header also has one of the aforementioned hypermedia in the preferred list.

Might want to remove (or disable) the [H] button

voir sur le staging:

http://w-stf-a128913.nrn.nrcan.gc.ca/info/swmonitoring/MDDELCC_030415
et
http://w-stf-a128913.nrn.nrcan.gc.ca/gsip/info/LOD_Node/CAN_Hydro_LOD_Node

Les blocs de représentations sont pas très beaux

Information handler uses this ugly code

if (format != null && format.trim().length() > 0)
		{
			// TODO: use the Configuration instead
			if ("rdf".equalsIgnoreCase(format) || "application/rdf+xml".equalsIgnoreCase(format)) of = InfoOutputFormat.ioRDFXML;
			if ("ttl".equalsIgnoreCase(format) || "text/turtle".equalsIgnoreCase(format)) of = InfoOutputFormat.ioTURTLE;
			if ("xml".equalsIgnoreCase(format) || "text/xml".equalsIgnoreCase(format)) of = InfoOutputFormat.ioXML;
			if ("html".equalsIgnoreCase(format) || "htm".equalsIgnoreCase(format) || "text/html".equalsIgnoreCase(format)) of = InfoOutputFormat.ioHTML;
			if ("json".equalsIgnoreCase(format) || "jsonld".equalsIgnoreCase(format) || "application/ld+json".equalsIgnoreCase(format))  of = InfoOutputFormat.ioJSONLD;
			// otherwise, file not found
			if (of == InfoOutputFormat.ioUnknown)
				return Response.status(HttpStatus.SC_UNSUPPORTED_MEDIA_TYPE).entity(format +" extension is not supported : use html/ttl/rdf/xml/json").build();

			
		}

instead of using list of mime-types in configuration file.

https://geoconnex.ca/gsip/info/LOD_Node/CAN_Hydro_LOD_Node

Representation: should be plural

https://sgillies.net/2014/05/22/the-geojson-media-type.html

Update (2019-11-13): Since the publication of RFC 7946 GeoJSON has a new official media type, application/geo+json.

Need to reflect the repository name change from GSIP to GeoConnex

eg:

https://geoconnex.ca/data/LOD_Node/CAN_Hydro_LOD_Node?f=json

should return 404 because json is not available (or should it ?). It returns 204 (no content) - not technically false, but not what is expected.