geonetwork / geonetwork-ui Goto Github PK

Having a "local" docker image would allow running end-to-end tests without relying on an external deployment of GeoNetwork, as well as running GeoNetwork-UI apps locally

This web component takes a metadata record UUID as input and shows a visual preview of the data on a map, including switching different layers.

This app could work with server-side rendering, keeping a cache of all records in a catalog and serving only the minimal information for search engines crawlers.

This could be seen as the equivalent of the Datahub but for machines.

Also this app should be very easy to deploy and enable high-quality SEO.

Currently, the content of the formats dropdown is filled by a terms aggregation on the format index field.
The issue is that the distribution format is not well used in GeoNetwork, it's often the source raw format which is mentionned (eg shp, postgis). For instance, if we have an online resource with a WFS protocol, the distribution format could be wfs, or csv,shp,xls,geojson ...
We were considering using links protocol, which seems more accurate, but still does not work, as we don't always know the mimetype of the linked resource. We also expect human readable formats.

We also need to avoid duplicates (eg shp, shapefile, esri shapefile).

In the organization tab, we display the number of records per organization, which sometimes becomes out of sync with the number of search results associated with the organization (upon clicking on the organization block).

This is due to an implementation detail: to retrieve the emails of the organizations in the index, we need to perform a "nested" aggregation on the contactForResource object.

[obj] contactForResource
  - organisation
  - email

We are using the contactForResource object instead of the OrgForResource field, which is not an object but a string. Unfortunately, there seems to be a discrepancy between the two.

It appears that some contactForResource.organisation values are not indexed in OrgForResource.

TODO: Inspect the organization indexing to achieve a perfect synchronization between the two index fields.

There is no real actual domain definition. Models design have been defined incrementally, mostly mapped on Elastisearch models.

We want to introduce a real domain definition, within the common/domain library, just defining types and interfaces with clean architecture in mind.

• Models (DTO)
• Repository (Interface)

The core component of this domain will be the Metadata Record type which will be based on the pivot format.
The pivot format will be the core model for any transformation (schemas, ogc api etc...).

The GeoIDE V0 forms contain links to Mapserver which exposes a WFS which does not support the geojson format.

Ex https://www.geo2france.fr/geonetwork/srv/api/records/fr-120066022-jdd-f20f8125-877e-46dc-8cf8-2a8a372045eb/formatters/xml?approved=true

In the datahub, the tabular view is unavailable because it is based on the geojson.

1. support GML in the datahub
2. see with geoide so that they support geojson (present in carto2 but not in carto0)

datafeeder.upload.error.subtitle.fileSize mentions 30Mb - should be variabilized

Would it make sense to have github actions build docker images for the apps from this repository ?

Example workflow:
https://github.com/georchestra/georchestra/blob/master/.github/workflows/datahub.yml

Purpose
Dataviz component can take a metadata record, fetch the data (if exists in csv, json, xsls, GML) and display the data in a chart.
Different options helps to define the chart

• type of chart
• axis
• yaxis
• aggregation function

Usage

• This component is used in the Datahub record page, beside the map and the table
• This component is exported as a Web Component
• This component can be loaded in a full HTML page in a standalone mode

Screenshot

Snyk offers github actions: https://github.com/snyk/actions

This would come in replacement of SonarCloud, which has been disabled on GeoNetwork-UI for various reasons.

When I import data from the test datafeeder (for example, https://dev.geo2france.fr/geonetwork/srv/eng/catalog.search#/metadata/9985fe5e-6760-49ff-89d6-e6986719cce4), I can't find it using the search input on the datahub. It seems like it's not indexed (I can find it if I enter the complete URL in the browser: https://dev.geo2france.fr/datahub/dataset/9985fe5e-6760-49ff-89d6-e6986719cce4).

• update caching actions
• update set-env usage (deprecated)
• build artifacts on demand
  Currently the CI builds docker images and publishes apps on GitHub pages for every commit on a PR; this takes a long time and is not part of the QA process.
  A solution could be to only do these tasks on demand, e.g. by offering a checkbox that someone can check; this would then trigger the two additional workflows.
  See: https://glebbahmutov.com/blog/re-run-the-tests-with-checkbox/
• more cleanup related to storybook-wc
• stop building all the applications on PR (only master)

Planned improvements:

• better handling of search reset when doing pagination/filtering
• isolate implementation related to GN4/ElasticSearch to allow support for other backends in the future

Upgrade the whole workspace core libraries (Nx, Angular etc...).

See what are the new features from those new versions:

• Angular signals
• Rustpack
• esbuild
• Storybook 8 with Vite

The datahub would benefit a lot from a component which allows the user to design data visualizations, something like...

dunno if there are specific requirements, but with npm 8.1.2/nodejs v16.13.1 the build fails

> nx run demo:build:production datahub
'_' is not found in schema
> nx run map-viewer:build:production datahub
'_' is not found in schema
> nx run search:build:production datahub
'_' is not found in schema
> nx run webcomponents:build:production datahub
'_' is not found in schema
> nx run util-data-fetcher:build datahub
Compiling TypeScript files for project "util-data-fetcher"...
Done compiling TypeScript files for project "util-data-fetcher".
> nx run datahub:build:production datahub
'_' is not found in schema
 
 >  NX   Running target "build" failed
   Failed tasks:
   
   - datafeeder:build:production
   - demo:build:production
   - map-viewer:build:production
   - search:build:production
   - webcomponents:build:production
   - datahub:build:production

according to buildbot logs it started failing with #276.

the command run by the bot is npm run build datahub -- --base-href="/datahub/" after npm i.

after looking at the docs, i changed the command for npm run build -- datahub --prod but the failure is the same. And also same thing after ditching node_modules.

There are several geonetwork-ui related docker images in https://hub.docker.com/u/geonetwork but one would probably expect to find also a composition file to spin a datahub instance in a few minutes (front + back).

Introduced in

Keeping improve web component architecture and integration

• get rid open open api default basePath
• improve doc
• check if any issue with multiple searches
• improve css variable mechanism
• revamp demo pages

In the search view, use the organisation logo if no thumbnail is present.

Testing does not need to be too deep but should make sure that:

• applications and web components correctly start
• the main screens can be accessed, e.g. search, newsfeed and record view for the Datahub
• if there is a change in the final render, let it be known

on a testing instance with gn4, the catalog on /geonetwork/ shows 77 metadata (as anonymous user), but a datahub instance deployed on the same georchestra (via georchestra/ansible#93) shows 105 entries. it seems to me that all the geonetwork templates are also displayed.. which is confusing.

@fvanderbiest @pmauduit ?

Related to #517

Email in the record view must have a content_copy icon beside it, pointer cursor and when clicked, content is copied to clipboard and a toast or alert tells the user that content is 'Copied to clipboard'

Add a strategy to retrieve metadata record organization.

• metadata: fetch the organization from the first resource contact of the record
• group: fetch the metadata groupOwner

Rules for the groups:

• Logos, descriptions should be taken from the groups
• Organizations displayed in the records should be groups and not metadata orgs
• When filtering on an organization, groups should be used instead of metadata orgs
• The organizations facets should use the groups instead of the metadata orgs

When you add a record as a favorite, the state of the figure beside the favorite is only updated within the component you clicked from, and is not propagated to other components which would refer to the same metadata (eg the metadata view, the feed view).

This lead to a de-synchronization of the favorite amount.

As the administrator of the datahub, I would like it to be harvestable by data.gouv.fr.

On one hand, data.gouv.fr offers harvesting mechanisms for:

• CKAN
• DKAN
• ODS
• MAAF
• DCAT

On the other hand, GN 4 does not provide virtual CSW, and in any case, the INSPIRE gateway is no longer maintained (and would be best to avoid).

Since geonetwork-ui runs independently of the GeoNetwork Java backend, we have no guarantee that the latter is indeed reachable. When not reachable the whole UI kind of stops working without any useful hint for the user, even made worse by the fact that translations are unavailable.

geonetwork-ui should detect of the backend is unreachable as early as possible and display an explicit message explaining what is going on in layman's terms. This is also true for webcomponents.

afaict right now it's hardcoded to 560px in https://github.com/geonetwork/geonetwork-ui/blob/main/apps/datahub/src/app/home/home-page/home-page.component.html#L2 but after looking a bit how all this works, i'm not sure if [fullHeightPx] is a config stanza to put in the [theme] section of default.toml.

i've looked a bit and to me the configurable values are the ones listed in https://github.com/geonetwork/geonetwork-ui/blob/main/libs/util/app-config/src/lib/app-config.ts#L145 but i'm not sure if adding some here would make them available to the html templates.. so hints welcome ?

ofc i can hardcode 300px and rebuild datahub, but that's suboptimal, and breaks the display of the searchbar anyway.. so how can we have that header size more configurable, and the searchbar in the middle ?

This application will offer a basic map viewer with the following features:

• Add layers from a GeoNetwork catalog or third party sources (e.g. OGC endpoints, etc.)
• Manipulate layers (order, opacity, delete)
• Show legend
• Import and export maps using a file format to be determined
• Print the map using e.g. https://github.com/camptocamp/inkmap

The "resourceType" field is not filled for records harvested by the SimpleUrl harvester.

Goal:

• The field should be set to "dataset" by default on all transformations
• This includes at least the OpenDataSoft transformation

Essential for the long-term pairing of metadata organizations with those in the system.

Looking on https://demo.georchestra.org/datahub/

• The count of records is bounded to 10K
• The count of organisations is bounded to 1K

Solution:

• orgs: increase the bucket limit
• records: track_total_hits:true to the ES query

Geoclip provides an API that allows you to download data in CSV and XLSX formats, and we can easily retrieve it through web scraping, for example: https://www.geo2france.fr/datahub/dataset/act_cond_emp.pxx_sal15p_pre

You can download the file in CSV and XLSX formats. However, the file formats are not recognized, so it is not possible to filter by format type, and there is also no tabular preview available.

The XML code of the resource seems fine, as it matches well with other functioning records (for example: https://www.geo2france.fr/datahub/dataset/50f0712b-80e8-4b44-9e90-0cad4e5f64b0).

Two ideas :

• Either we can request to evolve the metadata protocol with DOWNLOAD:mime-type;
• or we can implement format detection from the URL.

I've just found an issue on https://www.geo2france.fr/datahub/dataset/cc7af6df-b79a-47fc-a9ca-ab812c9c376d

Zooming on the map with shift + mouse ("box-zoom") is weird: the box is displaced.

is it possible to add support for html markup in text fields example record here https://geonetwork-test.tern.org.au/geonetwork/srv/eng/catalog.search#/metadata/db766c37-f00d-4dc4-aa1a-325b284da1a0
thanks
Jenny

https://hub.docker.com/r/geonetwork/geonetwork-ui-datahub/tags?page=1&name=latest

So we can refer to it in the geOrchestra docker composition

Would be very useful for eg deliberations like https://dev.geo2france.fr/datahub/dataset/mel-deliberations-new

The contact block has to be reworked.

Goal:

• Hovering on links should show an underline instead of changing the text color

• Link to the website should be shown separately (see screenshot)

• beside the email, there must be a copy icon and when clicked, displays a tooltip "Copied to clipboard"

Example: https://www.geo2france.fr/datahub/dataset/8ac4f051-e8c0-11ec-b13b-186571a04de2

Tested on firefox, I had the following lookup failing:
https://github.com/geonetwork/geonetwork-ui/blob/master/libs/ui/inputs/src/lib/chips-input/chips-input.component.ts#L48

because this.translate.currentLang was set to "en-US".

Applying the following hack fixed my issue:
georchestra@9a93021

In the datahub home page, in the search tab, the advanced filter panel contains a list of filter.
This list should be configurable.

# The advanced search filters available to the user can be customized with this setting.
# The following fields can be used for filtering: 'publisher', 'format', 'publicationYear', 'standard', 'inspireKeyword', 'topic', 'isSpatial', 'license'
# any other field will be ignored
advanced_filters = ['publisher', 'format', 'publicationYear', 'topic', 'isSpatial', 'license']

• Nx 15
• TailwindCSS 3

At the moment you can only sort the search results (i.e. datasets) by Relevancy, Last updates or Popularity. Quite a common sort option is the title (i.e. sorting the results in alphabetical order). Would it be possible to have that added in?

• Technical proposal voted by the PSC
• The vision
• The funding page

Sometimes, looking for something is not enough, we have to look for something somewhere.
Putting the location in the any search input is not accurate, cause it can miss larger scaled datasets.

eg. velo chambery search won't find a velo france dataset.

We will add the possibility to add a location input beside the full text search input like on the following screenshot.

I connect to:

https://gateway.mel.integration.apps.gs-fr-prod.camptocamp.com/datahub/

I'm using the identity provider 'MEL.' After attempting to log in: https://gateway.mel.integration.apps.gs-fr-prod.camptocamp.com/datahub/?login=

This page returns the following error message:

Whitelabel Error Page

This application has no configured error view, so you are seeing this as a fallback.
Mon Jun 19 08:02:12 UTC 2023
[80ef1128-16752] There was an unexpected error (type=Internal Server Error, status=500).

It seems that the server doesn't know which page to display upon returning from the LDAP login.

https://gateway.mel.integration.apps.gs-fr-prod.camptocamp.com/login?error
If I go back to the page, I see that I am successfully logged in. However, I can't navigate to the catalog, mapstore, or import. Instead, I get an error page similar to the one mentioned earlier.

If I log in with a local account, everything works fine.

At the bottom of the page https://www.geo2france.fr/portail/, we have the following block:

The links point to (for example) https://www.geo2france.fr/geonetwork/srv/metadata/afr.afr, but in terms of the overall integration of the solution, we would prefer it to point to https://www.geo2france.fr/datahub/dataset/afr.afr.

Seems to be a concurrency loads of angular sources that create issues while running storybook-wc with severals web components.

https://geonetwork.github.io/geonetwork-ui/master/storybook-wc/

VM1722 main.647ae3ccdecb25ee2746.bundle.js:12042 ERROR DOMException: Failed to execute 'define' on 'CustomElementRegistry': this name has already been used with this registry
    at CustomElementRegistry.target.(anonymous function) [as define] (http://localhost:4202/vendors~main.647ae3ccdecb25ee2746.bundle.js:166506:35)
    at GnResultsListModule.ngDoBootstrap (http://localhost:4202/main.647ae3ccdecb25ee2746.bundle.js:1743:2000)
    at PlatformRef._moduleDoBootstrap (http://localhost:4202/main.647ae3ccdecb25ee2746.bundle.js:21107:689)

Securing the registry does not work

    if(!customElements.get(WC_TAG_NAME)) {
      customElements.define(WC_TAG_NAME, customButton)
    }

If we only load one .stories file of web component, the storybook works.

Should be available in the datahub, if you are logged in.

• information about the user
• amount of your metadatas
• information about your organization
• show your metadatas
• the metadata of your organization