esgf / esgf-web-fe Goto Github PK

From Prashanth:

I'm running web-fe v 2.4.7 which I build from devel. It does not show the tracking id and checksum information for files in a dataset. It did not show this information even when it was running an updated fe from the master branch. On another test machine, I have a stock, unupdated web-fe from Brower Park (web-fe v 2.3.2) which is working fine, displaying the tracking id and checksum.

The technote links have disappeared after the latest commit. These should be there.

File counts that are displayed in the datacart can be misleading if the user has selected "filter over search constraints".

Users don't seem to be able to find data that is OpenDAP enabled - should we have a "service" facet, or somehow expose those links more prominently ?

If there is an error given from any ajax call, there should be a better handler than
the current implementation. As of now, only a simple alert message appears,
but the state of the search remains unchanged. The user must know to find a
different set of search criteria and re-issue the search.

The search results tab should have its own api for clients other than the front end.

This is a problem with the "Filter over text" feature of the data cart. If a dataset has >1000 files the default wget script will not contain files beyond that threshold. Therefore if a user searches for a variable they may get a wget script without any of the files for that variable unless they select "Filter over text".

This has happened when searching for variable "zg" and selecting cmip5.output1.NOAA-GFDL.GFDL-ESM2M.historical.mon.atmos.Amon.r1i1p1.v20111228.

It needs to become much more obvious how users refine by variable.

When the admin adds a group with no description through the shell program:

groupadd somegroup

the front end app for adding users to that group gives a javascript error.

Include a new "Help" tab pointing to a page with at least the following references:

o http://www.esgf.org/wiki/ESGF_Data_Download

o http://www.esgf.org/wiki/ESGF_User_FAQ

plus the link to sending email to esgf-user, pre-populated with some unique subject, or with a blank subject that the users need to fill. Alternatively, we could have a form with subject/body fields that sends email to esgf-user.

The entire front should be completely URL addressable. The URL will describe the state of the user's search, which tab is open, which facets have been used, etc. This will allow a user to bookmark searches via the browser and for other clients (like the BER Data clearinghouse) to navigate to the front end directly.

Users are getting really confused about the radio button "Filter Over Search Constraint" as it often leads to the error message "no files were found" when trying to create a wget script. This is because many dataset-level constraints, such as "ensemble", or "version:abc" text, do not apply to files.

Possible suggested action:

a) Make "Show All" the default, and always re-select it upon loading the page
b) Remove "Filter over search constraints" altogether - always create a wget script that contains all files in a dataset. If data providers want people to be able to download per-variable files, they need to structure their datasets differently (as they should have done as per CMIP5 guidelines). Or, users need to manually edit the wget scripts.

All the front end JS must be linted and compressed to ensure optimal performance of the portal. Place all updates here.

As of now, a user may only utilize the "current selections" box to capture
previous search criteria.

Many modern sites have the "breadcrumb" capability of storing the sequential
path of a search conducted by the user. With a breadcrumb trail, users can
navigate, add, delete, store this path for later searches, and/or share trails
with others.

This feature request will outline the progress of the breadcrumb creation
(which will probably be done in this order):

• Add to path
• Delete from path
• Store the path (either in HTML5 localStorage or DB)
• navigate/jump around path
• share path

As of now, Google Maps is the front end's only external dependency ('external'
meaning outside of the esgf family of libraries). If there is a broken
connection to the web, the entire page crashes because of this dependency
(exceptions are thrown preventing the page from loading all the components).

Thus, try-catch blocks should be inserted at all access points to Google Maps
so as to avoid complete failures of the page.

Goal: LAS will allow users to select, from within the LAS UI, other similar datasets, or model/observations, to compare to

What needs to happen (on the ESGF side): the web-fe needs to pass the current state of the query to the LAS, as well as the index node to query to:

http://esg-datanode.jpl.nasa.gov/las/getUI.do?catid=893EB2D5C79AD40EE2436A3F118649CE_ns_obs4MIPs.NASA-JPL.AIRS.mon.v1
&catalog=http://esg-datanode.jpl.nasa.gov/thredds/esgcet/1/obs4MIPs.NASA-JPL.AIRS.mon.v1.xml#obs4MIPs.NASA-JPL.AIRS.mon.v1
&index_node=esg-datanode.jpl.nasa.gov
&search="project:obs4MIPs&model:Obs-AIRS&query=airs"

NOTE: the URL of the search service to query is:

http://index_node/esg-search/search?.......

NOTE: if needed, the full dataset id can be reconstructed from the catid parameter:

catid=893EB2D5C79AD40EE2436A3F118649CE_ns_obs4MIPs.NASA-JPL.AIRS.mon.v1 --> obs4MIPs.NASA-JPL.AIRS.mon.v1|esg-datanode.jpl.nasa.gov

A local group configured "visible = false" is listed in the "Groups Available" section under the account detail. The background is, that I'd like to see the group hidden here for non-wheelies and thereby not allow users to join the group without accepting the Terms of Use explicitely like they have to when they http-testdownload a file without the needed permission.

Thank you

The front end has a lot of methods that require unit tests. The selenium toolkit may be able to help with this.

From Matthias Buchner:

Hi,
While exercising log ins, authentification, group registrations and downloads on my 1.5 Brower Park installation I ran into some problems. I have three 'buechner' accounts registered in the local database (local, esgf-data.dkrz.de and esg-dn1.nsc.liu.se).

rootAdmin@se38:[esgf-sh]:[admin]> show --au
Users:
username firstname lastname openid
rootAdmin Matthias Buechner https://esg.pik-potsdam.de/esgf-idp/openid/rootAdmin
buechner https://esgf-data.dkrz.de/esgf-idp/openid/buechner
buechner https://esg-dn1.nsc.liu.se/esgf-idp/openid/buechner
buechner Matthias Buechner https://esg.pik-potsdam.de/esgf-idp/openid/buechner

The overview (see attached screenshot) lists all three accounts with the same OpenID. When I alter the details of one entry the change is shown for all three accounts. Also deleting one of the accounts throws an error everytime I click on 'Manage Users'. So I assume the module can't handle multiple accounts with the same username although from different OpenID providers.

This might already have been addressed in the current devel branch but I just want to make sure it will be fixed in the next version.

Thank you
Matthias

Arguably, the current model where a few facets are grayed out in the UI, and become selectable only after two others are selected, is clanky. A user has suggested that all the facet options should be displayed, possibly with a system like the iPhone that has shortcuts "A B C D..." that center the display around that particular letter.
At the very minimum, the user has recommended that we insert a few words of explanation of why those facets are grayed out, and how to make them selectable.

workflow for creating and changing a password should be the same.

The datacart should have its own api for clients other than the front end.

There are problems with trying to include the shadowing border into blueprint.
An obvious solution would be to just wrap the border (which is written as a
repeating image in the y-direction) around the whole blueprint container. I'm
not sure that will work as blueprint automatically adjusts the margins when the
page is resized and coordinating the adjustment of the border may be difficult.
But if somebody knows how to do this, that would certainly be the easiest
solution.

I've attached a couple of powerpoint "slides" to visualize the problem
(apologies for the crudeness of the model - just wanted to put something
together quickly - wish I had autocad!).

On the first slide is the simple outline of the page with most of the required
images involved. The width determined by blueprint (by default) is 950 px (24
grid places at 30px and 23 gaps at 10px). The page would ideally span all grid
points in width ("span-24") - length can be determined by the internal
components for the most part. Thus the page is just a combination of sections
"spanning-24" grids. Note to get these to work, the map and top/bottom border
images would have to be resized to a width of 950px.

In this model, the border in the main content would be overwritten by the
background images (e.g. blue/logos in the header, beige for the main page,
etc). So the page will only have borders on the top and bottom, but not in the
middle.

This may be overcome by allotting the borders (left and right) to have 1 grid
square each, and the "main" content have 22 grid squares.
To do this, however, the border images would have to be 1 grid square and gap
long (40px) and include the background color of that particular section so that
there is no "whitespace" appearing in the page. The second slide shows this
and includes an example of how the images should be adjusted to work for this.

The world map that we have on the front page is great, however, it doesn't show Australia. We should have an alternative image that does show Australia and have that in the distribution. It should be able to be turned on with a simple flag.

Example:
In messages.properties
show.down.under.map=true

It would be great if we could make a short version openid nomenclature for esgf openids when entered in text fields.

Ex:
Long form: https://esg2.nci.org.au/esgf-idp/openid/gavin
to
Sort form: esg2.nci.org.au::gavin

At the input field, this would get interrogated and transformed into the full, "long form" / proper form for the submission.

We should have a mechanism for facets to be listed in the file facets.properties (so that they can be used to query the search back-end), but not show up as facet options in the web-fe. This can be needed, for example, when multiple portals use the same underlying services.

With respect to the current format of each line in the file facets.properties:

facet_short_name=number:facet_long_name:<true|false>:optional_facet_descrip

possible mechanisms include:

a) ignore the facet if the "facet number" is 0 or negative

b) ignore the facet if the facet label is empty

c) include a new field named "visible"

There should be a way to offer "negative" search facets. E.g. results which
are NOT from datanode='bcccsm.cma.gov.cn'.

The autocomplete feature (present in previous versions) was taken out due to some mysterious bugs that were introduced. A "good to have" feature.

There are a lot of files that were either committed from older versions of the
search page or were used for debugging purposes that are no longer used. These
files should be deleted from the git repo to eliminate ambiguity and foster
code readability

The facet side bar must be redesigned. As of now, the bar extends for thousands of pixels when expanded. There should be a way to increase the z-index so that it can hover OVER the main content tabs.

Goal: LAS will be configured "on-demand" by parsing the metadata contained within the specific THREDDS catalog, as opposed to be configured when the node starts up, which can take a long time

What needs to happen (on the ESGF side): the web-fe needs to construct richer LAS URLS that contain the URL of the THREDDS catalog for each dataset

current LAS URL exposed by the front-end:

http://esg-datanode.jpl.nasa.gov/las/getUI.do?catid=893EB2D5C79AD40EE2436A3F118649CE_ns_obs4MIPs.NASA-JPL.AIRS.mon.v1

new LAS URL:

http://esg-datanode.jpl.nasa.gov/las/getUI.do?catid=893EB2D5C79AD40EE2436A3F118649CE_ns_obs4MIPs.NASA-JPL.AIRS.mon.v1
&catalog=http://esg-datanode.jpl.nasa.gov/thredds/esgcet/1/obs4MIPs.NASA-JPL.AIRS.mon.v1.xml#obs4MIPs.NASA-JPL.AIRS.mon.v1

NOTE: all parameter values must be URL-encoded

Users need to understand there is an upper limit to the number of files returned by a wget script. They also need to be able to change that limit.

Ideally I don't see why the limit can't be set much higher than 1000 files. Can this be tuned at each node? If there has to be a limit the web-fe should warn on an attempt to generate a wget script for more files than the limit and allow that limit to be tuned.

The SRM workflow must be cleaned up on the front end. This entails links in the datacart as well as the separate SRM data request page.

It is not possible to navigate the pages of results, add them to the cart, and advance to next page: it reloads and goes back to first page. For the first pages it's ok, but when you have to go to page 35 it takes a while to get there (every time it needs a while, I guess that it is to reload all the data from different servers).

The "sender" address of the email functionality used by the web-fe needs to be read from a property set in esgf.properties. For example, currently the "Forgot Password" functionality fails to work on the NOAA server because of this error:

at java.lang.Thread.run(Thread.java:662)

Caused by: com.sun.mail.smtp.SMTPAddressFailedException: 554 5.7.1 [email protected]: Sender address rejected: Access denied

We should require a minimum of 6 or 8 characters when users create an account. Besides, I think MyProxy will not work with usernames less than 6 characters.

Let's consider the metadata display page for an example dataset:

http://esg-datanode.jpl.nasa.gov/esgf-web-fe/metadataview/obs4MIPs.CNES.AVISO.mon.v1|esg-datanode.jpl.nasa.gov.html

A scientist has commented that the for each field, the values should be ordered alphabetically - this would allow the most important variables to be displayed first, as in:

sea_surface_height_above_geoid
sea_surface_height_above_geoid_number_of_observations ;sea_surface_height_above_geoid_standard_error

The current privacy page is from LLNL. We should have a more general ESGF privacy policy page as the default. And the link pointer should be easy to repoint by the admin.

We should protect ourselves from spam bots creating users on the system by implementing a simple captcha mechanism.

Verified on pcmdi9.

If I select "Advanced Search" from the home page I go to

http://pcmdi9.llnl.gov/esgf-web-fe//live

Note the double slash. The page appears to work but other problems result.
E.g. the animated loading image isn't displayed
(http://pcmdi9.llnl.gov/esgf-web-fe//images/ajax-loader.gif)

Right now, if you want to run the ESGF web portal locally for any reason, you need to setup the Postgres database and related security. You can avoid doing so by switching to "false" the flag:

public final static boolean environmentSwitch = false;

in the file org.esgf.commonui.Utils.java

This setting would be better read from a configuration file of some sort, like a properties file.

Most of the controller and object classes on the front end have no unit tests. We need more coverage for these classes.

The "Display XX datasets per page" selector doesn't work, it only allows to show 10 results per page or 500, not 25 or 50. This forces users to download the files in little packs, as it is not possible to download them all on one pack.

Currently, if this property is set:

enableGlobusOnline=true

ALL files in the data cart will display a GO link. Instead, only those files with a GridFTP endpoint should display the GO option.

Currently, the front end supports the launch of LAS analysis tools with only a
single dataset. This should be extended so that a user can select several
datasets and view them simultaneously in LAS.

The latest NASA scan has identified two web-fe pages that are missing the requred alt="..." elements:

o login.jsp

o createAccountForm.jsp

Note that these elements cannot have empty values.

The front end should have several places that reveal the health of specific nodes in the federation. This will require communication with the dashboard services.

This is a nice to have... I think it would be great to save the last 5 (n)
searches the user performs, so they can quickly get back to results that they
had gone through the process of sifting and navigating to get.

This would mean an additional user interface element (drop down / selection
list).