esgf / esgf-installer Goto Github PK
View Code? Open in Web Editor NEWESGF P2P Node Installer
Home Page: https://esgf.llnl.gov/
License: Other
ESGF P2P Node Installer
Home Page: https://esgf.llnl.gov/
License: Other
The installer script doesn't put quotes around a passphrase which includes spaces. This causes MyProxy to not start-up.
Password in question: Simple CA Passphrase
File refered to: /etc/myproxy-server.config
Update:
Hi Gavin! I checked again, but the issue still persists. There were no junk characters along with the password, but it was not quoted. I'm using v1.4.2-brooklyn_heights-release-4-g84395ec-master
I'm attempting to set up a pure devel node and I can't seem to connect to esgf-test
I'm building on a RedHat6.4 server all ports and permissions are set to see the outside world.
Installing ESGF Node's Access Logging Filters To: [las]
Filter installation destination dir = /usr/local/tomcat/webapps/las
Filter entry file = esg-access-logging-filter-web.xml
Filter entry pattern =
WARNING: Could not find las's installation dir - Filter Not Applied
Setting Index Peer... to => [esgf-test](endpoint type = p2p)
Installing Public Certificate of Target Peer Node...[esgf-test]
==> ./InstallCert.class is up to date
==> ./InstallCert$SavingTrustManager.class is up to date
/esg/config/tomcat /usr/local/src/esgf/workbench/esg /usr/local/src/esgf/workbench/esg /usr/local/src/esgf/workbench/esg /usr/local/src/esgf/workbench/esg /usr/local/src/esgf/workbench/esg /usr/local/src/esgf/workbench/esg /usr/local/src/esgf/workbench/esg /usr/local/src/esgf/workbench/esg /usr/local/src/esgf/workbench/esg /usr/local/src/esgf/workbench/esg /usr/local/src/esgf/workbench/esg /usr/local/src/esgf/workbench/esg /usr/local/src/esgf/workbench/esg /usr/local/src/esgf/workbench/esg /usr/local/src/esgf/workbench/esg /usr/local/src/esgf/workbench/esg /usr/local/src/esgf/workbench/esg /usr/local/bin
/usr/local/java/bin/java -classpath .:/usr/local/src/esgf/workbench/esg InstallCert esgf-test:443 changeit /esg/config/tomcat/esg-truststore.ts
Loading KeyStore /esg/config/tomcat/esg-truststore.ts...
Opening connection to esgf-test:443...
Exception in thread "main" java.net.UnknownHostException: esgf-test
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618)
at sun.security.ssl.SSLSocketImpl.(SSLSocketImpl.java:407)
at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:88)
at InstallCert.main(InstallCert.java:85)
Sync'ing /esg/config/tomcat/esg-truststore.ts with /usr/local/java/jre/lib/security/jssecacerts ... [OK].
Error: could not import Index Peer's [esgf-test] certificate!
Sorry...
This action did not complete successfully
Please re-run this task until successful before continuing further
Also please review the installation FAQ it may assist you
http://esgf.org/wiki/ESGFNode/FAQ
Script done, file is /tmp/esgf_install.log
Please select the IDP Peer for this node:
-------------------------------------------
*[1] : ESGF-PCMDI-9 -> pcmdi9.llnl.gov
[2] : ESGF-PCMDI -> pcmdi3.llnl.gov
[3] : ESGF-JPL -> esg-gateway.jpl.nasa.gov
[4] : ESGF-ORNL -> esg2-gw.ccs.ornl.gov
[5] : ESGF-BADC -> cmip-gw.badc.rl.ac.uk
[6] : ESGF-DKRZ -> ipcc-ar5.dkrz.de
[7] : ESGF-PNNL -> esg1-gw.pnl.gov
[8] : ESGF-ANL -> dev.esg.anl.gov
[9] : ESGF-PCMDI-TEST3 -> esgf-node3.llnl.gov
-------------------------------------------
[C] : (Manual Entry)
-------------------------------------------
select [1] >
option 9 is no loner a valid node.
The version of git specified by the installer results in a 404 and the installer exits.
[root@esg-pre2 esgf-stuff]# wget http://198.128.245.140/dist/devel/git/git-1.8.4.2.tar.gz
--2013-11-08 11:07:07-- http://198.128.245.140/dist/devel/git/git-1.8.4.2.tar.gz
Connecting to 198.128.245.140:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2013-11-08 11:07:07 ERROR 404: Not Found.
Setting up GIT (dvcs) 1.8.4.3
Don't see GIT distribution dir git-1.8.4.3
Don't see GIT distribution file git-1.8.4.3.tar.gz either
Downloading GIT from http://kernel.org/pub/software/scm/git/git-1.8.4.3.tar.gz
--2014-09-02 14:21:55-- http://kernel.org/pub/software/scm/git/git-1.8.4.3.tar.gz
Resolving kernel.org... 199.204.44.194, 149.20.4.69, 198.145.20.140, ...
Connecting to kernel.org|199.204.44.194|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.kernel.org/pub/software/scm/git/git-1.8.4.3.tar.gz [following]
--2014-09-02 14:21:55-- https://www.kernel.org/pub/software/scm/git/git-1.8.4.3.tar.gz
Resolving www.kernel.org... 199.204.44.194, 149.20.4.69, 198.145.20.140, ...
Connecting to www.kernel.org|199.204.44.194|:443... connected.
ERROR: cannot verify www.kernel.org's certificate, issued by /C=US/ST=California/L=Livermore/O=Lawrence Livermore National Laboratory/OU=Cyber Security Program/CN=ca.llnl.gov/[email protected]': Self-signed certificate encountered. To connect to www.kernel.org insecurely, use
--no-check-certificate'.
ERROR: Could not download GIT:git-1.8.4.3.tar.gz
/usr/local/bin
Sorry...
This action did not complete successfully
Please re-run this task until successful before continuing further
Also please review the installation FAQ it may assist you
http://esgf.org/wiki/ESGFNode/FAQ
Script done, file is /tmp/esgf_install.log
line 826
wget -O ${git_dist_file} ${git_dist_url}
new line 826
wget --no-check-certificate -O ${git_dist_file} ${git_dist_url}
I have written a pre installer and uninstall python script. I'm not sure if you would find them helpful or would like to use them as a base for improvements.
https://github.com/webshootertk/utilitybelt/blob/master/esgf_preinstaller.py
https://github.com/webshootertk/utilitybelt/blob/master/esgf_uninstaller.py
I am trying to install 1.7.2 and, after the UV-CDAT part completed successfully, the installation if fetching afterward. I am trying for a second time, but the installer did not ask me wether I wanted to re-install UV-CDAT: it is doing it all over again, which is very time consuming.
This should usually be "http://pcmdi6.llnl.gov/cacert.pem", which is not at all clear from the help text. It would be simpler to simply make that a default which can be overridden as necessary.
Another problem due to relocation of globus binaries
Fetching Freshest ESG Federation Certificates...
Peforming additional checks (IDP node presumes simpleCA installation and configuration)
Checking for orphaned symlinks... (3 files)
globus-host-ssl.conf
globus-user-ssl.conf
grid-security.conf
/usr/local/bin/esg-node: line 2948: simpleCA_relink: command not found
I have completed the installation with no errors, I removed yum erase java
removing openjava off my machine. esgf-web-fe is now coming up but there seems to be an error with the idp security connection / package.
If you accept the default response for "Replica shard entry for port X is already present. Would you like to install it again?" the script fails. The default is N you need Y.
I just ran
export GIT_SSL_NO_VERIFY=1
./esg-node --install --type all --devel
it finishes with no reported errors.
I can bring up the front end but when I click to login --> create account
Tomcat returns a null pointer...
[root@esgf-dev bin]# ./esg-node --install --type all --devel
Please select the ESGF distribution mirror for this installation (fastest to slowest):
-------------------------------------------
[1] http://dist.ceda.ac.uk/esgf/dist
[2] http://distrib-coffee.ipsl.jussieu.fr/pub/esgf/dist
-------------------------------------------
select [1] > 1
1
node type set to: all
EEEEEEEEEEEEEEEEEEEEEE SSSSSSSSSSSSSSS GGGGGGGGGGGGGFFFFFFFFFFFFFFFFFFFFFF
E::::::::::::::::::::E SS:::::::::::::::S GGG::::::::::::GF::::::::::::::::::::F
E::::::::::::::::::::ES:::::SSSSSS::::::S GG:::::::::::::::GF::::::::::::::::::::F
EE::::::EEEEEEEEE::::ES:::::S SSSSSSS G:::::GGGGGGGG::::GFF::::::FFFFFFFFF::::F
E:::::E EEEEEES:::::S G:::::G GGGGGG F:::::F FFFFFF
E:::::E S:::::S G:::::G F:::::F
E::::::EEEEEEEEEE S::::SSSS G:::::G F::::::FFFFFFFFFF
E:::::::::::::::E SS::::::SSSSS G:::::G GGGGGGGGGG F:::::::::::::::F
E:::::::::::::::E SSS::::::::SS G:::::G G::::::::G F:::::::::::::::F
E::::::EEEEEEEEEE SSSSSS::::S G:::::G GGGGG::::G F::::::FFFFFFFFFF
E:::::E S:::::SG:::::G G::::G F:::::F
E:::::E EEEEEE S:::::S G:::::G G::::G F:::::F
EE::::::EEEEEEEE:::::ESSSSSSS S:::::S G:::::GGGGGGGG::::GFF:::::::FF
E::::::::::::::::::::ES::::::SSSSSS:::::S GG:::::::::::::::GF::::::::FF
E::::::::::::::::::::ES:::::::::::::::SS GGG::::::GGG:::GF::::::::FF
EEEEEEEEEEEEEEEEEEEEEE SSSSSSSSSSSSSSS GGGGGG GGGGFFFFFFFFFFF.org
Checking that you have root privs on esgf-dev.llnl.gov... [OK]
Checking requisites...
WARNING: ./esg-node could not be verified!!
(This file, ./esg-node, may have been tampered with or there is a newer version posted at the distribution server.
Please update this script.)
Do you wish to Update and exit [u], continue anyway [c] or simply exit [x]? [u/c/X]: c
Continuing...
The goal of this script is to automate as many tasks as possible
regarding the installation, maintenance and use of the ESGF
software stack that is know as the "ESGF Node". A software
stack is a collection of tools that work in concert to perform a
particular task or set of tasks that are semantically united. The
software stack is comprised of: Tomcat, Thredds, CDAT & CDMS,
PostgreSQL, MyProxy, and several ESGF.org custom software
applications running on a LINUX (RedHat/CentOS) operating system.
Through the installation process there are different accounts
that are created that facilitate the communication between the
software stack entities. These credentials are internal to the
stack. It is recommended that you use the defaults provided
throughout this installation. The security impact with regards
to the visibility and accessibility of the constituent components
of the stack depends on other factors to be addressed by your
organization.
Please be sure that you have gotten your created an account on
your ESGF IDP Peer.
The primary IDP Peer for ESGF is pcmdi9.llnl.gov
You may register for an account at PCMDI9 at the following URL:
http://pcmdi9.llnl.gov/esgf-web-fe/createAccount
Note: Account creation is prerequisite for publication!
ESGF P2P Node: ESGF P2P Node:
--------- ---------
|Tomcat | |Tomcat |
|-Node Mgr| <================= P2P =================> |-Node Mgr|
|-Thredds | |-Thredds |
|-ORP | |-ORP |
|---------| |---------|
|CDAT/CDMS| |CDAT/CDMS|
|---------| |---------|
|Postgres | |Postgres |
|---------| |---------|
| MyProxy | <===(HTTPS)===> [ESGF Peer Node(s)]* | MyProxy |
|---------| |---------|
| GridFTP | <=============> [End User(s)]* | GridFTP |
>---------< >---------<
| CentOS | | CentOS |
|(Virtual)| |(Virtual)|
| Machine | | Machine |
|---------| |---------|
--------- ---------
(Visit http://esgf.org , http://github.com/ESGF/esgf.github.io/wiki for more information)
EEEEEEEEEEEEEEEEEEEEEE SSSSSSSSSSSSSSS GGGGGGGGGGGGGFFFFFFFFFFFFFFFFFFFFFF
E::::::::::::::::::::E SS:::::::::::::::S GGG::::::::::::GF::::::::::::::::::::F
E::::::::::::::::::::ES:::::SSSSSS::::::S GG:::::::::::::::GF::::::::::::::::::::F
EE::::::EEEEEEEEE::::ES:::::S SSSSSSS G:::::GGGGGGGG::::GFF::::::FFFFFFFFF::::F
E:::::E EEEEEES:::::S G:::::G GGGGGG F:::::F FFFFFF
E:::::E S:::::S G:::::G F:::::F
E::::::EEEEEEEEEE S::::SSSS G:::::G F::::::FFFFFFFFFF
E:::::::::::::::E SS::::::SSSSS G:::::G GGGGGGGGGG F:::::::::::::::F
E:::::::::::::::E SSS::::::::SS G:::::G G::::::::G F:::::::::::::::F
E::::::EEEEEEEEEE SSSSSS::::S G:::::G GGGGG::::G F::::::FFFFFFFFFF
E:::::E S:::::SG:::::G G::::G F:::::F
E:::::E EEEEEE S:::::S G:::::G G::::G F:::::F
EE::::::EEEEEEEE:::::ESSSSSSS S:::::S G:::::GGGGGGGG::::GFF:::::::FF
E::::::::::::::::::::ES::::::SSSSSS:::::S GG:::::::::::::::GF::::::::FF
E::::::::::::::::::::ES:::::::::::::::SS GGG::::::GGG:::GF::::::::FF
EEEEEEEEEEEEEEEEEEEEEE SSSSSSSSSSSSSSS GGGGGG GGGGFFFFFFFFFFF.org
-ESGF.org
----------------------------------------------------------
Please make sure you have installed ALL the prerequisites
Please read the prerequisite list here:
https://github.com/ESGF/esgf.github.io/wiki/ESGFNode%7CFAQ#what-do-i-need-on-my-system-before-i-install
Before continuing make sure they are ALL present on this system!!!!!!
If you intend on publishing from this node please check the publishing network prerequisites
http://github.com/ESGF/esgf.github.io/wiki/ESGFNode%7CFAQ#prerequisites
----------------------------------------------------------
(Installing DEVELOPMENT tree...)
Are you ready to begin the installation? [Y/n]
Using IP: 128.115.184.94
(data node type selected)
(index node type selected)
(idp node type selected)
(compute node type selected)
Welcome to the ESGF Node installation program! :-)
esgf_host = [esgf-dev.llnl.gov]
esg_root_id = [llnl]
node_short_name = [esgf_devel_node]
node_long_name = [devel node only no production publications]
node_namespace = [gov.llnl]
node_peer_group = [esgf-test]
esgf_default_peer = [vesgint-idx.ipsl.jussieu.fr]
esgf_index_peer = [vesgint-idx.ipsl.jussieu.fr]
mail_admin_address = [[email protected]]
db connection string = [postgresql://dbsuper@localhost:5432/esgcet] [external = no]
publisher_db_user = [esgcet]
LOADING installer for security...
==> ./esg-security is up to date
Checking for esgf security (lib) [2.5.0-phoenix-1.7.2-devel]
The detected version of esgf-security [2.5.0] is not between [2.5.0-phoenix-1.7.2-devel] and []
Configuring Postgres... for ESGF Security
pg_ctl: server is running (PID: 29656)
/usr/local/pgsql/bin/postgres "-D" "/usr/local/pgsql/data"
psql: FATAL: password authentication failed for user "dbsuper"
psql: FATAL: password authentication failed for user "dbsuper"
Creating ESGF database: [esgcet]
==> esgf-user-migrate is up to date
LOADING installer for idp...
==> ./esg-idp is up to date
Checking for idp services 2.4.5-phoenix-1.7.2-devel
Sorry, the detected version of esgf-idp [2.4.5] is older than required minimum version [2.4.5-phoenix-1.7.2-devel]
Setting up The ESGF Idp Services
Detected an existing idp services installation...
Do you want to continue with idp services installation and setup? [Y/n]
Do you want to make a back up of the existing distribution?? [Y/n]
Creating a backup archive of this web application /usr/local/tomcat/webapps/esgf-idp
Backup - Creating a backup archive of /usr/local/apache-tomcat-7.0.55/webapps/esgf-idp
Created backup: /esg/backups/esgf-idp.2014_09_04_101337.tgz
==> /usr/local/src/esgf/workbench/esg/esgf-idp.war is up to date
Tomcat (jsvc) process is running...
stop tomcat: /usr/local/tomcat/bin/jsvc -pidfile /var/run/tomcat-jsvc.pid -stop org.apache.catalina.startup.Bootstrap
(please wait)
Moving Previous Installation of esgf-idp...
/usr/local/tomcat/webapps/esgf-idp' ->
/usr/local/tomcat/webapps/esgf-idp.bak'
Expanding war /usr/local/src/esgf/workbench/esg/esgf-idp.war in /usr/local/tomcat/webapps/esgf-idp
sanity checking webapps' web.xml files accordingly...
|--setting ownership of web.xml files... to tomcat.tomcat
|--inspecting web.xml files for proper authorization service assignment...
**
ROOT app in place... [OK]
Cleared tomcat cache...
starting services... (60)
Running startup hooks...
ESGCET (Publisher) Startup Hook: Setting perms... :-)
TDS (THREDDS) Startup Hook: Setting perms... :-)
ORP Startup Hook: Syncing Credential Tokens... Oo.:-)
Security Startup Hook: Setup policy and whitelists... (p) Could not determine location of security jar, exiting...
Web-FE Startup Hook... :-)
Desktop Startup Hook: Setting up config file(s) for desktop... :-)
Search Startup Hook... :-)
Publisher Resources... (https://github.com/ESGF/esgf-publisher-resources.git)
Fetching origin
Already up-to-date.
Fetching federation certificates... [OK]
Fetching federation truststore..... [OK]
ntpd (pid 8824) is running...
pg_ctl: server is running (PID: 29656)
/usr/local/pgsql/bin/postgres "-D" "/usr/local/pgsql/data"
Starting Tomcat (jsvc)...
JAVA_HOME=/usr/local/java /usr/local/tomcat/bin/jsvc -Djava.awt.headless=true -Dcom.sun.enterprise.server.ss.ASQuickStartup=false -Dcatalina.home=/usr/local/tomcat -pidfile /var/run/tomcat-jsvc.pid -cp /usr/local/apache-tomcat-7.0.55/bin/tomcat-juli.jar:/usr/local/apache-tomcat-7.0.55/bin/bootstrap.jar:/usr/local/apache-tomcat-7.0.55/bin/commons-daemon.jar -outfile /usr/local/tomcat/logs/catalina.out -errfile /usr/local/tomcat/logs/catalina.err -user tomcat -Xmx2048m -server -Xms1024m -XX:MaxPermSize=512m -Dsun.security.ssl.allowUnsafeRenegotiation=false org.apache.catalina.startup.Bootstrap
..Tomcat (jsvc) process is running...
check_tomcat_process [OK]
Starting dashboard provider...
Starting Globus Services for Data-Node... (GridFTP server(s)) : [bdm end-user]
Starting Globus services for datanode
GridFTP - Starting server... bdm end-user
---------esgsaml_auth.conf---------
Setting up GridFTP... chroot jail
Creating chroot jail @ /esg/gridftp_root
Finished setting up a chroot dir at /esg/gridftp_root.
You may wish to create data directories or use
mount --bind datadir /esg/gridftp_root/datadir
to link in external directories.
Reading ESGINI=[/esg/config/esgcet/esg.ini] for thredds_dataset_roots to mount...
mounting [/esg/data] into chroot jail [/esg/gridftp_root/] as [esg_dataroot]
mount: special device /esg/data does not exist
*
writing sanitized passwd file to [/esg/gridftp_root/etc/passwd] [OK]
writing sanitized group file to [/esg/gridftp_root/etc/group] [OK]
syncing local certificates into chroot jail... [OK]
configuring publisher to use this GridFTP server... [OK]
gridftp-server process is running on port [2812]...
gridftp-server process is running on port [2811]...
Starting Globus Services for IDP-Node... (MyProxy server)
Starting Globus services for gateway
MyProxy - Starting server...
Starting up MyProxy server... Failed to start MyProxy server!
Could Not Start Globus IDP Node related services (MyProxy)
Starting search services... master slave
Solr process for slave running on port [8983]...
Solr process for master running on port [8984]...
Running Node Services...
jsvc 1204 tomcat 40u IPv6 681118 0t0 TCP *:80 (LISTEN)
jsvc 1204 tomcat 41u IPv6 681125 0t0 TCP *:443 (LISTEN)
postgres 29656 postgres 3u IPv4 673142 0t0 TCP 127.0.0.1:5432 (LISTEN)
globus-gr 30544 root 9u IPv4 675260 0t0 TCP *:2812 (LISTEN)
globus-gr 30552 root 9u IPv4 675338 0t0 TCP *:2811 (LISTEN)
java 30642 root 50u IPv6 675581 0t0 TCP *:8983 (LISTEN)
==> /usr/local/src/esgf/workbench/esg/bash-completion-20060301-1.noarch.rpm is up to date
warning: bash-completion-20060301-1.noarch.rpm: Header V3 DSA/SHA1 Signature, key ID bfbaf0e8: NOKEY
error: Failed dependencies:
bash-completion < 20080705 conflicts with rpmdevtools-7.5-2.el6.noarch
Finished!...
In order to see if this node has been installed properly you may direct your browser to:
http://esgf-dev.llnl.gov/
http://esgf-dev.llnl.gov/esgf-node-manager
http://esgf-dev.llnl.gov/esgf-dashboard
http://esgf-dev.llnl.gov/thredds
http://esgf-dev.llnl.gov/esg-orp
http://esgf-dev.llnl.gov/esgf-desktop
http://esgf-dev.llnl.gov/esgf-web-fe
http://esgf-dev.llnl.gov/las
Your peer group membership -- : [esgf-test]
Your specified "default" peer : [vesgint-idx.ipsl.jussieu.fr]
Your specified "index" peer - : [vesgint-idx.ipsl.jussieu.fr](url = http://vesgint-idx.ipsl.jussieu.fr/)
Your specified "idp" peer --- : [](name =)
[Note: Use UNIX group permissions on /esg/content/thredds/esgcet to enable users to be able to publish thredds catalogs from data therein]
%> chgrp -R /esg/content/thredds
-------------------------------------------------------
Administrators of this node should subscribe to the
[email protected] by sending email to: [email protected]
with the body: subscribe esgf-node-admins
-------------------------------------------------------
v0.0.0-devel
HTTP Status 500 - Request processing failed; nested exception is java.lang.NullPointerException
type Exception report
message Request processing failed; nested exception is java.lang.NullPointerException
description The server encountered an internal error that prevented it from fulfilling this request.
exception
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.NullPointerException
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:948)
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827)
javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812)
javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
org.tuckey.web.filters.urlrewrite.NormalRewrittenUrl.doRewrite(NormalRewrittenUrl.java:195)
org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:159)
org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:141)
org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:90)
org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:417)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:144)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.esgf.security.OpenidCookieFilter.doFilter(OpenidCookieFilter.java:81)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
root cause
java.lang.NullPointerException
org.esgf.accounts.CreateAccountController.formBackingObject(CreateAccountController.java:91)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:601)
org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219)
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
org.springframework.web.method.annotation.ModelFactory.invokeModelAttributeMethods(ModelFactory.java:123)
org.springframework.web.method.annotation.ModelFactory.initModel(ModelFactory.java:97)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:722)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686)
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936)
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827)
javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812)
javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
org.tuckey.web.filters.urlrewrite.NormalRewrittenUrl.doRewrite(NormalRewrittenUrl.java:195)
org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:159)
org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:141)
org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:90)
org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:417)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:144)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.esgf.security.OpenidCookieFilter.doFilter(OpenidCookieFilter.java:81)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
note The full stack trace of the root cause is available in the Apache Tomcat/7.0.55 logs.
Apache Tomcat/7.0.55
The installer (devel, 9a0c6de ) tries to pull down tomcat v7.0.42 which doesn't exist on apache.mirrors.pair.com. The only version that's available there is v7.0.47. The script should be updated.
The installer died on us because the tomcat current version needs to be updated.
in the file esg-node on (around line) 243:
tomcat_version=${tomcat_version:-"7.0.54"} ; tomcat_min_version=${tomcat_min_version:-"7.0.53"}
when the link is build to download 7.0.54 it 404's as the current version is 7.0.55, I manually updated my file and it worked again.
tomcat_version=${tomcat_version:-"7.0.55"} ; tomcat_min_version=${tomcat_min_version:-"7.0.53"}
Latest diagnosis from Nicolas:
On CentOS5 with default ESGF 1.7 + default UV-CDAT 1.5
Specifying uv-cdat's lxml works:
gcc -pthread -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/uvcdat/1.5.0/Externals/include -I/usr/local/uvcdat/1.5.0/Externals/include -fPIC -D_REENTRANT=1 -I/usr/local/uvcdat/1.5.0/Externals/include/libxml2 -I/usr/include -I/usr/local/uvcdat/1.5.0/include/python2.7 -c libxml2-py.c -o build/temp.linux-x86_64-2.7/libxml2-py.o
where the default gcc command executed by the installer fails:
gcc -pthread -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/uvcdat/1.5.0/Externals/include -I/usr/local/uvcdat/1.5.0/Externals/include -fPIC -D_REENTRANT=1 -I/usr/include/libxml2 -I/usr/include -I/usr/local/uvcdat/1.5.0/include/python2.7 -c libxml2-py.c -o build/temp.linux-x86_64-2.7/libxml2-py.o
So by default, without the flag "-DCDAT_USE_SYSTEM_LIBXLM2=ON": UVCDAT still uses the system libxml2 instead of its own.
Where to tell the compiler to use UV-CDAT libxml2 instead of the system one?
Martin suggested that drslib should be part of ESGF installation.
To do:
The uvcdat part of the install is NOT idempotent;
If you need to redo the uvcdat install when it is present on the system (it's not
absent, but partially installed/corrupted), the uvcdat related flags
(path, library path etc) need to be unset before you continue. Failure to do so results in the 'Sorry.... ' error and aborted installation of uvcdat. /etc/esg.env needs to be manually cleaned up and sourced before running the installer again.
On a machine with a different outward-facing name than internal hostname, the certificate generated by esg-node --generate-ssl-key-and-csr
will have a filename based on the outward-facing name, but esg-node --install-keypair
will default to a filename based on the internal hostname, and this cannot be overridden. The only way to make it work currently is to symlink the first filename to the second. It would be nice to have this as something that can be specified, perhaps defaulting to the autogenerated name.
after an update esgf.properties is will reset my mail server to:
mail.smtp.host=smtp.llnl.gov
Is there a way to leave this settings alone? we need it to be:
mail.smtp.host=nospam.llnl.gov
This issue has occured on data/compute and idp/index/data/compute nodes from scratch installations. Rerun the installation again doest not solve the problem. The only woraround I found is to remove /usr/local/ferret and rerun the installation.
*******************************
Setting up LAS Product Server...
*******************************
Getting LAS...
Don't see LAS tar file las-esgf-v8.1.tar.gz Downloading LAS from las-esgf-v8.1.tar.gz -to-> /usr/local/src/esgf/workbench/esg/ferret/8.1/las-esgf-v8.1.tar.gz
wget -O 'las-esgf-v8.1.tar.gz' 'ftp://ftp.pmel.noaa.gov/pub/las/las-esgf-v8.1.tar.gz'
/usr/local/bin/esg-product-server: line 426: /usr/bin/wget: Argument list too long
ERROR: Could not download LAS:las-esgf-v8.1.tar.gz
This would appear to be a low-level bash bug (because the command being executed definitely only has 3 arguments).
One possible work-around would be to try to increase ulimit -s. it is 10240 by default. On Linux, the maximum amount of space for command arguments is 1/4th of the amount of available stack space.
Updating a data node and the installer doesn't update esgcet because of a github connectivity problem. Unfortunately this is not apparent until later. Somehow, it reverted v 2.12.1 which is older than the previous version altogether.
Additionally with this datanode, the hessian url in esg.ini got changed (can this even happen?) to something in IPSL.
if there's a problem with curl, cog won't install. In this case the libcurl that the installer uses is missing https. Not using esg.env you can git clone cog with the default libcurl from your os which has https. It would be helpful if a problem with the cog installation is detected so we can attend to it.
PAM only looks up the first result for the password and when the first entry in the database happens to be from a non-local site, the password field is empty and PAM regards that as a successful auth, no matter what password is presented.
The following script is used to setup GO support on the server:
http://www.mcs.anl.gov/~neillm/esg/install_mkproxy.sh
It should probably be run as part of the standard installation process.
Installer tries to fetch version 7.0.47. It's no longer available on the apache site and installation fails.
The available version is 7.0.52.
http://apache.mirrors.pair.com/tomcat/tomcat-7/v7.0.52/bin/apache-tomcat-7.0.52.tar.gz
When running the install with the --force option, there is no option to skip reinstalling CDAT all over again. It simply starts building all over again.
Fix sought: option to skip, just like the other components.
If installing a node of type "data" (i.e. without compute) TDS fails to start.
The filter AuthenticationByIPFilter is installed in TDS web.xml. This filter
looks for las_servers.xml and las_servers_static.xml. These files have not
been created in a non-compute installation. The filter crashes preventing TDS
from serving any pages.
From web.xml:
<filter>
<filter-name>ipFilter</filter-name>
<filter-class>esg.orp.app.AuthenticationByIPFilter</filter-class>
<init-param>
<param-name>ip_whitelist</param-name>
<param-value>/esg/content/las/conf/server/las_servers.xml,
/esg/content/las/conf/server/las_servers_static.xml</param-value>
</init-param>
</filter>
<filter-mapping>
<filter-name>ipFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
Node Version:
[root@esgf-data2 ~]# esg-node --version
Version: v1.5.0-brower_park-release-3-g9a0c6de-master
Release: brower_park
Earth Systems Grid Federation (http://esgf.org)
ESGF Node Installation Script
The following list of things were provided by the installer in the past, but are not being supported by the current installer. As a result, extra work has to be put in preparing the node for publication when it could be easily handled by the installer.
For ESGF 1.9 or 2.0?
esg-node --test makes lots of globus function calls refering to old globus binary locations, and fails.
Ex:
GridFTP - BDM Test... [bdm]/usr/local/bin/esg-globus: line 909: /usr/local/globus/bin/globus-url-copy: No such file or directory
[FAIL]
/usr/local/bin/esg-node: line 2730: /usr/local/globus/bin/myproxy-logon: No such file or directory
ERROR: MyProxy not setup properly. Unable to execute command.
http://www.us.apache.org/dist/tomcat/tomcat-7/v7.0.54/bin/apache-tomcat-7.0.54.tar.gz
now
http://www.us.apache.org/dist/tomcat/tomcat-7/v7.0.55/bin/apache-tomcat-7.0.55.tar.gz
line 244
tomcat_version=${tomcat_version:-"7.0.54"} ; tomcat_min_version=${tomcat_min_version:-"7.0.53"}
Some directories' permissions are changed to tomcat or root after an update is performed. Directory paths like /esg/config/esgcet and /esg/content/thredds are always affected by this. Therefore, the installer should not change the permissions that were there originally. This will safe the extra time invested in fixing these directories after running the installer and it will keep systems consistent.
ESGF installs postgresql v8.4.7 but v8.4 support has been stopped in july 2014
The script first installs gridftp and tries to install myproxy. When an existing globus installation is detected it asks whether you want to install globus again, defaulting to "N". If you follow the default myproxy is not installed.
We recommend at the minimum the default should be to install globus. Ideally the installer would only query the user to install globus once and would know whether to install gridftp and/or myproxy.
The installer (devel, 9a0c6de ) while installing CDAT, appears to hang. It is actually waiting for a keyboard input. There's no indication of this on stdout. Entering 'y' allows the installation to proceed. Fix would be a message indicating the requirement of user response.
When not installing all components the directory /esg/content/thredds is not created and the script fails. The solution is to create /esg/content/thredds and re-run the script.
When not installing the compute component thredds will not start because it is looking for las_servers.xml. The solution is to comment out the ipFilter declaration in thredds' web.xml.
We recommend the ipFilter declaration in web.xml should only be included in the compute configuration.
6.9 is higher than 6.842 but the installer still says:
Starting Product Services installation (Ferret/LAS)...
Checking for Ferret v6.842 of [/usr/local/ferret/bin/ferret]
The detected version of ferret [6.9] is not between [6.842] and []
Since ESGF relies on a system curl. The uvcdat one has to be removed. Charles says CMakeLists.txt has to be modified.
If you know for sure you have curl, simply patch uvcdat install to not
build curl, look for
include(curl)
in CMakeLists.txt at the top of uvcdat repo and comment it out.
I can send a patch if you don't find it.
C.
Since it doesn't install itself, it fails to fetch a newer esg-node if asked to update by selecting 'u'. (fails with a command not found /usr/local/bin/esg-bootstrap)
When trying to set up a data node on a clean virtual machine (CentOS) I get the following error when the script attempts to build OpenSSL:
Failed at line 315 if statement…
installing man1/smime.1
smime.pod around line 272: Expected text after =item, not a number
smime.pod around line 276: Expected text after =item, not a number
smime.pod around line 280: Expected text after =item, not a number
smime.pod around line 285: Expected text after =item, not a number
smime.pod around line 289: Expected text after =item, not a number
POD document had syntax errors at /bin/pod2man line 69.
I have also attempted install on Ubuntu 15.04, which yields the same error. Both systems have latest available OpenSSL install present (1.0.1e).
Replacing the build statement with 'true' in within the if block allows the install to bypass this step and run all the way through.
I tried to use the esgf repository manager (http://esgf.org/artifactory ) for pulling the esg-orp and esgf-security jars from it with maven but if fails because the pom files are unavailable.
Currently the installer overrides the Tomcat server.xml with a default Tomcat alias of "my_esgf_node". It would be good to allow sites to specify their own value for the keystore "alias" and "password" in the esgf.properties file, and have the installer use those values.
Setting up CDAT - (Python + CDMS)... 1.5.0
Fetching the cdat project from GIT Repo...
Cloning into 'uvcdat'...
fatal: unable to connect to github.com:
github.com[0: 192.30.252.131]: errno=Connection refused
Apparently was not able to fetch from GIT repo using git protocol... trying http protocol...
Cloning into 'uvcdat'...
fatal: unable to access 'https://github.com/UV-CDAT/uvcdat.git/': SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
Could not fetch from cdat's repo (with git nor http protocol)
Sorry...
This action did not complete successfully
Please re-run this task until successful before continuing further
Also please review the installation FAQ it may assist you
http://esgf.org/wiki/ESGFNode/FAQ
export GIT_SSL_NO_VERIFY=1
This is just a test commit
If the password for the user dbsuper is not entered correctly the second time (confirmation), it results in an immediate exit with the 'Sorry' message.
The installer tries to download apache-tomcat-7.0.55 from http://www.us.apache.org/dist/tomcat/tomcat-7. The latest 7.X version of tomcat is 7.0.56, and that seems to be the only one downloadabe from us.apache.org. If you really need 7.0.55, it would have to be downloaded from http://archive.apache.org/dist/tomcat/tomcat-7 instead.
I've been trying to install an index node on my computer, for testing purposes. I already had a data node installed, but simply stopping the node and rerunning esg-node --install --type index on top of that didn't seem to give the desired result, so I wound up doing a full uninstall first (had to umount /esg/gridftp_root/esg_dataroot before I could delete /esg).
I decided to go ahead and do esg-node --install --type all, so I should get a full data + index + IDP + compute node. I had to restart the installation a few times, because my internet connection seemed to be a bit flaky, but the install seemed to finish successfully. Node start seemed to work fine.
However, when I went to check the web front end pages, only aquarius.llnl.gov/esgf-node manager , /thredds, and /las seem to work:
https://aquarius.llnl.gov/esgf-node manager/:
Works!
https://aquarius.llnl.gov/esg-orp/:
HTTP Status 500 - java.lang.ClassNotFoundException: org.apache.jsp.index_jsp
https://aquarius.llnl.gov/esg-orp/home.htm
Works! Gives a Data Access Login prompt, but I can't get it to recognize my OpenID
https://aquarius.llnl.gov/esgf-desktop/
HTTP Status 500 - java.lang.ClassNotFoundException: org.apache.jsp.index_jsp
https://aquarius.llnl.gov/esgf-dashboard
HTTP Status 404 -
http://aquarius.llnl.gov/thredds/
works! directory with Earth System Root catalog
http://aquarius.llnl.gov/esgf-web-fe/
HTTP Status 500 - java.lang.ClassNotFoundException: org.apache.jsp.WEB_002dINF.views.welcome_jsp
http://aquarius.llnl.gov/esgf-idp
HTTP Status 404 - /esgf-idp
http://aquarius.llnl.gov/las
Works! Forwards to http://aquarius.llnl.gov/las/UI.vm
Looks to me like the compute node may be working, the data node works partially, but the index and IDP nodes aren’t working.
Any suggestions for how to diagnose this further?
Patrik
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.