Error in starting server, when passwords are encrypted with secure vault

[2017-10-07 11:29:43,754] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Starting WSO2 Carbon...
[2017-10-07 11:29:43,756] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Operating System : Linux 3.10.0-514.21.1.el7.x86_64, amd64
[2017-10-07 11:29:43,756] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Java Home : /home/o2_adm/java/jre
[2017-10-07 11:29:43,756] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Java Version : 1.8.0_121
[2017-10-07 11:29:43,757] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Java VM : Java HotSpot(TM) 64-Bit Server VM 25.121-b13,Oracle Corporation
[2017-10-07 11:29:43,757] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Carbon Home : /home/o2_adm/wso2am-analytics-2.1.0
[2017-10-07 11:29:43,758] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Java Temp Dir : /home/o2_adm/wso2am-analytics-2.1.0/tmp
[2017-10-07 11:29:43,759] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - User : 1001580000, en-US, Etc/UTC
[2017-10-07 11:29:44,380] INFO {org.wso2.carbon.event.output.adapter.kafka.internal.ds.KafkaEventAdapterServiceDS} - Successfully deployed the Kafka output event adaptor service
[2017-10-07 11:29:44,563] ERROR {org.wso2.carbon.event.processor.manager.core.internal.EventHandler} - Unable to start TCPEventServer for EventReceiver started on port 11225
[2017-10-07 11:29:44,824] INFO {org.wso2.carbon.event.template.manager.core.internal.ds.TemplateDeployerServiceTrackerDS} - Successfully deployed the execution manager tracker service
[2017-10-07 11:29:45,876] FATAL {org.wso2.carbon.tomcat.internal.TomcatBundleActivator} - Error while starting server NOT_FOUND_ERR: An attempt is made to reference a node in a context where it does not exist.
org.w3c.dom.DOMException: NOT_FOUND_ERR: An attempt is made to reference a node in a context where it does not exist.
at org.apache.xerces.dom.AttributeMap.internalRemoveNamedItem(Unknown Source)
at org.apache.xerces.dom.AttributeMap.removeNamedItem(Unknown Source)
at org.wso2.carbon.tomcat.internal.ServerManager.init(ServerManager.java:106)
at org.wso2.carbon.tomcat.internal.TomcatBundleActivator.start(TomcatBundleActivator.java:45)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl$1.run(BundleContextImpl.java:711)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.startActivator(BundleContextImpl.java:702)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.start(BundleContextImpl.java:683)
at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(BundleHost.java:381)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.resume(AbstractBundle.java:390)
at org.eclipse.osgi.framework.internal.core.Framework.resumeBundle(Framework.java:1176)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:559)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:544)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.incFWSL(StartLevelManager.java:457)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(StartLevelManager.java:243)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.dispatchEvent(StartLevelManager.java:438)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.dispatchEvent(StartLevelManager.java:1)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:340)
[2017-10-07 11:29:47,466] INFO {org.wso2.carbon.databridge.receiver.binary.internal.BinaryDataReceiver} - Started Binary SSL Transport on port : 9712
[2017-10-07 11:29:47,467] INFO {org.wso2.carbon.databridge.receiver.binary.internal.BinaryDataReceiver} - Started Binary TCP Transport on port : 9612

Related Issues:
https://wso2.org/jira/browse/CARBON-16104

Thanks,
Youcef HILEM

Hi,

Is the value of PolicyDeployer:ServiceURL for pattern-3/dockerfiles/apim-pubstore-tm-2/carbon/repository/conf/api-manager.xml should be https://localhost:${mgt.transport.https.port}${carbon.context}services/ and not https://pubstore-tm-km-1:9443${carbon.context}services/ ?

Example : the value of PolicyDeployer:ServiceURL for pattern-3/dockerfiles/apim-pubstore-tm-1/carbon/repository/conf/api-manager.xml is https://localhost:${mgt.transport.https.port}${carbon.context}services/

In addition, the hostname pubstore-tm-km-1 is not defined for pattern-3.

Thanks,
Youcef HILEM

Hi,

This is the exception thrown when we access api stats

[2017-09-19 12:20:56,955] ERROR - APIUsageStatisticsRdbmsClientImpl Error occurred while querying from JDBC database Table 'o2_apim_stat_db.API_REQUEST_SUMMARY' doesn't exist
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'o2_apim_stat_db.API_REQUEST_SUMMARY' doesn't exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
at com.mysql.jdbc.Util.getInstance(Util.java:360)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:978)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2526)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484)
at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1446)
at org.wso2.carbon.apimgt.usage.client.impl.APIUsageStatisticsRdbmsClientImpl.queryFirstAccess(APIUsageStatisticsRdbmsClientImpl.java:2210)
at org.wso2.carbon.apimgt.usage.client.impl.APIUsageStatisticsRdbmsClientImpl.getFirstAccessTime(APIUsageStatisticsRdbmsClientImpl.java:2155)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
at org.mozilla.javascript.NativeJavaMethod.call(NativeJavaMethod.java:225)
at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
at org.jaggeryjs.rhino.publisher.modules.statistics.c1._c_getFirstAccessTime_13(/publisher/modules/statistics/usage.jag:398)
at org.jaggeryjs.rhino.publisher.modules.statistics.c1.call(/publisher/modules/statistics/usage.jag)
at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430)
at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269)
at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97)
at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42)
at org.jaggeryjs.rhino.publisher.modules.statistics.c0._c_anonymous_16(/publisher/modules/statistics/module.jag:39)
at org.jaggeryjs.rhino.publisher.modules.statistics.c0.call(/publisher/modules/statistics/module.jag)
at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
at org.jaggeryjs.rhino.publisher.site.blocks.stats.api_usage_user.ajax.c0._c_anonymous_1(/publisher/site/blocks/stats/api-usage-user/ajax/stats.jag:68)
at org.jaggeryjs.rhino.publisher.site.blocks.stats.api_usage_user.ajax.c0.call(/publisher/site/blocks/stats/api-usage-user/ajax/stats.jag)
at org.mozilla.javascript.optimizer.OptRuntime.call0(OptRuntime.java:23)
at org.jaggeryjs.rhino.publisher.site.blocks.stats.api_usage_user.ajax.c0._c_script_0(/publisher/site/blocks/stats/api-usage-user/ajax/stats.jag:4)
at org.jaggeryjs.rhino.publisher.site.blocks.stats.api_usage_user.ajax.c0.call(/publisher/site/blocks/stats/api-usage-user/ajax/stats.jag)
at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394)
at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3091)
at org.jaggeryjs.rhino.publisher.site.blocks.stats.api_usage_user.ajax.c0.call(/publisher/site/blocks/stats/api-usage-user/ajax/stats.jag)
at org.jaggeryjs.rhino.publisher.site.blocks.stats.api_usage_user.ajax.c0.exec(/publisher/site/blocks/stats/api-usage-user/ajax/stats.jag)
at org.jaggeryjs.scriptengine.engine.RhinoEngine.execScript(RhinoEngine.java:567)
at org.jaggeryjs.scriptengine.engine.RhinoEngine.exec(RhinoEngine.java:273)
at org.jaggeryjs.jaggery.core.manager.WebAppManager.exec(WebAppManager.java:588)
at org.jaggeryjs.jaggery.core.manager.WebAppManager.execute(WebAppManager.java:508)
at org.jaggeryjs.jaggery.core.JaggeryServlet.doPost(JaggeryServlet.java:29)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:747)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:485)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:377)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:337)
at org.jaggeryjs.jaggery.core.JaggeryFilter.doFilter(JaggeryFilter.java:21)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.wso2.carbon.ui.filters.cache.ContentTypeBasedCachePreventionFilter.doFilter(ContentTypeBasedCachePreventionFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:120)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
at org.wso2.carbon.event.receiver.core.internal.tenantmgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:48)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:958)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1756)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1715)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
[2017-09-19 12:20:56,964] ERROR - usage:jag org.wso2.carbon.apimgt.usage.client.exception.APIMgtUsageQueryServiceClientException: Error occurred while querying from JDBC databaseTable 'o2_apim_stat_db.API_REQUEST_SUMMARY' doesn't exist

Thanks,
Youcef HILEM

Description:
I'm trying to deploy pattern 1 on kubernetes, but the apim container is crashing with this error.
/home/wso2user/scripts/init_carbon.sh: /home/wso2user/wso2am-analytics-2.1.0/bin/wso2server.sh: /bin/sh^M: bad interpreter: No such file or directory

Suggested Labels:
dockerfile script line-ending

Suggested Assignees:

Affected Product Version:
docker.wso2.com/wso2am-kubernetes 2.1.0
docker.wso2.com/wso2am-analytics-kubernetes 2.1.0

CoreOS, Kubernetes 1.79 (also reproduced in minikube)

Steps to reproduce:
in pattern 1, run deploy-kubernetes.sh, look at log for pod

Related Issues:
I believe this is due to a trailing ^M (windows line ending) in one of the scripts used to build the Docker image.

Hi,

The exception "org.apache.thrift.transport.TTransportException: Could not bind to port 10500" is thrown when HostName and MgtHostName are not set to lacalhost in carbon.xml.

Exemple :

  <!--<HostName>wso2apim-all.wso2.svc</HostName>-->
  <HostName>localhost</HostName>


    <!--<MgtHostName>wso2apim-all.wso2.svc</MgtHostName>-->
<MgtHostName>localhost</MgtHostName>

Thanks
Youcef HILEM

Description:
There can be situations where you won't be able to access the following endpoints (routes) when deployed in minishift or openshift.

https://wso2apim
https://wso2apim-analytics
https://wso2apim-gw

This could happen if there aren't any routers in the deployment.

Suggested Labels:

Suggested Assignees:

Affected Product Version:

OS, DB, other environment details and versions:

Steps to reproduce:

Related Issues:

Description:
Helm charts should be created for the deployment of APIM with analytics.

Suggested Labels:
Type/NewFeature

Affected Product Version:
WSO2 API Manager 2.2.x

Hi,

For pattern-2 (https://github.com/wso2/kubernetes-apim/tree/2.1.0/pattern-2), these exception is continously thrown for POD apim-gw-worker :

[2017-09-16 07:15:56,676] INFO - ChannelOpenHandler Connecting to: carbon
[2017-09-16 07:15:56,788] INFO - AndesChannel Channel created (ID: X.X.X.X:47620)
[2017-09-16 07:15:56,965] WARN - JMSUtils Cannot locate destination : throttleData
[2017-09-16 07:15:57,015] INFO - RegistryEventingServiceComponent Successfully Initialized Eventing on Registry
[2017-09-16 07:15:57,164] INFO - QueueDeclareHandler Queue tmp_X.X.X.X_47620_1 bound to default exchange(<>)
[2017-09-16 07:15:57,164] INFO - QueueDeclareHandler Queue tmp_X.X.X.X_47620_1 declared successfully
[2017-09-16 07:15:57,228] INFO - QueueBindHandler Binding queue tmp_X.X.X.X_47620_1 to exchange TopicExchange[amq.topic] with routing key throttleData
[2017-09-16 07:15:57,323] INFO - SubscriptionEngine Local subscription ADDED [throttleData]ID=0@NODEwso2apim-worker-1-2o4mh/X.X.X.X/T=1505546157288/D=false/X=true/O=clientid/E=amq.topic/ET=org.wso2.andes.server.exchange.TopicExchange$1@5f40412b/EUD=0/S=true
[2017-09-16 07:15:57,419] ERROR - SourceHandler I/O error: Received fatal alert: unknown_ca
javax.net.ssl.SSLException: Received fatal alert: unknown_ca
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:245)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:280)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:410)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:119)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:159)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:338)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:586)
at java.lang.Thread.run(Thread.java:745)
[2017-09-16 07:15:57,542] INFO - JMXServerManager JMX Service URL : service:jmx:rmi://localhost:11111/jndi/rmi://localhost:9999/jmxrmi
[2017-09-16 07:15:57,543] INFO - StartupFinalizerServiceComponent Server : WSO2 API Manager-2.1.0
[2017-09-16 07:15:57,544] INFO - StartupFinalizerServiceComponent WSO2 Carbon started in 111 sec
[2017-09-16 07:15:57,797] INFO - JMSListener Started to listen on destination : throttleData of type topic for listener Siddhi-JMS-Consumer#throttleData
[2017-09-16 07:15:58,412] INFO - CarbonUIServiceComponent Mgt Console URL : https://X.X.X.X:9443/carbon/
[2017-09-16 07:15:58,414] INFO - CarbonUIServiceComponent API Publisher Default Context : https://X.X.X.X:9443/publisher
[2017-09-16 07:15:58,414] INFO - CarbonUIServiceComponent API Store Default Context : https://X.X.X.X:9443/store
[2017-09-16 07:15:58,518] ERROR - SourceHandler I/O error: Received fatal alert: unknown_ca
javax.net.ssl.SSLException: Received fatal alert: unknown_ca
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:245)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:280)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:410)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:119)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:159)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:338)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:586)
at java.lang.Thread.run(Thread.java:745)
[2017-09-16 07:15:59,075] ERROR - SourceHandler I/O error: Received fatal alert: unknown_ca
javax.net.ssl.SSLException: Received fatal alert: unknown_ca
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:245)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:280)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:410)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:119)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:159)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:338)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:586)
at java.lang.Thread.run(Thread.java:745)
[2017-09-16 07:15:59,421] ERROR - SourceHandler I/O error: Received fatal alert: unknown_ca
javax.net.ssl.SSLException: Received fatal alert: unknown_ca
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)
....
....

To facilitate the analysis, I stopped all other PODs.
As a result, the exchanges that cause this exception are within the only POD that remains: apim-gw-worker

One solution that satisfies us is not to use SSL when exchanges are intra-POD.
Where to apply changes And so quickly fix this problem ?

Apparently, this happens when a pub / subscribe channel is created between the client (gw) and the broker node. But this connection is not ssl :

Regards,
Youcef HILEM

Description:
Create Kubernetes resources for WSO2 API Manager version 2.2.0 deployment pattern 1 [1].

[1]: Deployment pattern 1

Suggested Labels:
Type/Task

Suggested Assignees:
chirangaalwis

Affected Product Version:
WSO2 API Manager 2.2.x

Description:

We need to update as follows for APIM and Analytics.

APIM:

  resources:
    requests:
      memory: "2Gi"
      cpu: "2000m"
    limits:
      memory: "3Gi"
      cpu: "3000m"

Analytics:

 resources:
    requests:
      memory: "6Gi"
      cpu: "4000m"
    limits:
      memory: "9Gi"
      cpu: "6000m"

Sample:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: wso2apim-manager-worker
  labels:
    app: wso2apim
    node-m: manager-worker
    pattern: wso2apim-pattern-1
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wso2apim
        node-m: manager-worker
        pattern: wso2apim-pattern-1
    spec:
      containers:
      -
        image: docker.wso2.com/wso2am-kubernetes:2.1.0
        name: wso2apim
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            memory: "2Gi"
            cpu: "2000m"
          limits:
            memory: "3Gi"
            cpu: "3000m"
        ports:
        -
          containerPort: 8243
          protocol: "TCP"
        -
          containerPort: 9763
          protocol: "TCP"
        -
          containerPort: 9443
          protocol: "TCP"
        -
          containerPort: 5672
          protocol: "TCP"
        -
          containerPort: 9711
          protocol: "TCP"
        -
          containerPort: 9611
          protocol: "TCP"
        -
          containerPort: 7711
          protocol: "TCP"
        -
          containerPort: 7611
          protocol: "TCP"

        volumeMounts:
        -
          name: apim-mgt-storage-volume
          mountPath: "/home/wso2user/wso2am-2.1.0/repository/deployment/server"
        - name: apim-manager-worker-bin
          mountPath: "/home/wso2user/wso2am-2.1.0-conf/bin/"
        - name: apim-manager-worker-conf
          mountPath: "/home/wso2user/wso2am-2.1.0-conf/conf/"
        - name: apim-manager-worker-identity
          mountPath: "/home/wso2user/wso2am-2.1.0-conf/conf-identity/"
        - name: apim-manager-worker-axis2
          mountPath: "/home/wso2user/wso2am-2.1.0-conf/conf-axis2/"
        - name: apim-manager-worker-datasources
          mountPath: "/home/wso2user/wso2am-2.1.0-conf/conf-datasources/"
        - name: apim-manager-worker-tomcat
          mountPath: "/home/wso2user/wso2am-2.1.0-conf/conf-tomcat/"
        livenessProbe:
          tcpSocket:
            port: 9443
          initialDelaySeconds: 300
          periodSeconds: 20
      serviceAccountName: "wso2svcacct"
      volumes:
      - name: apim-mgt-storage-volume
        persistentVolumeClaim:
          claimName: apim-mgt-volume-claim
      - name: apim-manager-worker-bin
        configMap:
          name: apim-manager-worker-bin
      - name: apim-manager-worker-conf
        configMap:
          name: apim-manager-worker-conf
      - name: apim-manager-worker-identity
        configMap:
          name: apim-manager-worker-identity
      - name: apim-manager-worker-axis2
        configMap:
          name: apim-manager-worker-axis2
      - name: apim-manager-worker-datasources
        configMap:
          name: apim-manager-worker-datasources
      - name: apim-manager-worker-tomcat
        configMap:
          name: apim-manager-worker-tomcat

Suggested Labels:

Suggested Assignees:

Affected Product Version:

OS, DB, other environment details and versions:

Steps to reproduce:

Related Issues:

Description:
In release version v2.1.0.2 Following pods does not start succesfully

wso2apim-manager-worker-455294819-tgx7h    0/1       CrashLoopBackOff   4          2m
wso2apim-worker-458565719-6h1vk                  0/1       CrashLoopBackOff   4          1m

and the log message is

user: wso2user
user home: /home/wso2user
carbon server: wso2am-2.1.0
/bin/chown: changing ownership of '/home/wso2user/wso2am-2.1.0/repository/deployment/server/': Operation not permitted

Cloud Provider : OpenStack
kubectl version : v1.8.2
More info : APIM-kubernetes Pattern-1 deployment errors - dev group mail thread.

Suggested Labels:
Type/Bug
Priority/High

Affected Product Version:
v2.1.0.2

Steps to reproduce:
Follow the steps provided in the readme file.

Description:
Observed a warning as below while build the docker images(apim, analytics, rdbms) for apim-kubernetes deployment. Can we improve the docker file[1] with run install for "apt-utils" if no impacts?

Suggested Labels:
apim-kubernetes

Affected Product Version:
apim-kubernetes-v2.1.0.1

OS, DB, other environment details and versions:

Steps to reproduce:
[1] [https://github.com/wso2/kubernetes-apim/blob/v2.1.0.2/base/analytics/Dockerfile#L41]

Description:
Create Kubernetes resources for WSO2 API Manager version 2.5.0 deployment pattern 2 [1].

[1]: Deployment pattern 2

Suggested Labels:
Type/Task

Suggested Assignees:
chirangaalwis

Affected Product Version:
WSO2 API Manager v2.5.x

Description:

When listing Roles in the admin console w/ LDAP Primary User Store, the UI reports the following error

Error while listing roles. Error is : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=DEFAULT' at line 1

Logs:

[2018-04-06 23:42:24,278] ERROR - JDBCUserStoreManager Error while retrieving roles from Internal JDBC role store
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=100' at line 1
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
	at com.mysql.jdbc.Util.getInstance(Util.java:381)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1031)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:957)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3376)
	<remaining stack trace omitted>
[2018-04-06 23:42:24,287] ERROR - DatabaseUtil Database error. Could not close statement. Continuing with others. - You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=DEFAULT' at line 1
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=DEFAULT' at line 1
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
	at com.mysql.jdbc.Util.getInstance(Util.java:381)
	<remaining stack trace omitted>

Steps to reproduce:

Using tag 2.1.0-2 w/ images from docker.wso2.com.

• Installed using Openshift pattern-1 deployment.
• Changed the Primary User Store to use ReadOnlyLDAPUserStoreManager, using a UserStoreManager configuration that works without problem when tested w/ wso2/docker-apim.
• List Roles in management console

Hi,

When run with two or more replicas, the apim-analytics can be resilient to failures, providing a highly-available service.

Why there is two dockers images apim-analytics-1 (https://github.com/wso2/kubernetes-apim/tree/2.1.0/pattern-1/dockerfiles/apim-analytics-1) and apim-analytics-2 (https://github.com/wso2/kubernetes-apim/tree/2.1.0/pattern-1/dockerfiles/apim-analytics-2) ? While it is enough, from my point of view, a single image with 2 as a replica in the wso2apim-analytics-deployment.yaml file.

Thanks,
Youcef HILEM

Hi,

The ports 9763 and 9443 are missed in PODs tm1 & tm2 for pattern-3.
These ports are used in service : wso2apim-tm-service.yaml

Many exceptions, like this are thrown

[2017-09-29 15:41:27,288] ERROR - KeyTemplateRetriever Exception when retrieving throttling data from remote endpoint
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:522)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:401)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:178)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at org.wso2.carbon.apimgt.gateway.throttling.util.KeyTemplateRetriever.retrieveKeyTemplateData(KeyTemplateRetriever.java:83)
at org.wso2.carbon.apimgt.gateway.throttling.util.KeyTemplateRetriever.loadKeyTemplatesFromWebService(KeyTemplateRetriever.java:111)
at org.wso2.carbon.apimgt.gateway.throttling.util.KeyTemplateRetriever.run(KeyTemplateRetriever.java:54)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Exception in thread "Timer-6" java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at java.util.Arrays$ArrayList.(Arrays.java:3813)
at java.util.Arrays.asList(Arrays.java:3800)
at org.wso2.carbon.apimgt.gateway.throttling.util.KeyTemplateRetriever.loadKeyTemplatesFromWebService(KeyTemplateRetriever.java:111)
at org.wso2.carbon.apimgt.gateway.throttling.util.KeyTemplateRetriever.run(KeyTemplateRetriever.java:54)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)

Thanks,
Youcef HILEM

Description:

The below confs refers to the headless KM service wso2apim-key-manager which is not load balanced by the platform as per [1]. Instead, the wso2apim-km-service should be used which will be load balanced.

• https://github.com/wso2/kubernetes-apim/blob/2.1.0-nfs/pattern-3/confs/apim-publisher/repository/conf/api-manager.xml#L19
• https://github.com/wso2/kubernetes-apim/blob/2.1.0-nfs/pattern-3/confs/apim-store/repository/conf/api-manager.xml#L19

Suggested Labels:
Type/Bug, Priority/High

Suggested Assignees:
@pubudu538

Affected Product Version:
2.1.0

OS, DB, other environment details and versions:

Steps to reproduce:

Related Issues:

Hi,
I have defined a system variable in repository/conf/axis2/axis2.xml :

    <parameter name="keystore" locked="false">
        <KeyStore>
            <Location>${wso2_keystores_keystore_location}</Location>
            <Type>JKS</Type>
            <Password>wso2carbon</Password>
            <KeyPassword>wso2carbon</KeyPassword>
        </KeyStore>
    </parameter>

</transportReceiver>

But apparently, reading the environment variables is not supported for the file axis2.xml.
At startup, I have this exception:

2017-08-13T07:21:42.766223800Z [2017-08-13 07:21:42,765] INFO - PassThroughHttpSSLSender No proxy configuration found
2017-08-13T07:21:42.770248300Z [2017-08-13 07:21:42,768] ERROR - ClientConnFactoryBuilder HTTPS Error opening Keystore : ${wso2_keystores_keystore_location}
2017-08-13T07:21:42.770296200Z java.io.FileNotFoundException: ${wso2_keystores_keystore_location} (No such file or directory)
2017-08-13T07:21:42.770398800Z at java.io.FileInputStream.open0(Native Method)
2017-08-13T07:21:42.770413100Z at java.io.FileInputStream.open(FileInputStream.java:195)
2017-08-13T07:21:42.770449800Z at java.io.FileInputStream.(FileInputStream.java:138)
2017-08-13T07:21:42.770477000Z at java.io.FileInputStream.(FileInputStream.java:93)
2017-08-13T07:21:42.770508600Z at org.apache.synapse.transport.nhttp.config.ClientConnFactoryBuilder.createSSLContext(ClientConnFactoryBuilder.java:285)
2017-08-13T07:21:42.770519000Z at org.apache.synapse.transport.nhttp.config.ClientConnFactoryBuilder.parseSSL(ClientConnFactoryBuilder.java:100)
2017-08-13T07:21:42.770562700Z at org.apache.synapse.transport.passthru.PassThroughHttpSSLSender.initConnFactoryBuilder(PassThroughHttpSSLSender.java:45)
2017-08-13T07:21:42.770573300Z at org.apache.synapse.transport.passthru.PassThroughHttpSender.init(PassThroughHttpSender.java:160)
2017-08-13T07:21:42.770605700Z at org.apache.synapse.transport.passthru.PassThroughHttpSSLSender.init(PassThroughHttpSSLSender.java:33)
2017-08-13T07:21:42.770616200Z at org.apache.axis2.context.ConfigurationContextFactory.initTransportSenders(ConfigurationContextFactory.java:300)
2017-08-13T07:21:42.770656900Z at org.apache.axis2.context.ConfigurationContextFactory.init(ConfigurationContextFactory.java:231)
2017-08-13T07:21:42.770683500Z at org.apache.axis2.context.ConfigurationContextFactory.createConfigurationContext(ConfigurationContextFactory.java:93)
2017-08-13T07:21:42.770714300Z at org.wso2.carbon.core.CarbonConfigurationContextFactory.createNewConfigurationContext(CarbonConfigurationContextFactory.java:65)
2017-08-13T07:21:42.770725700Z at org.wso2.carbon.core.init.CarbonServerManager.initializeCarbon(CarbonServerManager.java:398)
2017-08-13T07:21:42.770753200Z at org.wso2.carbon.core.init.CarbonServerManager.start(CarbonServerManager.java:219)
2017-08-13T07:21:42.770770500Z at org.wso2.carbon.core.internal.CarbonCoreServiceComponent.activate(CarbonCoreServiceComponent.java:94)
2017-08-13T07:21:42.770798200Z at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2017-08-13T07:21:42.770808400Z at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2017-08-13T07:21:42.770835100Z at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2017-08-13T07:21:42.770845400Z at java.lang.reflect.Method.invoke(Method.java:498)
2017-08-13T07:21:42.770870200Z at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
2017-08-13T07:21:42.770880200Z at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
2017-08-13T07:21:42.770905900Z at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345)
2017-08-13T07:21:42.770916000Z at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
2017-08-13T07:21:42.770941600Z at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
2017-08-13T07:21:42.770952600Z at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
2017-08-13T07:21:42.770982900Z at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
2017-08-13T07:21:42.771328700Z at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
2017-08-13T07:21:42.771360900Z at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
2017-08-13T07:21:42.771368200Z at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
2017-08-13T07:21:42.771372200Z at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
2017-08-13T07:21:42.771376800Z at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
2017-08-13T07:21:42.771380900Z at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
2017-08-13T07:21:42.771384000Z at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
2017-08-13T07:21:42.771387000Z at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
2017-08-13T07:21:42.771389800Z at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
2017-08-13T07:21:42.771393000Z at org.eclipse.equinox.http.servlet.internal.Activator.registerHttpService(Activator.java:81)
2017-08-13T07:21:42.771395900Z at org.eclipse.equinox.http.servlet.internal.Activator.addProxyServlet(Activator.java:60)
2017-08-13T07:21:42.771398900Z at org.eclipse.equinox.http.servlet.internal.ProxyServlet.init(ProxyServlet.java:40)
2017-08-13T07:21:42.771402000Z at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.init(DelegationServlet.java:38)
2017-08-13T07:21:42.771408300Z at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1269)
2017-08-13T07:21:42.771411200Z at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
2017-08-13T07:21:42.771414000Z at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
2017-08-13T07:21:42.771416800Z at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
2017-08-13T07:21:42.771419600Z at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
2017-08-13T07:21:42.771422300Z at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
2017-08-13T07:21:42.771425100Z at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1571)
2017-08-13T07:21:42.771427900Z at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1561)
2017-08-13T07:21:42.771430900Z at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2017-08-13T07:21:42.771433900Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2017-08-13T07:21:42.771436700Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2017-08-13T07:21:42.771449300Z at java.lang.Thread.run(Thread.java:748)

.........................................................................................................................

Some values (variables) are specific to each environment (DEV, QA, PRE-PROD, PROD).
If I can not put them in variables, this will result in the creation of as many docker images as the environments for the same component.

In addition, it is already supported for other configuration files (https://wso2.org/jira/browse/CARBON-15791).

Thanks,
Youcef HILEM

Issue Description:

Not support Amazon EFS.
We can't install correctly the pattern-2 in Kubernetes (AWS). We've found that in the community has the same problem, and they solved themselves, but not in this repository.
For a clean Installation, we need to Install another component (Addons EFS only in incubator version, not ready for production) and adapt this code.

• Could you give support for Kubernetes on AWS?
• Is this repository only supported by the community?

Warning ProvisioningFailed 7s (x23 over 5m) persistentvolume-controller Failed to provision volume with StorageClass "gp2": invalid AccessModes [ReadWriteMany]: only AccessModes [ReadWriteOnce] are supported

Affected Product Version:

branch master pattern-2
(We use this version because the last commit of the last tag has 8 month ago)

OS, DB, other environment details and versions:

• Cloud: Amazon Web Service
• Tool to Deploy Kubernetes in AWS: Kops Version 1.9.0
• Cluster Kubernetes version 1.9.3.
• Addon Kube-Ingress Controller v1.6.0.yaml (This is a requirement but it isn't in the documentation)

Steps to reproduce:

1. Create EFS in AWS. (fs-xxxxx.efs.eu-west-1.amazonaws.com)

2. Update the configuration with the new value, in all the lines.
   artifacts/volumes/persistent-volumes.yaml

   Old value:

   server: 10.11.253.199

   New value:

   server: fs-xxxxx.efs.eu-west-1.amazonaws.com

3. To Update (deploy-kubernetes.yaml with Identity Manager) and run installation:
   $ ./deploy-kubernetes.yaml
   The pvc's always are in Pending state.

   NAME                      STATUS    VOLUME
   apim-is-km-volume-claim   Pending    gp2
   apim-mgt-volume-claim     Pending    gp2
   apim-rdbms-volume-claim   Pending    gp2
   apim-tm1-volume-claim     Pending    gp2
   

   $ kubectl describe pvc apim-is-km-volume-claim

     Name:          apim-is-km-volume-claim
     Namespace:     wso2
     StorageClass:  gp2
     Status:        Pending
     Volume:
     Labels:        app=apim-is-km
                   pattern=wso2apim-pattern-2
     Annotations:   volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/aws-ebs
     Finalizers:    []
     Capacity:
     Access Modes:
     Events:
        Type     Reason              Age                From                         Message
      ----     ------              ----               ----                         -------
    Warning  ProvisioningFailed  3m (x522 over 2h)  persistentvolume-controller  Failed to provision volume with StorageClass "gp2": invalid AccessModes [ReadWriteMany]: only AccessModes [ReadWriteOnce] are supported
   
   

Warning ProvisioningFailed 3m (x522 over 2h) persistentvolume-controller Failed to provision volume with StorageClass "gp2": invalid AccessModes [ReadWriteMany]: only AccessModes

Related Issues:

Has anyone deployed this on Kubernetes running on AWS? #79

Hi,

Could you please integrate secure vault?
It is a prerequisite for any deployment in production.
I have read (http://wso2-oxygen-tank.10903.n7.nabble.com/Kubernetes-Artifacts-Applying-secure-vault-for-wso2-products-when-building-docker-images-tt132003.html#a151560) and (https://github.com/wso2/docker-common/tree/master/scripts) but it is better to have a qualified solution.

Thanks
Youcef HILEM

Description:
We are having volume mounts for <ANALYTICS_HOME>/data directory where the indexing data resides and <ANALYTICS_HOME>/repository/conf/analytics directory to the file system.
In the rolling-updates scenario when the updated node tries to acquire the lock from the file system, the previous node might still hold on to the lock.

Hi,

Can you please add a rdbms folder at https://github.com/wso2/kubernetes-apim/tree/2.1.0/base for the mysql docker image: docker.wso2.com/apim-rdbms-kubernetes:2.1.0 used in pattern-X/artifacts/rdbms/rdbms-deployment.yaml ?

Thanks,
Youcef HILEM

Description:
I am wondering if anyone successfully implemented this on Kubernetes running on AWS EC2? I got everything deployed, including nginx ingress. Everything seems to be working, it even created Classic Load Balancer on AWS with two listeners 80/443, and a security group that accepts all inbound traffic. I am not able to get store or publisher sites working. It is always going to default backend.

kubectl get pods --namespace wso2

apim-rdbms-deployment-fc688fcd8-8l2mp     1/1       Running   0          21h
efs-provisioner-578dc8989f-kt2f4          1/1       Running   0          15d
wso2apim-analytics-1-84d66b6d66-9mxjc     1/1       Running   0          21h
wso2apim-analytics-2-579645c7f4-lgz44     1/1       Running   0          21h
wso2apim-is-as-km-69f85fb856-46vxs        1/1       Running   0          21h
wso2apim-manager-worker-69fdc9994-9tk6c   1/1       Running   0          21h
wso2apim-manager-worker-69fdc9994-g7j7p   1/1       Running   0          21h
wso2apim-pubstore-tm-1-7cf75854bf-jdqmt   1/1       Running   0          21h
wso2apim-pubstore-tm-2-8568c75b45-fn9tn   1/1       Running   0          21h

kubectl get services --namespace wso2

apim-rdbms                ClusterIP   None             <none>        3306/TCP                                                         21h
wso2apim                  NodePort    10.96.252.21     <none>        9763:31232/TCP,9443:31389/TCP                                    17h
wso2apim-analytics        ClusterIP   10.96.105.108    <none>        9764/TCP,9444/TCP,8082/TCP,4041/TCP,11501/TCP                    21h
wso2apim-analytics-1      ClusterIP   None             <none>        7612/TCP,7712/TCP,11225/TCP,10006/TCP,11001/TCP                  21h
wso2apim-analytics-2      ClusterIP   None             <none>        7612/TCP,7712/TCP,11225/TCP,10006/TCP,11001/TCP                  21h
wso2apim-gw               ClusterIP   10.106.238.224   <none>        8280/TCP,8243/TCP                                                21h
wso2apim-gw-sv            ClusterIP   10.106.110.32    <none>        9763/TCP,9443/TCP                                                21h
wso2apim-km-clustering    ClusterIP   None             <none>        9673/TCP,9443/TCP                                                21h
wso2apim-km-internal      ClusterIP   10.108.235.110   <none>        9763/TCP,9443/TCP                                                21h
wso2apim-manager-worker   ClusterIP   None             <none>        9673/TCP,9443/TCP                                                21h
wso2apim-pubstore-tm-1    ClusterIP   None             <none>        5672/TCP,9711/TCP,9611/TCP,7711/TCP,7611/TCP,9763/TCP,9443/TCP   17h
wso2apim-pubstore-tm-2    ClusterIP   None             <none>        5672/TCP,9711/TCP,9611/TCP,7711/TCP,7611/TCP,9763/TCP,9443/TCP   17h

kubectl get pods --namespace ingress-nginx

default-http-backend-55c6c69b88-bp4wt       1/1       Running   0          20h
nginx-ingress-controller-78cc9d4ccb-xtjxt   1/1       Running   0          20h

kubectl get services --namespace ingress-nginx

default-http-backend   ClusterIP      10.96.63.34      <none>             80/TCP                       20h
ingress-nginx          LoadBalancer   10.108.178.119   a0c93fd6939d6...   80:31310/TCP,443:30640/TCP   20h

curl -v http://a0c93fd6939d611e885020a6cd68e025-1142036405.us-east-1.elb.amazonaws.com:443/store
*   Trying 34.202.146.173...
* TCP_NODELAY set
* Connected to a0c93fd6939d611e885020a6cd68e025-1142036405.us-east-1.elb.amazonaws.com (34.202.146.173) port 443 (#0)
> GET /store HTTP/1.1
> Host: a0c93fd6939d611e885020a6cd68e025-1142036405.us-east-1.elb.amazonaws.com:443
> User-Agent: curl/7.57.0
> Accept: */*
> 
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

This error can be seen in analytics pod 2.

[2017-09-17 11:43:11,554] ERROR {org.wso2.carbon.event.processor.manager.core.internal.HAManager} - Cannot start CEP HA Snapshot Server on port wso2apim-analytics-1:10006 org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address wso2apim-analytics-1/X.Y.125.24:10006. at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:109) at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:91) at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:87) at org.wso2.carbon.event.processor.manager.core.internal.HAManager$SnapshotServer.start(HAManager.java:383) at org.wso2.carbon.event.processor.manager.core.internal.HAManager.<init>(HAManager.java:84) at org.wso2.carbon.event.processor.manager.core.internal.CarbonEventManagementService.init(CarbonEventManagementService.java:140) at org.wso2.carbon.event.processor.manager.core.internal.ds.EventManagementServiceDS.setHazelcastInstance(EventManagementServiceDS.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.eclipse.equinox.internal.ds.model.ComponentReference.bind(ComponentReference.java:376) at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.bindReference(ServiceComponentProp.java:430) at org.eclipse.equinox.internal.ds.InstanceProcess.dynamicBind(InstanceProcess.java:416) at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:334) at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222) at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861) at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230) at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:950) at org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent.init(HazelcastClusteringAgent.java:229) at org.wso2.carbon.core.util.ClusteringUtil.enableClustering(ClusteringUtil.java:38) at org.wso2.carbon.core.internal.StartupFinalizerServiceComponent.completeInitialization(StartupFinalizerServiceComponent.java:177) at org.wso2.carbon.core.internal.StartupFinalizerServiceComponent.serviceChanged(StartupFinalizerServiceComponent.java:288) at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861) at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230) at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451) at org.wso2.carbon.server.admin.internal.ServerAdminServiceComponent.activate(ServerAdminServiceComponent.java:106) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260) at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146) at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345) at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620) at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197) at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343) at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222) at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861) at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230) at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451) at org.wso2.carbon.core.init.CarbonServerManager.initializeCarbon(CarbonServerManager.java:514) at org.wso2.carbon.core.init.CarbonServerManager.start(CarbonServerManager.java:219) at org.wso2.carbon.core.internal.CarbonCoreServiceComponent.activate(CarbonCoreServiceComponent.java:94) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260) at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146) at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345) at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620) at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197) at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343) at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222) at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861) at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230) at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433) at org.eclipse.equinox.http.servlet.internal.Activator.registerHttpService(Activator.java:81) at org.eclipse.equinox.http.servlet.internal.Activator.addProxyServlet(Activator.java:60) at org.eclipse.equinox.http.servlet.internal.ProxyServlet.init(ProxyServlet.java:40) at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.init(DelegationServlet.java:38) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1269) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1571) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1561) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

I would appreciate guidance on proper configuration to support installation of a CA-signed certificate for external HTTPS connections. I am restricted to generating CA-signed certificates for a given subdomain, say *.someorganization.org. So this certificate isn't valid for the *.wso2.svc hostnames that are used for internal connections between the Kubernetes pods. So it seems that I should use my CA-signed certificate for the external facing HTTPS connections (e.g. the API gateway, store, publisher, Carbon admin) and use the *.wso2.svc certificate for connections internal to the OpenShift / Kubernetes network.

Is this possible?

By the way, previously I had been using re-encrypted route to provide by external HTTPS certificates rather than passthrough SSL, however this seems to have broken with the latest wso2carbon.jks keystore. (Even when I provide the new wso2carbon certificate as the destination CA certificate).

Hi,

It is stated in the documentation (https://docs.wso2.com/display/AM210/Authorization+Code+Grant) that
"In a distributed API Manager setup .... be sure to give the full URL of the authentication server in the <APIM_HOME>/repository/conf/identity/application-authentication.xml file. "

For my case (https://github.com/wso2/kubernetes-apim/tree/2.1.0/pattern-2), this configuration is not modified. Thus it's the default configuration that has a relative path, which works in a standalone API Manager that is applied.

In our case (with wso2apim-gw-route) , we have the following urls for:

• Authorize Endpoint : "https://wso2apim-gw/authorize"
• Token endpoint : "https://wso2apim-gw/token"
• Userinfo endpoint : "https://wso2apim-gw/userinfo"
• Revoke token endpoint : "https://wso2apim-gw/revoke"

Thanks,
Youcef HILEM

Hi,
the installation part need to login to wso2 registry to download image, but the login information was missing, can anyone fix it , or build image only through Dockerfile it provided, thx

Currently, with Kubernetes v1.8.0 API-M Analytics pods do not get started because of apim-analytics-1-conf-analytics and apim-analytics-2-conf-analytics configmap volumes not being created as they do not contain any config files:

going to change ownership of <ANALYTICS_HOME>/repository/data directory: 
user: root
user home: /home/wso2user
carbon server: wso2am-analytics-2.1.0
cp: cannot stat '/home/wso2user/wso2am-analytics-2.1.0-conf/conf-analytics/*': No such file or directory

[1] https://github.com/wso2/kubernetes-apim/blob/master/base/analytics/init_carbon.sh#L88
[2] ​https://github.com/wso2/kubernetes-apim/blob/master/pattern-1/artifacts/apim-analytics/wso2apim-analytics-1-deployment.yaml

Description:
Given that API Manager 2.2 was just released, is there a timeline for supporting 2.2 with this project?

If this isn't imminent, aside from following the 2.1 -> 2.2 upgrade documentation, any recommendations on installing 2.2 using this project? We were getting close to deploying 2.1, but are considering options for deploying 2.2 and avoiding the need to update.

Thanks!

Suggested Labels:
Question

Hi,

H2 is used for for message broker database (https://github.com/wso2/kubernetes-apim/blob/2.1.0/pattern-2/dockerfiles/apim-pubstore-tm-1/carbon/repository/conf/datasources/master-datasources.xml#L118).

H2 DB is not recommend for a production environment (http://wso2-oxygen-tank.10903.n7.nabble.com/Traffic-Manager-HA-deployment-on-DC-OS-td146737.html)

Also, we have more than one Traffic Manager node, and each Traffic Manager node must have its own WSO2_MB_STORE_DB database (https://docs.wso2.com/display/AM210/Distributed+Deployment+of+API+Manager).

Thanks
Youcef HILEM

Description:
Create Kubernetes resources for WSO2 API Manager version 2.5.0 deployment pattern 1 [1].

[1]: Deployment pattern 1

Suggested Labels:
Type/Task

Suggested Assignees:
chirangaalwis

Affected Product Version:
WSO2 API Manager v2.5.x

Issue:

There are more than 63 yaml files which they have hardcoded configurations. We need to check one by one in order to make the proper changes. This is a problem when you have multiple environments.

Do you believe that is good idea?

Request:

The configuration has to be parametrized following the standard Charts so we could use Helm to following the pattern IaC (Infrastructure as Code).
i.e https://github.com/kubernetes/charts/tree/master/stable

Hi,

Is the hostname at https://github.com/wso2/kubernetes-apim/blob/2.1.0/pattern-2/dockerfiles/apim-pubstore-tm-2/carbon/repository/conf/api-manager.xml#L592 is wso2apim-pubstore-tm-2.default.svc instead of wso2apim-pubstore-tm-1.default.svc

This is what is done in https://github.com/wso2/kubernetes-apim/blob/2.1.0/pattern-2/dockerfiles/apim-pubstore-tm-1/carbon/repository/conf/api-manager.xml#L592

Thanks
Youcef HILEM

Hi,

There is deploy-openshift.sh and undeploy-openshift.sh scripts for pattern-1 and pattern-2.

But not for pattern-3.

Can you please add deploy-openshift.sh and undeploy-openshift.sh scripts for pattern-3 ?

Thanks,
Youcef HILEM

Hi,

Ports exposed in docker image (7711 7611) (https://github.com/wso2/kubernetes-apim/blob/2.1.0/base/analytics/Dockerfile#L95) for thrift doesn't match POD targetport (7712 7612) for wso2apim-analytics-1-deployment (https://github.com/wso2/kubernetes-apim/blob/2.1.0/pattern-2/artifacts/apim-analytics/wso2apim-analytics-1-deployment.yaml#L46) & wso2apim-analytics-2-deployment

Thus, these exception are thrown :

[2017-09-16 22:10:03,190] ERROR - DataEndpointConnectionWorker Error while trying to connect to ssl://wso2apim-analytics-1:7712
org.wso2.carbon.databridge.agent.exception.DataEndpointSecurityException: Error while trying to connect to ssl://wso2apim-analytics-1:7712
at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftSecureClientPoolFactory.createClient(ThriftSecureClientPoolFactory.java:81)
at org.wso2.carbon.databridge.agent.client.AbstractClientPoolFactory.makeObject(AbstractClientPoolFactory.java:39)
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1212)
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:91)
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:42)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: Could not connect to wso2apim-analytics-1 on port 7712
at org.apache.thrift.transport.TSSLTransportFactory.createClient(TSSLTransportFactory.java:237)
at org.apache.thrift.transport.TSSLTransportFactory.getClientSocket(TSSLTransportFactory.java:169)
at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftSecureClientPoolFactory.createClient(ThriftSecureClientPoolFactory.java:63)
... 9 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at sun.security.ssl.SSLSocketImpl.(SSLSocketImpl.java:427)
at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:88)
at org.apache.thrift.transport.TSSLTransportFactory.createClient(TSSLTransportFactory.java:233)
... 11 more

.........

Thanks,
Youcef HILEM

Hi,

An exception is thrown when we access the following api stats

[2017-09-19 16:46:01,108] ERROR - APIUsageStatisticsRdbmsClientImpl Error occurred while checking existence of the table:API_EXE_TIME_HOUR_SUMMARY
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'o2_apim_stat_db.API_EXE_TIME_HOUR_SUMMARY' doesn't exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
at com.mysql.jdbc.Util.getInstance(Util.java:360)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:978)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2526)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484)
at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1446)
at org.wso2.carbon.apimgt.usage.client.impl.APIUsageStatisticsRdbmsClientImpl.isTableExist(APIUsageStatisticsRdbmsClientImpl.java:2040)
at org.wso2.carbon.apimgt.usage.client.impl.APIUsageStatisticsRdbmsClientImpl.getExecutionTimeByAPI(APIUsageStatisticsRdbmsClientImpl.java:2644)
at org.wso2.carbon.apimgt.usage.client.impl.APIUsageStatisticsRdbmsClientImpl.getExecutionTimeByAPI(APIUsageStatisticsRdbmsClientImpl.java:2599)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
at org.mozilla.javascript.NativeJavaMethod.call(NativeJavaMethod.java:225)
at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
at org.jaggeryjs.rhino.publisher.modules.statistics.c1._c_getExecutionTimeByAPI_22(/publisher/modules/statistics/usage.jag:661)
at org.jaggeryjs.rhino.publisher.modules.statistics.c1.call(/publisher/modules/statistics/usage.jag)
at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430)
at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269)
at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97)
at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42)
at org.jaggeryjs.rhino.publisher.modules.statistics.c0._c_anonymous_25(/publisher/modules/statistics/module.jag:57)
at org.jaggeryjs.rhino.publisher.modules.statistics.c0.call(/publisher/modules/statistics/module.jag)
at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
at org.jaggeryjs.rhino.publisher.site.blocks.stats.api_latencytime.ajax.c0._c_anonymous_1(/publisher/site/blocks/stats/api-latencytime/ajax/stats.jag:31)
at org.jaggeryjs.rhino.publisher.site.blocks.stats.api_latencytime.ajax.c0.call(/publisher/site/blocks/stats/api-latencytime/ajax/stats.jag)
at org.mozilla.javascript.optimizer.OptRuntime.call0(OptRuntime.java:23)
at org.jaggeryjs.rhino.publisher.site.blocks.stats.api_latencytime.ajax.c0._c_script_0(/publisher/site/blocks/stats/api-latencytime/ajax/stats.jag:4)
at org.jaggeryjs.rhino.publisher.site.blocks.stats.api_latencytime.ajax.c0.call(/publisher/site/blocks/stats/api-latencytime/ajax/stats.jag)
at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394)
at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3091)
at org.jaggeryjs.rhino.publisher.site.blocks.stats.api_latencytime.ajax.c0.call(/publisher/site/blocks/stats/api-latencytime/ajax/stats.jag)
at org.jaggeryjs.rhino.publisher.site.blocks.stats.api_latencytime.ajax.c0.exec(/publisher/site/blocks/stats/api-latencytime/ajax/stats.jag)
at org.jaggeryjs.scriptengine.engine.RhinoEngine.execScript(RhinoEngine.java:567)
at org.jaggeryjs.scriptengine.engine.RhinoEngine.exec(RhinoEngine.java:273)
at org.jaggeryjs.jaggery.core.manager.WebAppManager.exec(WebAppManager.java:588)
at org.jaggeryjs.jaggery.core.manager.WebAppManager.execute(WebAppManager.java:508)
at org.jaggeryjs.jaggery.core.JaggeryServlet.doPost(JaggeryServlet.java:29)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:747)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:485)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:377)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:337)
at org.jaggeryjs.jaggery.core.JaggeryFilter.doFilter(JaggeryFilter.java:21)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.wso2.carbon.ui.filters.cache.ContentTypeBasedCachePreventionFilter.doFilter(ContentTypeBasedCachePreventionFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:120)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
at org.wso2.carbon.event.receiver.core.internal.tenantmgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:48)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:958)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1756)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1715)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
[2017-09-19 16:46:01,111] ERROR - APIUsageStatisticsClient Statistics Table:API_EXE_TIME_HOUR_SUMMARY does not exist.

Thank,
Youcef HILEM

Hi,

Affter we have changed the admin password with console, we were forced to rebuild the dockers images by also changing the password in the config (because we are no longer able to login).
And of course, do another deployment.
Is this a bug?

Thanks
Youcef HILEM

The following error is raised when Docker image build process is executed via the build.sh file on Minishift v1.5.0+ae62cf2 with Docker 1.24:

"--squash" requires API version 1.25, but the Docker daemon API version is 1.24

Issue Description:

The current WSO2’s repository has the kind property set as Deployment value. It is not a good practice to use Deployment with multiples Availability Zones with AWS block devices volumes (EBS). The reason is because the database POD could be recreated in other Availability Zone if it goes down, causing the POD can’t attach the volume again, since they’re resources attached just to one Availability Zones. This will cause the DB will get corrupted.

In the official documentation of kubernetes recommend to use Statefulset for these apps. IMPORTANT: StatefulSets are stable (GA) in 1.9.
https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/

$ cat pattern-2/artifacts/rdbms/rdbms-deployment.yaml

apiVersion: extensions/v1beta1 ## IMPORTANT: Not ready for production update the version of your Kubernetes Cluster.
kind: Deployment
metadata:
  name: apim-rdbms-deployment
  labels:
    app: apim-rdbms
    pattern: wso2apim-pattern-2

Possible solution:

To change the Deployment to Statefulset (stable version only in Kubernetes version 1.9.0)

Affected Product Version:

branch master pattern-2

OS, DB, other environment details and versions:

• Cloud: Amazon Web Service
• Tool to Deploy Kubernetes in AWS: Kops Version 1.9.0
• Cluster Kubernetes version 1.9.3.
• Addon Kube-Ingress Controller v1.6.0.yaml

Related Issues:

Move from Deployment resource to StatefulSet resource #72
Database Connectivity Issue: #34

Description:
Create Kubernetes resources for WSO2 API Manager version 2.2.0 deployment pattern 2 [1].

[1]: Deployment pattern 2

Suggested Labels:
Type/NewFeature

Suggested Assignees:
chirangaalwis

Affected Product Version:
WSO2 API Manager 2.2.x

Description:
In [1] we use statdb as event store database and processed database. IMO it should be analyticseventstoredb and analyticprocesseddatastoredb which we created in [2].

[1] https://github.com/wso2/kubernetes-apim/blob/master/pattern-2/confs/apim-analytics-1/repository/conf/datasources/analytics-datasources.xml#L16
[2] https://github.com/wso2/kubernetes-apim/blob/master/base/mysql/scripts/mysql-statdbs.sql

Description:

In deploy-kubernetes.sh changing context globally is problematic.

If changing context fails, script deploy artifacts in the default context.

The script needs to check the NS before the deployment.

Description:
In Kubernetes, deployment is for the stateless applications and statefulset is for stateful applications. We should move to statefulset.

https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/

Hi,
With NFS, configuring GW is simpler. Only one POD: gw-manager-worker.
This is scalable by simply increasing the value of the replication factor.

The same applies to apim-pubstore-tm1 and apim-pubstore-tm2 of the pattern-2, but by distributing the TM on other PODs.
It is therefore relevant to remove the TM component from this PODs.

This is why I propose to create another pattern-5 from pattern-2 by :

• putting TM1 and TM2 on two other PODs.
• Keep only one POD : apim-pubstore

This is also beneficial from the operational point of view, the only three critical components for API consumption will be the two autoscalable POD:

• gw-manager-worker
• KM

And :

• TM1 & TM2

Thanks in advance.
Youcef HILEM

The following error was noticed in the Analytics node at the startup, may be max_connections might need to be increased with MySQL 5.7:

Successfully updated localMemberHost with 172.17.0.4
JAVA_HOME environment variable is set to /home/wso2user/java
CARBON_HOME environment variable is set to /home/wso2user/wso2am-analytics-2.1.0
Loading spark environment variables 
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
[2017-09-13 01:28:19,517]  INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} -  Starting WSO2 Carbon...
[2017-09-13 01:28:19,518]  INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} -  Operating System : Linux 4.4.27-boot2docker, amd64
[2017-09-13 01:28:19,518]  INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} -  Java Home        : /home/wso2user/java/jre
[2017-09-13 01:28:19,518]  INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} -  Java Version     : 1.8.0_144
[2017-09-13 01:28:19,518]  INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} -  Java VM          : Java HotSpot(TM) 64-Bit Server VM 25.144-b01,Oracle Corporation
[2017-09-13 01:28:19,519]  INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} -  Carbon Home      : /home/wso2user/wso2am-analytics-2.1.0
[2017-09-13 01:28:19,519]  INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} -  Java Temp Dir    : /home/wso2user/wso2am-analytics-2.1.0/tmp
[2017-09-13 01:28:19,519]  INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} -  User             : wso2user, en-US, Etc/UTC
[2017-09-13 01:28:19,894]  INFO {org.wso2.carbon.event.output.adapter.kafka.internal.ds.KafkaEventAdapterServiceDS} -  Successfully deployed the Kafka output event adaptor service
[2017-09-13 01:28:19,989]  INFO {org.wso2.carbon.event.processor.manager.commons.transport.server.TCPEventServer} -  EventServer starting event listener on 172.17.0.4:11225
[2017-09-13 01:28:19,990]  INFO {org.wso2.carbon.event.processor.manager.core.internal.EventHandler} -  Event Management TCPEventServer for EventReceiver started on port 11225
[2017-09-13 01:28:20,088]  INFO {org.wso2.carbon.event.template.manager.core.internal.ds.TemplateDeployerServiceTrackerDS} -  Successfully deployed the execution manager tracker service
Wed Sep 13 01:28:22 UTC 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
[2017-09-13 01:28:23,484]  INFO {org.wso2.carbon.databridge.receiver.binary.internal.BinaryDataReceiver} -  Started Binary SSL Transport on port : 9712
[2017-09-13 01:28:23,489]  INFO {org.wso2.carbon.databridge.receiver.binary.internal.BinaryDataReceiver} -  Started Binary TCP Transport on port : 9612
[2017-09-13 01:28:23,502]  INFO {org.wso2.carbon.databridge.core.internal.DataBridgeDS} -  Successfully deployed Agent Server 
Wed Sep 13 01:28:24 UTC 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
[2017-09-13 01:28:28,507] ERROR {org.wso2.carbon.registry.core.config.RegistryContext} -  Unable to get instance of the registry context
org.wso2.carbon.registry.core.exceptions.RegistryException: Unable to connect to Data Source
	at org.wso2.carbon.registry.core.config.RegistryConfigurationProcessor.populateRegistryConfig(RegistryConfigurationProcessor.java:165)
	at org.wso2.carbon.registry.core.config.RegistryContext.<init>(RegistryContext.java:462)
	at org.wso2.carbon.registry.core.config.RegistryContext.<init>(RegistryContext.java:443)
	at org.wso2.carbon.registry.core.config.RegistryContext.getBaseInstance(RegistryContext.java:322)
	at org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent.getEmbeddedRegistryService(RegistryCoreServiceComponent.java:587)
	at org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent.buildRegistryService(RegistryCoreServiceComponent.java:520)
	at org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent.activate(RegistryCoreServiceComponent.java:126)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
	at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
	at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345)
	at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
	at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
	at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
	at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
	at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
	at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
	at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
	at org.wso2.carbon.user.core.internal.Activator.startDeploy(Activator.java:69)
	at org.wso2.carbon.user.core.internal.BundleCheckActivator.start(BundleCheckActivator.java:61)
	at org.eclipse.osgi.framework.internal.core.BundleContextImpl$1.run(BundleContextImpl.java:711)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.eclipse.osgi.framework.internal.core.BundleContextImpl.startActivator(BundleContextImpl.java:702)
	at org.eclipse.osgi.framework.internal.core.BundleContextImpl.start(BundleContextImpl.java:683)
	at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(BundleHost.java:381)
	at org.eclipse.osgi.framework.internal.core.AbstractBundle.resume(AbstractBundle.java:390)
	at org.eclipse.osgi.framework.internal.core.Framework.resumeBundle(Framework.java:1176)
	at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:559)
	at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:544)
	at org.eclipse.osgi.framework.internal.core.StartLevelManager.incFWSL(StartLevelManager.java:457)
	at org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(StartLevelManager.java:243)
	at org.eclipse.osgi.framework.internal.core.StartLevelManager.dispatchEvent(StartLevelManager.java:438)
	at org.eclipse.osgi.framework.internal.core.StartLevelManager.dispatchEvent(StartLevelManager.java:1)
	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
	at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:340)
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
	at com.mysql.jdbc.Util.getInstance(Util.java:408)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:918)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:897)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:886)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860)
	at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2104)
	at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2029)
	at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779)
	at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
	at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389)
	at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
	at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:278)
	at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182)
	at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:701)
	at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:635)
	at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)
	at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:127)
	at org.wso2.carbon.registry.core.config.RegistryConfigurationProcessor.populateRegistryConfig(RegistryConfigurationProcessor.java:149)
	... 43 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet successfully received from the server was 67 milliseconds ago.  The last packet sent successfully to the server was 67 milliseconds ago.
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
	at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
	at com.mysql.jdbc.ExportControlled.transformSocketToSSLSocket(ExportControlled.java:178)
	at com.mysql.jdbc.MysqlIO.negotiateSSLConnection(MysqlIO.java:4901)
	at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1659)
	at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1226)
	at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2198)
	at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2045)
	... 60 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
	at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
	at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
	at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:886)
	at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:857)
	at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:727)
	at sun.security.ssl.Handshaker.sendChangeCipherSpec(Handshaker.java:1124)
	at sun.security.ssl.ClientHandshaker.sendChangeCipherAndFinish(ClientHandshaker.java:1216)
	at sun.security.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:1128)
	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:348)
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
	at sun.security.ssl.Handshaker.process_record(Handshaker.java:961)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
	at com.mysql.jdbc.ExportControlled.transformSocketToSSLSocket(ExportControlled.java:163)
	... 65 more
[2017-09-13 01:28:28,539] ERROR {org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent} -  Failed to activate Registry Core bundle 
java.lang.NullPointerException
	at org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent.getEmbeddedRegistryService(RegistryCoreServiceComponent.java:588)
	at org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent.buildRegistryService(RegistryCoreServiceComponent.java:520)
	at org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent.activate(RegistryCoreServiceComponent.java:126)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
	at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
	at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345)
	at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
	at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
	at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
	at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
	at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
	at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
	at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
	at org.wso2.carbon.user.core.internal.Activator.startDeploy(Activator.java:69)
	at org.wso2.carbon.user.core.internal.BundleCheckActivator.start(BundleCheckActivator.java:61)
	at org.eclipse.osgi.framework.internal.core.BundleContextImpl$1.run(BundleContextImpl.java:711)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.eclipse.osgi.framework.internal.core.BundleContextImpl.startActivator(BundleContextImpl.java:702)
	at org.eclipse.osgi.framework.internal.core.BundleContextImpl.start(BundleContextImpl.java:683)
	at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(BundleHost.java:381)
	at org.eclipse.osgi.framework.internal.core.AbstractBundle.resume(AbstractBundle.java:390)
	at org.eclipse.osgi.framework.internal.core.Framework.resumeBundle(Framework.java:1176)
	at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:559)
	at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:544)
	at org.eclipse.osgi.framework.internal.core.StartLevelManager.incFWSL(StartLevelManager.java:457)
	at org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(StartLevelManager.java:243)
	at org.eclipse.osgi.framework.internal.core.StartLevelManager.dispatchEvent(StartLevelManager.java:438)
	at org.eclipse.osgi.framework.internal.core.StartLevelManager.dispatchEvent(StartLevelManager.java:1)
	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
	at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:340)
[2017-09-13 01:28:28,648]  INFO {org.wso2.carbon.user.core.internal.UserStoreMgtDSComponent} -  Carbon UserStoreMgtDSComponent activated successfully.
[2017-09-13 01:28:35,863]  INFO {org.apache.catalina.startup.TaglibUriRule} -  TLD skipped. URI: http://tiles.apache.org/tags-tiles is already defined

Description:
The wso2carbon.jks file found in the base/analytics/files folder has used the key alias dockerapim and in the configurations the key alias has been specified as wso2carbon.

Affected Product Version:
v2.1.0-2

Steps to reproduce:

1. Extract API-M Analytics 2.1.0 distribution.
2. Copy the wso2carbon.jks file found in base/analytics/files to repository/resources/security/ folder.
3. Start the API-M Analytics server, now the following error message will appear:

[2017-11-04 19:09:39,446] ERROR {org.wso2.carbon.analytics.spark.core.internal.AnalyticsComponent} -  Error initializing analytics executor: Cannot process Carbon datasources during Spark executor initialization: null
org.wso2.carbon.analytics.datasource.commons.exception.AnalyticsException: Cannot process Carbon datasources during Spark executor initialization: null
	at org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.exportDataSourcesAsProperties(SparkAnalyticsExecutor.java:746)
	at org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.initializeSparkConf(SparkAnalyticsExecutor.java:578)
	at org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.<init>(SparkAnalyticsExecutor.java:185)
	at org.wso2.carbon.analytics.spark.core.internal.AnalyticsComponent.activate(AnalyticsComponent.java:89)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)

Hi,

Without adding these &verifyServerCertificate=false&useSSL=false&requireSSL=false parameters to jdbc urls, example :

jdbc:mysql://apim-rdbms:3306/apimgtdb?autoReconnect=true&verifyServerCertificate=false&useSSL=false&requireSSL=false

the following exception is thrown:

[2017-08-14 06:29:31,065] ERROR - RegistryContext Unable to get instance of the registry context org.wso2.carbon.registry.core.exceptions.RegistryException: Unable to connect to Data Source
at org.wso2.carbon.registry.core.config.RegistryConfigurationProcessor.populateRegistryConfig(RegistryConfigurationProcessor.java:165)
[...]
... 60 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:886)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:857)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:727)
at sun.security.ssl.Handshaker.sendChangeCipherSpec(Handshaker.java:1124)
at sun.security.ssl.ClientHandshaker.sendChangeCipherAndFinish(ClientHandshaker.java:1216)
at sun.security.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:1128)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:348)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:961)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
at com.mysql.jdbc.ExportControlled.transformSocketToSSLSocket(ExportControlled.java:163)
... 65 more

[2017-08-14 06:29:31,077] ERROR - RegistryCoreServiceComponent Failed to activate Registry Core bundle

.....................................................................................................

Best regards,
Youcef HILEM

Hi,

A Gateway worker (https://docs.wso2.com/display/AM210/Product+Profiles) : Only starts the components related to the API Gateway.
You use this when the API Gateway acts as a worker node in a cluster. This profile starts the backend features for data processing and communicating with the manager node.

In analyzing the trace I find that many services are started.

......
[2017-09-16 04:59:26,556] INFO - ManagementModeConfigurationLoader CEP started in Single node mode
[2017-09-16 04:59:26,626] INFO - TemplateDeployerServiceTrackerDS Successfully deployed the execution manager tracker service
[2017-09-16 04:59:32,688] INFO - BinaryDataReceiver Started Binary SSL Transport on port : 9711
[2017-09-16 04:59:32,690] INFO - BinaryDataReceiver Started Binary TCP Transport on port : 9611
[2017-09-16 04:59:32,709] INFO - DataBridgeDS Successfully deployed Agent Server
[2017-09-16 04:59:33,280] INFO - EmbeddedRegistryService Configured Registry in 216ms
.....
......
[2017-09-16 05:00:56,576] INFO - Andes Starting Safe Zone Calculator for slots.
[Broker] BRK-1002 : Starting : Listening on TCP port 5672
[2017-09-16 05:00:56,624] INFO - listening [Broker] BRK-1002 : Starting : Listening on TCP port 5672
[Broker] BRK-1002 : Starting : Listening on TCP/SSL port 8672
[2017-09-16 05:00:56,628] INFO - listening [Broker] BRK-1002 : Starting : Listening on TCP/SSL port 8672
[Broker] BRK-1004 : Qpid Broker Ready
[2017-09-16 05:00:56,632] INFO - MessagingEngine Starting SlotDelivery Workers.
[2017-09-16 05:00:56,635] INFO - MessagingEngine Start Disruptor writing messages to store.
.......
.......
[2017-09-16 05:00:56,699] INFO - QpidServiceComponent AMQP Host Address : 0.0.0.0 Port : 5672
[2017-09-16 05:00:56,699] INFO - QpidServiceComponent Successfully connected to AMQP server on port 5672
[2017-09-16 05:00:56,700] WARN - QpidServiceComponent MQTT Transport is disabled as per configuration.
[2017-09-16 05:00:56,700] INFO - QpidServiceComponent WSO2 Message Broker is started.
.......
[2017-09-16 05:00:58,453] INFO - ThriftDataReceiver Thrift Server started at 0.0.0.0
....
[2017-09-16 05:00:58,509] INFO - ThriftDataReceiver Thrift SSL port : 7711
.....
[2017-09-16 05:00:58,547] INFO - ThriftDataReceiver Thrift port : 7611
.......
[2017-09-16 05:00:59,011] INFO - TaskServiceImpl Task service starting in STANDALONE mode...
[2017-09-16 05:00:59,392] ERROR - SourceHandler I/O error: Received fatal alert: unknown_ca
javax.net.ssl.SSLException: Received fatal alert: unknown_ca
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
.....
.....
[2017-09-16 05:00:59,530] INFO - RegistryEventingServiceComponent Successfully Initialized Eventing on Registry
[2017-09-16 05:00:59,900] INFO - JMXServerManager JMX Service URL : service:jmx:rmi://localhost:11111/jndi/rmi://localhost:9999/jmxrmi
[2017-09-16 05:00:59,904] INFO - StartupFinalizerServiceComponent Server : WSO2 API Manager-2.1.0
[2017-09-16 05:00:59,905] INFO - StartupFinalizerServiceComponent WSO2 Carbon started in 104 sec
[2017-09-16 05:01:00,188] INFO - CarbonUIServiceComponent Mgt Console URL : https://X.X.X.X:9443/carbon/
[2017-09-16 05:01:00,188] INFO - CarbonUIServiceComponent API Publisher Default Context : https://X.X.X.X:9443/publisher
[2017-09-16 05:01:00,188] INFO - CarbonUIServiceComponent API Store Default Context : https://X.X.X.X:9443/store
[2017-09-16 05:01:00,329] WARN - JMSListener Polling tasks on destination : throttleData of type topic for listener Siddhi-JMS-Consumer#throttleData have not yet started after 3 seconds ..
[2017-09-16 05:01:02,974] ERROR - SourceHandler I/O error: Received fatal alert: unknown_ca
javax.net.ssl.SSLException: Received fatal alert: unknown_ca
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)
.....
.....

This will probably correct : #40

Thanks,
Youcef HILEM

Description:
We have not added ingress resources to key manager URLs. Need to add them in a future release.

Suggested Labels:
enhancement

Suggested Assignees:

Affected Product Version:
2.1.0, 2.1.0-nfs

OS, DB, other environment details and versions:

Steps to reproduce:

Related Issues:

Hi,

Many exceptions are thrown in PODs wso2apim-analytics-1 & wso2apim-analytics-2 are thrown.

1)____________________________________________

[2017-09-22 09:47:44,103] INFO {org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor} - Starting SPARK MASTER...
[2017-09-22 09:47:45,012] ERROR {org.wso2.carbon.analytics.spark.core.internal.AnalyticsComponent} - Error initializing analytics executor: failure to login
java.io.IOException: failure to login
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:823)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2198)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2198)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2198)
at org.apache.spark.SecurityManager.(SecurityManager.scala:214)
at org.apache.spark.deploy.master.Master$.startRpcEnvAndEndpoint(Master.scala:1121)
at org.apache.spark.deploy.master.Master.startRpcEnvAndEndpoint(Master.scala)
at org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.startMaster(SparkAnalyticsExecutor.java:424)
at org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.runClusteredSetupLogic(SparkAnalyticsExecutor.java:225)
..........
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
at com.sun.security.auth.UnixPrincipal.(UnixPrincipal.java:71)
at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:133)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:798)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2198)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2198)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2198)
at org.apache.spark.SecurityManager.(SecurityManager.scala:214)
at org.apache.spark.deploy.master.Master$.startRpcEnvAndEndpoint(Master.scala:1121)
at org.apache.spark.deploy.master.Master.startRpcEnvAndEndpoint(Master.scala)
at org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.startMaster(SparkAnalyticsExecutor.java:424)
at org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.runClusteredSetupLogic(SparkAnalyticsExecutor.java:225)
at org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.initializeSparkServer(SparkAnalyticsExecutor.java:190)
at org.wso2.carbon.analytics.spark.core.internal.AnalyticsComponent.activate(AnalyticsComponent.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
........
........
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1571)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1561)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

at javax.security.auth.login.LoginContext.invoke(LoginContext.java:856)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:798)
... 152 more

2)___________________________________________________________________________

[2017-09-22 09:50:00,110] ERROR {org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter} - Error in executing task: null
java.lang.NullPointerException
at com.hazelcast.executor.impl.ExecutorServiceProxy.submitToMember(ExecutorServiceProxy.java:298)
at org.wso2.carbon.analytics.dataservice.core.clustering.AnalyticsClusterManagerImpl.executeOneFuture(AnalyticsClusterManagerImpl.java:271)
at org.wso2.carbon.analytics.dataservice.core.clustering.AnalyticsClusterManagerImpl.executeOne(AnalyticsClusterManagerImpl.java:260)
at org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.executeQuery(SparkAnalyticsExecutor.java:715)
at org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService.executeQuery(CarbonAnalyticsProcessorService.java:201)
at org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService.executeScript(CarbonAnalyticsProcessorService.java:151)
at org.wso2.carbon.analytics.spark.core.AnalyticsTask.execute(AnalyticsTask.java:60)
at org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:67)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2017-09-22 09:50:00,118] ERROR {org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter} - Error in executing task: null
java.lang.NullPointerException
at com.hazelcast.executor.impl.ExecutorServiceProxy.submitToMember(ExecutorServiceProxy.java:298)
at org.wso2.carbon.analytics.dataservice.core.clustering.AnalyticsClusterManagerImpl.executeOneFuture(AnalyticsClusterManagerImpl.java:271)
at org.wso2.carbon.analytics.dataservice.core.clustering.AnalyticsClusterManagerImpl.executeOne(AnalyticsClusterManagerImpl.java:260)
at org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.executeQuery(SparkAnalyticsExecutor.java:715)
at org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService.executeQuery(CarbonAnalyticsProcessorService.java:201)
at org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService.executeScript(CarbonAnalyticsProcessorService.java:151)
at org.wso2.carbon.analytics.spark.core.AnalyticsTask.execute(AnalyticsTask.java:60)
at org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:67)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
....
.....

3)_________________________________________________________________________________________

[2017-09-22 09:48:47,060] ERROR {org.wso2.carbon.event.processor.manager.core.internal.HAManager} - CEP HA State syncing failed, No execution plans with name 'APIMAnalytics-APIRequestPatternChangeAnalysisMatrixBuilder' exist for tenant -1234
org.wso2.carbon.event.processor.manager.core.exception.EventManagementException: No execution plans with name 'APIMAnalytics-APIRequestPatternChangeAnalysisMatrixBuilder' exist for tenant -1234
at org.wso2.carbon.event.processor.core.internal.CarbonEventProcessorManagementService.restoreState(CarbonEventProcessorManagementService.java:79)
at org.wso2.carbon.event.processor.manager.core.internal.HAManager.syncState(HAManager.java:336)
at org.wso2.carbon.event.processor.manager.core.internal.HAManager.access$100(HAManager.java:49)
at org.wso2.carbon.event.processor.manager.core.internal.HAManager$2.run(HAManager.java:276)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

4)___________________________________________________________________________________________________
NON STOP

[2017-09-22 12:00:00,242] ERROR {org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter} - Error in executing task: null
java.lang.NullPointerException
at com.hazelcast.executor.impl.ExecutorServiceProxy.submitToMember(ExecutorServiceProxy.java:298)
at org.wso2.carbon.analytics.dataservice.core.clustering.AnalyticsClusterManagerImpl.executeOneFuture(AnalyticsClusterManagerImpl.java:271)
at org.wso2.carbon.analytics.dataservice.core.clustering.AnalyticsClusterManagerImpl.executeOne(AnalyticsClusterManagerImpl.java:260)
at org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.executeQuery(SparkAnalyticsExecutor.java:715)
at org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService.executeQuery(CarbonAnalyticsProcessorService.java:201)
at org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService.executeScript(CarbonAnalyticsProcessorService.java:151)
at org.wso2.carbon.analytics.spark.core.AnalyticsTask.execute(AnalyticsTask.java:60)
at org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:67)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Thanks
Youcef HILEM