How to Guide For Newbies: May 2017

Thursday, May 25, 2017

How to configure reverse proxy?

I'm going to discuss two different reverse proxy configurations

Use case 01:

If someone wants to configure a reverse proxy service to hide URI of the actual backend API and allow client to invoke the particular resource URI using a different resource URI the configuration would be as follow.

API URL that the user invokes
http://host:port/test/customer/v1/xyz

API URL exposed by the API Manager

http://host:port/customer/v1/xyz

Sample Nginx configuration

upstream apimworker{
      server 10.100.11.22:8280;
}

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

server{
      listen 8280;
      server_name gw.wrk.wso2.com;
      underscores_in_headers on;

       #chunked_transfer_encoding off;

location /test/customer {
       proxy_set_header X-Forwarded-Host $host;
       proxy_set_header X-Forwarded-Server $host;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_http_version 1.1;
       proxy_pass http://apimworker/customer/;
       proxy_redirect http://apimworker/customer/ http://gw.wrk.wso2.com:8280/customer/;

        }
}

In the sample Nginx configuration I have also configured load balancing configurations together with proxy redirect. See How Nginx process requests for information on request processing.

Use case 02:

In this scenario when ever the client sends a URI that matches the URI pattern defined in the first location, the particular URL is being rewritten (reconstructed) to match the actual backend URI. In the second location we have defined how actual URI begins. As the reconstructed URI matches the URI defined in the second location the request gets passes to the backend.

API URL that the user invokes

http://host:port/test/{path_parameter}/customer/search

API URL exposed by the API Manager

http://host:port/customer/search/v1

Sample Nginx configuration
server {
   listen       81;
   server_name  gateway.apim.com;

     location ~ /test/.*/customer/search.* {
     rewrite ^.*/(customer/search/)(.*)$ /$1v1/$2 last;
   }

     location  /customer/search {
     proxy_pass http://localhost:8000;
   }

Below I have listed a sample URL to identify how above configurations works.

API URL that the user invoke:
http://gateway.apim.com:81/test/1/customer/search/xyz

API URL send to the backend by the reverse proxy:
http://localhost:8000/customer/search/xyz

Monday, May 8, 2017

How to Fix "com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Data source rejected establishment of connection, message from server: "Too many connections" "

Reason for this error

This error occurs when number of concurrent connections used in the product exceeds the maximum number of connections defined in the database.

Fix

In MySQL, you can use below command to check the maximum number of concurrent connections.

mysql> show variables like 'max_connections';

+-----------------+-------+

| Variable_name | Value |

+-----------------+-------+

| max_connections | 151 |

+-----------------+-------+

1 row in set (0.00 sec)

If you want to change the maximum number of concurrent connections in the database,

Open 'my.cnf' file located in /etc/mysql folder.
Change set the number of maximum concurrent connections you need to have to 'max_connections' property.
Restart mysql.

Error

ERROR - TransactionManager Failed to start new registry transaction.

com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.

at sun.reflect.GeneratedConstructorAccessor66.newInstance(Unknown Source)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)

at com.mysql.jdbc.Util.getInstance(Util.java:387)

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:919)

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:898)

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:887)

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:862)

at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2164)

at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2089)

at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:795)

at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:44)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)

at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:400)

at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:327)

at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:278)

at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182)

at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:701)

at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:635)

at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)

at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)

at org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)

at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443)

at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.resourceExists(EmbeddedRegistry.java:631)

at org.wso2.carbon.registry.core.session.UserRegistry.resourceExistsInternal(UserRegistry.java:774)

at org.wso2.carbon.registry.core.session.UserRegistry.access$800(UserRegistry.java:61)

at org.wso2.carbon.registry.core.session.UserRegistry$9.run(UserRegistry.java:757)

at org.wso2.carbon.registry.core.session.UserRegistry$9.run(UserRegistry.java:754)

at java.security.AccessController.doPrivileged(Native Method)

at org.wso2.carbon.registry.core.session.UserRegistry.resourceExists(UserRegistry.java:754)

at org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler.get(MountHandler.java:392)

at org.wso2.carbon.registry.core.jdbc.handlers.HandlerManager.get(HandlerManager.java:2444)

at org.wso2.carbon.registry.core.jdbc.handlers.UserDefinedHandlerManager.get(UserDefinedHandlerManager.java:200)

at org.wso2.carbon.registry.core.jdbc.handlers.HandlerLifecycleManager.get(HandlerLifecycleManager.java:914)

at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:512)

at org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:203)

at org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:617)

at org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:61)

at org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:597)

at org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:594)

at java.security.AccessController.doPrivileged(Native Method)

at org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:594)

at org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:242)

at org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:210)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Data source rejected establishment of connection, message from server: "Too many connections"

at sun.reflect.GeneratedConstructorAccessor66.newInstance(Unknown Source)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)

at com.mysql.jdbc.Util.getInstance(Util.java:387)

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:919)

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:898)

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:887)

at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1038)

at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2254)

at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2105)

... 44 more

[2017-05-08 11:20:47,779] ERROR - TransactionManager Failed to start new registry transaction.

com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.

at sun.reflect.GeneratedConstructorAccessor66.newInstance(Unknown Source)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)

at com.mysql.jdbc.Util.getInstance(Util.java:387)

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:919)

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:898)

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:887)

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:862)

at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2164)

at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2089)

at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:795)

at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:44)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)

at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:400)

at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:327)

A possible reason for getting ERROR - APIUtil Error occurred while executing SubscriberKeyMgtClient. java.lang.RuntimeException: Error occurred while calling token endpoint: HTTP error code : 302 in WSO2 API manager.

Intension is this blog post to helping out to trouble shoot below error [1] on your own.

A possible cause can be:

'oauth2.war' web application located in /repository/deployment/server/webapps folder is not deployed.

[1] Error Stack Trace

[2017-05-08 11:39:22,161] ERROR - APIUtil Error occurred while executing SubscriberKeyMgtClient.

java.lang.RuntimeException: Error occurred while calling token endpoint: HTTP error code : 302

at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.getNewApplicationAccessToken(AMDefaultKeyManagerImpl.java:367)

at org.wso2.carbon.apimgt.impl.workflow.AbstractApplicationRegistrationWorkflowExecutor.dogenerateKeysForApplication(AbstractApplicationRegistrationWorkflowExecutor.java:151)

at org.wso2.carbon.apimgt.impl.workflow.AbstractApplicationRegistrationWorkflowExecutor.generateKeysForApplication(AbstractApplicationRegistrationWorkflowExecutor.java:118)

at org.wso2.carbon.apimgt.impl.workflow.ApplicationRegistrationSimpleWorkflowExecutor.complete(ApplicationRegistrationSimpleWorkflowExecutor.java:78)

at org.wso2.carbon.apimgt.impl.workflow.ApplicationRegistrationSimpleWorkflowExecutor.execute(ApplicationRegistrationSimpleWorkflowExecutor.java:54)

at org.wso2.carbon.apimgt.impl.APIConsumerImpl.requestApprovalForApplicationRegistration(APIConsumerImpl.java:2789)

at org.wso2.carbon.apimgt.impl.UserAwareAPIConsumer.requestApprovalForApplicationRegistration(UserAwareAPIConsumer.java:36)

at org.wso2.carbon.apimgt.hostobjects.APIStoreHostObject.jsFunction_getApplicationKey(APIStoreHostObject.java:385)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)

at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:386)

at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)

at org.jaggeryjs.rhino.store.modules.subscription.c2._c_anonymous_2(/store/modules/subscription/key.jag:39)

at org.jaggeryjs.rhino.store.modules.subscription.c2.call(/store/modules/subscription/key.jag)

at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430)

at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269)

at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97)

at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42)

at org.jaggeryjs.rhino.store.modules.subscription.c0._c_anonymous_10(/store/modules/subscription/module.jag:35)

at org.jaggeryjs.rhino.store.modules.subscription.c0.call(/store/modules/subscription/module.jag)

at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)

at org.jaggeryjs.rhino.store.site.blocks.subscription.subscription_add.ajax.c0._c_anonymous_1(/store/site/blocks/subscription/subscription-add/ajax/subscription-add.jag:240)

at org.jaggeryjs.rhino.store.site.blocks.subscription.subscription_add.ajax.c0.call(/store/site/blocks/subscription/subscription-add/ajax/subscription-add.jag)

at org.mozilla.javascript.optimizer.OptRuntime.call0(OptRuntime.java:23)

at org.jaggeryjs.rhino.store.site.blocks.subscription.subscription_add.ajax.c0._c_script_0(/store/site/blocks/subscription/subscription-add/ajax/subscription-add.jag:3)

at org.jaggeryjs.rhino.store.site.blocks.subscription.subscription_add.ajax.c0.call(/store/site/blocks/subscription/subscription-add/ajax/subscription-add.jag)

at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394)

at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3091)

at org.jaggeryjs.rhino.store.site.blocks.subscription.subscription_add.ajax.c0.call(/store/site/blocks/subscription/subscription-add/ajax/subscription-add.jag)

at org.jaggeryjs.rhino.store.site.blocks.subscription.subscription_add.ajax.c0.exec(/store/site/blocks/subscription/subscription-add/ajax/subscription-add.jag)

at org.jaggeryjs.scriptengine.engine.RhinoEngine.execScript(RhinoEngine.java:567)

at org.jaggeryjs.scriptengine.engine.RhinoEngine.exec(RhinoEngine.java:273)

at org.jaggeryjs.jaggery.core.manager.WebAppManager.exec(WebAppManager.java:588)

at org.jaggeryjs.jaggery.core.manager.WebAppManager.execute(WebAppManager.java:508)

at org.jaggeryjs.jaggery.core.JaggeryServlet.doPost(JaggeryServlet.java:29)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:747)

at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:485)

at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:377)

at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:337)

at org.jaggeryjs.jaggery.core.JaggeryFilter.doFilter(JaggeryFilter.java:21)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

at org.wso2.carbon.ui.filters.cache.ContentTypeBasedCachePreventionFilter.doFilter(ContentTypeBasedCachePreventionFilter.java:53)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:120)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)

at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)

at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)

at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)

at org.wso2.carbon.event.receiver.core.internal.tenantmgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:48)

at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)

at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)

at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)

at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:958)

at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452)

at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)

at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1756)

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1715)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Thread.java:745)

[2017-05-08 11:39:22,168] ERROR - ApplicationRegistrationSimpleWorkflowExecutor Error occurred when updating the status of the Application creation process

org.wso2.carbon.apimgt.api.APIManagementException: Error occurred while executing SubscriberKeyMgtClient.

at org.wso2.carbon.apimgt.impl.utils.APIUtil.handleException(APIUtil.java:1269)

at org.wso2.carbon.apimgt.impl.workflow.AbstractApplicationRegistrationWorkflowExecutor.dogenerateKeysForApplication(AbstractApplicationRegistrationWorkflowExecutor.java:155)