Closing Off the Blogger and Moving to Medium

It's being years since I started blogging in blogger and Its time to bid farewell to blogger. So from now on I'll be writing all my content on medium. You can find me at : https://medium.com/@ycrnet

Read More

How to create a Docker image with a Ballerina Program

Prerequisites : docker configured and running, curl

In this tutorial we will explore how we can create a docker container which has a Ballerina service embedded within. Ballerina supports dockerizing out of the box. Here is how you can achieve this.

Method 01 : Creating your own docker image

1. First download the ballerina tools distribution. The latest ballerina pack can be found at : https://ballerinalang.org/downloads/
2. Extract the downloaded zip and set up ballerina runtime.
3. Now create a ballerina service to be added to the docker container. You can create the following service. (You can also create a main function as well) Name the file, docker-service.bal

import ballerina.net.http;

@http:configuration {basePath:"/echo"}

service<http> echo {

   @http:resourceConfig {

       methods:["POST"],

       path:"/"

   }

   resource echo (http:Request req, http:Response resp) {

       string payload = req.getStringPayload();

       resp.setStringPayload(payload);

       resp.send();

   }

}

6. Create the docker image with the following command.

ballerina docker docker-service.bal

If the above command succeeds you will see something like below.

To verify image creation  execute the following command which will list all the images in local docker registry.

docker images

7. Start the container with the following command.

docker run -p 39165:9090 --name docker-sample -d docker-service:latest

8. The service should be up and running. You can verify this by listing all the docker processes.

docker ps

9. Invoke the service with following command.

curl -X POST http://localhost:39165/echo -d 'This is a sample message'

If everything is successful, you will see the echoed response from the ballerina server.

Method 2 : Adding your services to existing docker container

The Docker distribution for Ballerina is available on Docker Hub as ballerinalang/ballerina. To run a Ballerina package using the Ballerina Docker image, simply mount the folder containing the file to /ballerina/files folder inside the container. Following is how you can do this.

1. First pull the ballerina docker image.

docker pull ballerinalang/ballerina

2. Create a directory and copy the packages needed to be run.

mkdir -p ~/ballerina/service/

3. Add the following service to ~/ballerina/service

import ballerina.net.http;

@http:configuration {basePath:"/echo"}

service<http> echo {

   @http:resourceConfig {

       methods:["POST"],

       path:"/"

   }

   resource echo (http:Request req, http:Response resp) {

       string payload = req.getStringPayload();

       resp.setStringPayload(payload);

       resp.send();

   }

}

4. Mount the volume and start the docker container

docker run -v ~/ballerina/service:/ballerina/files -p 9090:9090 -it ballerinalang/ballerina:0.95.0

5. Now invoke the service with following curl command.

curl -X POST http://localhost:9090/echo -d 'This is a sample message'

If everything is successful, you will see the echoed response from the ballerina server

Read More

Customizing the service URI of a Proxy Service in Enterprise Integrator

In WSO2 Enterprise integrator we have proxy services. In a proxy service by default if you create a proxy service the context of the service URL will be auto generated with the name of the proxy service name, For example if you create a proxy service name "myproxy". The service URL will be like the below.

http://localhost:8290/services/myproxy

But what if you want to have a custom URI? For ecample.

http://localhost:8290/services/myproxy/idservice/1.0

Following is how you can do this.

Step 01 - Open your axis2.xml, this can be found <EI_HOME>/conf/axis2/axis2.xml

Step 02 : In the axis2xml find the phaseOrder type="InFlow" and add the following section.

Add following dispatcher to the Phaseorder,

<handler name="RequestURIBasedDispatcher"
class="org.apache.axis2.dispatchers.RequestURIBasedDispatcher">

The dispatcher should be inserted in to the In-Flow at the Dispatch phase. It should be the first handler in the Dispatch phase.

Note : In axis2.xml we have different message phases. Namely (Inflow, Outflow and FaultFlow) So make sure you have added the dispatcher to the correct phase. You need to add the dispatcher to InFlow.

<phaseOrder type="InFlow">
<!-- System pre defined phases -->
<!--
The MsgInObservation phase is used to observe messages as soon as they are
received. In this phase, we could do some things such as SOAP message tracing & keeping
track of the time at which a particular message was received


NOTE: This should be the very first phase in this flow
-->
<phase name="MsgInObservation">
<handler name="TraceMessageBuilderDispatchHandler"
class="org.apache.synapse.transport.passthru.util.TraceMessageBuilderDispatchHandler"/>
</phase>
<phase name="Validation"/>
<phase name="Transport">
<handler name="RequestURIBasedDispatcher"
class="org.apache.axis2.dispatchers.RequestURIBasedDispatcher">
<order phase="Transport"/>
</handler>
<handler name="CarbonContextConfigurator"
class="org.wso2.carbon.mediation.initializer.handler.CarbonContextConfigurator"/>
<handler name="RelaySecuirtyMessageBuilderDispatchandler"
class="org.apache.synapse.transport.passthru.util.RelaySecuirtyMessageBuilderDispatchandler"/>
<handler name="SOAPActionBasedDispatcher"
class="org.apache.axis2.dispatchers.SOAPActionBasedDispatcher">
<order phase="Transport"/>
</handler>
<!--handler name="SMTPFaultHandler"
class="org.wso2.carbon.core.transports.smtp.SMTPFaultHandler">
<order phase="Transport"/>
</handler-->
</phase>
<phase name="Addressing">
<handler name="AddressingBasedDispatcher"
class="org.apache.axis2.dispatchers.AddressingBasedDispatcher">
<order phase="Addressing"/>
</handler>
</phase>
<phase name="Security"/>
<phase name="PreDispatch">
<!--Uncomment following handler to enable logging in ESB log UI-->
<!--<handler name="TenantDomainSetter"-->
<!--class="org.wso2.carbon.utils.logging.handler.TenantDomainSetter"/>-->
</phase>
<phase name="Dispatch" class="org.apache.axis2.engine.DispatchPhase">
<handler name="CustomURIBasedDispatcher"
class="org.apache.synapse.core.axis2.CustomURIBasedDispatcher"/>
<handler name="RequestURIBasedDispatcher"
class="org.apache.axis2.dispatchers.RequestURIBasedDispatcher"/>
<handler name="SOAPActionBasedDispatcher"
class="org.apache.axis2.dispatchers.SOAPActionBasedDispatcher"/>
<handler name="RequestURIOperationDispatcher"
class="org.apache.axis2.dispatchers.RequestURIOperationDispatcher"/>
<handler name="SOAPMessageBodyBasedDispatcher"
class="org.apache.axis2.dispatchers.SOAPMessageBodyBasedDispatcher"/>


<handler name="HTTPLocationBasedDispatcher"
class="org.apache.axis2.dispatchers.HTTPLocationBasedDispatcher"/>
<handler name="MultitenantDispatcher"
class="org.wso2.carbon.tenant.dispatcher.MultitenantDispatcher"/>
<handler name="SynapseDispatcher"
class="org.apache.synapse.core.axis2.SynapseDispatcher"/>
<handler name="SynapseMustUnderstandHandler"
class="org.apache.synapse.core.axis2.SynapseMustUnderstandHandler"/>
</phase>
<!-- System pre defined phases -->
<phase name="RMPhase"/>
<phase name="OpPhase"/>
<phase name="AuthPhase"/>
<phase name="MUPhase"/>
<!-- After Postdispatch phase module author or or service author can add any phase he want -->
<phase name="OperationInPhase"/>
</phaseOrder>

Step 03 : Now restart your server and you can create the following proxy service. Note the ServiceURI parameter.

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="SparkJobServerProxy"
       transports="https,http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <log>
            <property name="=============TReached" value="=============="/>
         </log>
      </inSequence>
   </target>
   <parameter name="ServiceURI">/services/myproxy/idservice/1.0</parameter>
   <description/>
</proxy>

                                

So that is it, please drop a comment if you have any queries or need assistance.

Read More

Reading Query Parameters in Enterprise Integrator

Following is how you can read query params from your rest service when using Enterprise Integrator.

There are two ways to read your query params.

1. Assigning value to a property within a URI template. (get-property('uri.var.qtest')).

If you look at the following synapse config I'm assigning the required query param to

<api xmlns="http://ws.apache.org/ns/synapse" name="test" context="/qtest">
   <resource methods="GET" uri-template="/search?q={qtest}" outSequence="" faultSequence="">
      <inSequence>
         <log>
            <property name="QUERY PARAM IS =" expression="get-property('uri.var.qtest')"/>
         </log>
      </inSequence>
   </resource>
</api>

2. Reading as a query params. (get-property('query.param.q'))

By default query params are loaded by the EI it self, so you can directly access them without any extra configurations. Following is how you can do this.

<api xmlns="http://ws.apache.org/ns/synapse" name="test" context="/qtest">
   <resource methods="GET" uri-template="/search?q=*" outSequence="" faultSequence="">
      <inSequence>
         <log>
            <property name="QUERY PARAM IS =" expression="get-property('query.param.q')"/>
         </log>
      </inSequence>
   </resource>
</api>

Please drop a comment if you have any further queries.

Read More

How to allow Insecure/Non SSL connections to Kubernetes Master

If you need to allow insecure connections (non-SSL) to your K8S API Server, following is how you can get this done.

First Open your API Server manifest.

sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml

Now add the following properties.

    - --insecure-bind-address=0.0.0.0
    - --insecure-port=8080

The complete kube-apiserver.yaml will look like following, (This is a fraction of the yaml file)

apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
  - name: kube-apiserver
    image: quay.io/coreos/hyperkube:v1.6.1_coreos.0
    command:
    - /hyperkube
    - apiserver
    - --bind-address=0.0.0.0
    - --etcd-servers=http://192.168.57.13:2379
    - --allow-privileged=true
    - --service-cluster-ip-range=10.3.0.0/24
    - --secure-port=443
    - --insecure-bind-address=0.0.0.0
    - --insecure-port=8080
    - --advertise-address=192.168.57.12
    - --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota
    - --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem
    - --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
    - --client-ca-file=/etc/kubernetes/ssl/ca.pem
    - --service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem
    - --runtime-config=extensions/v1beta1/networkpolicies=true
    - --anonymous-auth=false

Now restart your kubelet service.

Then in the client machine export the Kubernetes Master URL

export KUBERNETES_MASTER=http://192.168.57.12:8080

And thats it now you can call your kubernetes master through a non secured channel.

Please drop a comment if you have queries.

Read More

HTTP 404 when wget Jenkins Artefacts

I was trying to wget one of Jenkins Artifacts, but was continuously getting a 404 error.

HTTP request sent, awaiting response... 404 Not Found
2017-05-19 13:12:13 ERROR 404: Not Found.

My request was as follows.

wget https://wso2.org/jenkins/job/ballerinalang/job/tools-distribution/257/org.ballerinalang.tools$ballerina-tools/artifact/org.ballerinalang.tools/ballerina-tools/0.87-SNAPSHOT/ballerina-tools-0.87-SNAPSHOT.zip

So my issue was, My URL had some special charactors. (tools$ballerina-tools) A $ character. So Bash droped this when fetching the artefact, So Jenkins was unable to find the actual resource. To solve this type of issue you can use a scape charater to skip the special character.

\$

Full Request is as Follows.

wget https://wso2.org/jenkins/job/ballerinalang/job/tools-distribution/257/org.ballerinalang.tools\$ballerina-tools/artifact/org.ballerinalang.tools/ballerina-tools/0.87-SNAPSHOT/ballerina-tools-0.87-SNAPSHOT.zip

This is one of million ways to get an 404 error, Just mentioning to help someone to save couple of hours. :)

Read More

Blocking non existing server_name in NginX

Server names are defined using the server_name directive and determine which server block is used for a given request. See also “How nginx processes a request”. They may be defined using exact names, wildcard names, or regular expressions:

NginX behaves in a way, such that,

Nginx first decides which server should process the request. Let’s start with a simple configuration where all three virtual servers listen on port *:80:

server {
    listen      80;
    server_name www.yasassri.org;
    ...
}

server {
    listen      80;
    server_nam www.yasassri.net;
    ...
}

server {
    listen      80;
    server_name www.yasassri.com;
    ...
}

In the above configuration Nginx checks only the request’s header field “Host” to determine which server the request should be routed to. If its value does not match any server name, or the request does not contain this header field at all, then Nginx will route the request to the default server for this port.

Let me elaborate this with an example, If a client sends a request to www.yasassri.org or www.yasassri.net or www.yasassri.com NginX will route the messages to the corresponding server block. (If the Host header contains the Host-name) But what if client sends a message with Host Header www.abcd.com, this message doesn't match with any server names, so it shouldn't be routed anywhere Right? No that's not what really happens, the default behavior of NginX is to route this message to the default server configuration. In the configuration above, the default server is the first one — this is Nginx’s standard default behavior. It can also be set explicitly which server should be default, with the default_server parameter in the listen directive:

server {
    listen      80 default_server;
    server_name example.net www.yasassri.net;
    ...
}

So what if you want to block all the calls that doesn't match with the defined server names? NginX doesn't provide a cofiguration for this, to achieve this you can simply add the following server blocks as a workaround. So the following will be your default server block.

server {
  listen 80 default_server;
  return 404;
}

So when ever your server name doesn't match the request will be routed to the above server block, and a 404 is sent to the client.

So that's it, please drop a comment if you have more queries.

Read More

Debugging : unable to find valid certification path to requested target

SSL can be a pain some times. Recently I was getting the following Exception continuously no-matter what ever certificate I import to the client-truststore. So it took the best out of me to debug and find-out the real issue behind this. In this post I'll explain how one can debug a SSL connection issue.

org.apache.axis2.AxisFault: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
 at org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:78)
 at org.apache.axis2.transport.http.AxisRequestEntity.writeRequest(AxisRequestEntity.java:84)
 at org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:499)
 at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114)
 at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
 at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
 at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
 at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
 at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:622)
 at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:193)
 at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:75)
 at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451)
 at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278)
 at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
 at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430)
 at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
 at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
 at org.apache.axis2.client.ServiceClient.sendReceive(ServiceClient.java:554)
 at org.apache.axis2.client.ServiceClient.sendReceive(ServiceClient.java:530)
 at SecurityClient.runSecurityClient(SecurityClient.java:99)
 at SecurityClient.main(SecurityClient.java:34)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:483)
 at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: javax.xml.stream.XMLStreamException: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at com.sun.xml.internal.stream.writers.XMLStreamWriterImpl.close(XMLStreamWriterImpl.java:378)
 at org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.close(XMLStreamWriterWrapper.java:46)
 at org.apache.axiom.om.impl.MTOMXMLStreamWriter.close(MTOMXMLStreamWriter.java:188)
 at org.apache.axiom.om.impl.dom.NodeImpl.serializeAndConsume(NodeImpl.java:844)
 at org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:74)
 ... 25 more
Caused by: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1509)
 at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1521)
 at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
 at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
 at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
 at org.apache.commons.httpclient.ChunkedOutputStream.flush(ChunkedOutputStream.java:191)
 at com.sun.xml.internal.stream.writers.UTF8OutputStreamWriter.flush(UTF8OutputStreamWriter.java:138)
 at com.sun.xml.internal.stream.writers.XMLStreamWriterImpl.close(XMLStreamWriterImpl.java:376)
 ... 29 more
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
 at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1917)
 at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:301)
 at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:295)
 at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1369)
 at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:156)
 at sun.security.ssl.Handshaker.processLoop(Handshaker.java:925)
 at sun.security.ssl.Handshaker.process_record(Handshaker.java:860)
 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1043)
 at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1343)
 at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:728)
 at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
 at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
 at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
 at org.apache.commons.httpclient.ChunkedOutputStream.flush(ChunkedOutputStream.java:191)
 at com.sun.xml.internal.stream.writers.UTF8OutputStreamWriter.flush(UTF8OutputStreamWriter.java:138)
 at com.sun.xml.internal.stream.writers.XMLStreamWriterImpl.flush(XMLStreamWriterImpl.java:397)
 at org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.flush(XMLStreamWriterWrapper.java:50)
 at org.apache.axiom.om.impl.MTOMXMLStreamWriter.flush(MTOMXMLStreamWriter.java:198)
 at org.apache.axiom.om.impl.dom.NodeImpl.serializeAndConsume(NodeImpl.java:842)
 ... 26 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
 at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
 at sun.security.validator.Validator.validate(Validator.java:260)
 at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
 at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
 at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
 at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1351)
 ... 41 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:145)
 at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131)
 at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
 at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
 ... 47 more
org.apache.axis2.AxisFault: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
 at org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:78)
 at org.apache.axis2.transport.http.AxisRequestEntity.writeRequest(AxisRequestEntity.java:84)
 at org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:499)
 at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114)
 at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
 at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
 at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
 at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
 at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:622)
 at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:193)
 at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:75)
 at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451)
 at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278)
 at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
 at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430)
 at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
 at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
 at org.apache.axis2.client.ServiceClient.sendReceive(ServiceClient.java:554)
 at org.apache.axis2.client.ServiceClient.sendReceive(ServiceClient.java:530)
 at SecurityClient.runSecurityClient(SecurityClient.java:99)
 at SecurityClient.main(SecurityClient.java:34)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:483)
 at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: javax.xml.stream.XMLStreamException: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at com.sun.xml.internal.stream.writers.XMLStreamWriterImpl.close(XMLStreamWriterImpl.java:378)
 at org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.close(XMLStreamWriterWrapper.java:46)
 at org.apache.axiom.om.impl.MTOMXMLStreamWriter.close(MTOMXMLStreamWriter.java:188)
 at org.apache.axiom.om.impl.dom.NodeImpl.serializeAndConsume(NodeImpl.java:844)
 at org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:74)
 ... 25 more
Caused by: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1509)
 at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1521)
 at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
 at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
 at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
 at org.apache.commons.httpclient.ChunkedOutputStream.flush(ChunkedOutputStream.java:191)
 at com.sun.xml.internal.stream.writers.UTF8OutputStreamWriter.flush(UTF8OutputStreamWriter.java:138)
 at com.sun.xml.internal.stream.writers.XMLStreamWriterImpl.close(XMLStreamWriterImpl.java:376)
 ... 29 more
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
 at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1917)
 at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:301)
 at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:295)
 at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1369)
 at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:156)
 at sun.security.ssl.Handshaker.processLoop(Handshaker.java:925)
 at sun.security.ssl.Handshaker.process_record(Handshaker.java:860)
 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1043)
 at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1343)
 at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:728)
 at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
 at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
 at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
 at org.apache.commons.httpclient.ChunkedOutputStream.flush(ChunkedOutputStream.java:191)
 at com.sun.xml.internal.stream.writers.UTF8OutputStreamWriter.flush(UTF8OutputStreamWriter.java:138)
 at com.sun.xml.internal.stream.writers.XMLStreamWriterImpl.flush(XMLStreamWriterImpl.java:397)
 at org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.flush(XMLStreamWriterWrapper.java:50)
 at org.apache.axiom.om.impl.MTOMXMLStreamWriter.flush(MTOMXMLStreamWriter.java:198)
 at org.apache.axiom.om.impl.dom.NodeImpl.serializeAndConsume(NodeImpl.java:842)
 ... 26 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
 at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
 at sun.security.validator.Validator.validate(Validator.java:260)
 at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
 at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
 at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
 at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1351)
 ... 41 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:145)
 at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131)
 at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
 at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
 ... 47 more
Exception in thread "main" java.lang.NullPointerException
 at SecurityClient.main(SecurityClient.java:38)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:483)
 at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

I'm assuming that you have parsed the certificate importing step which is the most common cause for this issue. You simply need to import the servers public certificate to the Java clients trust-store. To import a certificate you can use the following keytool commnad.

keytool -import -v -alias wso2 -file nginx.crt -keystore client-truststore.jks -storepass wso2carbon

Its important to know when the client is making  a SSL Connection what happens.
Following image depicts the SSL handshake process.

If you haven't enabled Mutual SSL the step 4 will be skipped in SSL handshake. When the server receives a client hello the server will reply with the servers public certificate and the client will validate whether this certificate is available in the clients trust-store to make sure the client is talking with the actual server. (To avoid Man in the Middle attack). This is where the above error will be thrown. If the client is not able to find the servers certificate in the trust-store it will break the handshake and will start complaining.


So How can we debug this issue. First let make sure that your trust-store has the actual certificate. To do that you can list all the ertificates in the client-trust store.

#If you do not know the alias

keytool -list -v -keystore keystore.jks

#If you know the alias

keytool -list -v -keystore keystore.jks -alias abc.com

If the certificate is not available we need to import the certificate. Also makesure you don't have multiple certificates with same CN  (Common Name) if you are using wildcard certificates.

So what if you have the certificate but you are still getting this issue. So lets make sure that the Server or Load Balancer is sending the correct certificate. In my case I have a NginX server running and my client is connecting through NginX.

To check the servers certificate you can use the openssl client. Simply execute the following in your terminlal.

openssl s_client -connect wso2.com:443

If everything is working correctly your certificates CN should match the servers Host name.

[yasassri@yasassri-device wso2esb-analytics-5.0.0]$ openssl s_client -connect wso2.com:443
CONNECTED(00000003)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 High Assurance Server CA
verify return:1
depth=0 C = US, ST = California, L = Palo Alto, O = "WSO2, Inc.", CN = *.wso2.com
verify return:1
---
Certificate chain
 0 s:/C=US/ST=California/L=Palo Alto/O=WSO2, Inc./CN=*.wso2.com
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
 1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFSTCCBDGgAwIBAgIQB1fk8mjmJAD836dv4rBT7zANBgkqhkiG9w0BAQsFADBw
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
d3cuZGlnaWNlcnQuY29tMS8wLQYDVQQDEyZEaWdpQ2VydCBTSEEyIEhpZ2ggQXNz
dXJhbmNlIFNlcnZlciBDQTAeFw0xNTEwMjYwMDAwMDBaFw0xODEwMjkxMjAwMDBa
MGAxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRIwEAYDVQQHEwlQ
YWxvIEFsdG8xEzARBgNVBAoTCldTTzIsIEluYy4xEzARBgNVBAMMCioud3NvMi5j
b20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDmRnXn8ez+xcD0f+x1
BF76v0SlKLb1KxjXTWZ9IPwUa9H6XxNbbIymxgFPrPitzL+JH6o90JW+BNqm1+Wk
MEhvDakuShA462vrrKKlj0S+wSecT/rbCJ/hZ9a5T8hRhLv75H8+7Kq3BYmPOryC
lalisdsvCM9yMzXxFmyCC2DHIvm4yhYl6jsuNirkw5WF6ep12ywPbRcKjU3YMBrG
khNtbIJLbHaR+JiziR3WlXR2R8nEmdeHs98p8YTVJH52ohCNrIEjHuDdOCE0nLg/
ZZqmO5PUKF3RE5s3Nqmoe7FFps3uDghdwhtqHQ4xsPAAZDflcpyov6dnjPDifa7P
K8S9AgMBAAGjggHtMIIB6TAfBgNVHSMEGDAWgBRRaP+QrwIHdTzM2WVkYqISuFly
OzAdBgNVHQ4EFgQUCobs4BBRc7f2I1GLS6XIOthCR+AwHwYDVR0RBBgwFoIKKi53
c28yLmNvbYIId3NvMi5jb20wDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsG
AQUFBwMBBggrBgEFBQcDAjB1BgNVHR8EbjBsMDSgMqAwhi5odHRwOi8vY3JsMy5k
aWdpY2VydC5jb20vc2hhMi1oYS1zZXJ2ZXItZzQuY3JsMDSgMqAwhi5odHRwOi8v
Y3JsNC5kaWdpY2VydC5jb20vc2hhMi1oYS1zZXJ2ZXItZzQuY3JsMEwGA1UdIARF
MEMwNwYJYIZIAYb9bAEBMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8vd3d3LmRpZ2lj
ZXJ0LmNvbS9DUFMwCAYGZ4EMAQICMIGDBggrBgEFBQcBAQR3MHUwJAYIKwYBBQUH
MAGGGGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBNBggrBgEFBQcwAoZBaHR0cDov
L2NhY2VydHMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0U0hBMkhpZ2hBc3N1cmFuY2VT
ZXJ2ZXJDQS5jcnQwDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQsFAAOCAQEAgx6w
WDDP3AMZ4Ez5TB/Tu57hVmaDZlMB+chV89u4ns426iQKIf82CBJ880R/R9adxfNn
kBuNF0mwF7BCzgp7R62L0PqLWB0cO7ExhixIPdXceH3T1x2Jsjnv+BiyO+HFdNbP
fhdbTmaEKehjWUwIA36QGi8AdG3FXEr1ijlilj3dYfgfm7qLAQIUEcf9ww12eeR3
far103txuZn3P5Lsc6aV8SZdMrlsdceCn+2EsK+Vf7PJBWfUkeXH3KGdXAlTHxSY
IodGC5B2ACFW2C2H69t4Ec+9FrFLPV8rWXxmBO+44t+opCHvqpZ3yBgFPhncE2Fy
ju9e8Gag5kRWanNQMw==
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Palo Alto/O=WSO2, Inc./CN=*.wso2.com
issuer=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3240 bytes and written 327 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 43BD18F9F2D84C05ECFF44189DBFA7E94D3FB569EDBABB79864BCE5E715698E3
    Session-ID-ctx: 
    Master-Key: 23934BED53F879565B01055F9C9FA98CF8DFA8E8E4F1C5FD07C5630D4A68C60CC7B3D15D2AC5E3DEFED7DC0A442BBEEC
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 71 59 c8 ea 79 a8 4e 76-65 1f ed ca 8d 71 3f d3   qY..y.Nve....q?.
    0010 - f7 cd 68 b8 03 75 6d b2-73 66 e1 90 2c 22 92 fd   ..h..um.sf..,"..
    0020 - 19 7d 98 c5 0a bb 82 b1-b0 84 3b 37 c0 72 57 c3   .}........;7.rW.
    0030 - c0 e1 9d d2 bf 7d 7d 8f-ce 3e af 5d 13 4d b9 c2   .....}}..>.].M..
    0040 - bd e0 8f c9 1a 58 d3 48-8e 04 96 5c c0 50 3a a6   .....X.H...\.P:.
    0050 - bc 74 18 89 95 49 e6 d9-7d 5d 7d 1a 0b 77 56 7b   .t...I..}]}..wV{
    0060 - f5 2b 87 6c af 4a 3d 16-61 a8 f9 b5 46 e6 c2 9f   .+.l.J=.a...F...
    0070 - cb 4f 11 52 d9 30 ea 62-d3 31 49 0e 8f 32 6b 58   .O.R.0.b.1I..2kX
    0080 - 9f 45 ab db 71 7b 29 7e-24 1d 0f d8 fa 67 59 39   .E..q{)~$....gY9
    0090 - 6f f3 23 1b 43 64 c9 45-c8 7f b7 33 2e 01 e8 0a   o.#.Cd.E...3....
    00a0 - f5 85 79 64 69 b9 3c af-33 63 26 2f 36 a2 5b 63   ..ydi.<.3c&/6.[c

    Start Time: 1484740335
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
closed

What if your certificate is different????? Why and How? In my case I had a similar issue, my NginX server was sending me the wrong certificate. After debuging a lot it turn out, that my client is using SSLv2. So let me explain this further.

In my NginX configurations I have configured multiple certificates for multiple servers. So I figured-out that the NginX sending me the certificate of a different server. So Why?  It turns out in older days it was not possible to add multiple certificates to same IP+PORT. In the SSL handshake level there is no way for the server to know whether you are calling foo.com or bar.com. But in later iterrations in SSL, in TLS 1.2+ there is a concept called SNI(Server Name Identifier) with SNI the client can send the servers hostname at the SSL handshake level. So since my client was using SSLv2, NginX didn't have a clue to send the correct certificate so it randomly sends the certificate which matches first. In my case it was done in alphabetical order.

So the correct fix for this is to use later SSL protocols like TLS. Or you can simply move different servers to different ports in NginX so nginX will always have a single certificate to deal with. Aother workaround is to import all the certificates to client-truststore.

In my case I moves some servers to different ports in NginX since I didn't have any control over the clients. So how can I use SNI when connecting with openssl client. You can simply use the following command for this.

openssl s_client -servername wso2.com -connect wso2.com:443

So hope this will help someone. Drop a comment if you have any queries.

Read More