Debugging : unable to find valid certification path to requested target

SSL can be a pain some times. Recently I was getting the following Exception continuously no-matter what ever certificate I import to the client-truststore. So it took the best out of me to debug and find-out the real issue behind this. In this post I'll explain how one can debug a SSL connection issue.

org.apache.axis2.AxisFault: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
 at org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:78)
 at org.apache.axis2.transport.http.AxisRequestEntity.writeRequest(AxisRequestEntity.java:84)
 at org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:499)
 at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114)
 at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
 at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
 at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
 at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
 at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:622)
 at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:193)
 at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:75)
 at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451)
 at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278)
 at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
 at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430)
 at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
 at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
 at org.apache.axis2.client.ServiceClient.sendReceive(ServiceClient.java:554)
 at org.apache.axis2.client.ServiceClient.sendReceive(ServiceClient.java:530)
 at SecurityClient.runSecurityClient(SecurityClient.java:99)
 at SecurityClient.main(SecurityClient.java:34)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:483)
 at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: javax.xml.stream.XMLStreamException: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at com.sun.xml.internal.stream.writers.XMLStreamWriterImpl.close(XMLStreamWriterImpl.java:378)
 at org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.close(XMLStreamWriterWrapper.java:46)
 at org.apache.axiom.om.impl.MTOMXMLStreamWriter.close(MTOMXMLStreamWriter.java:188)
 at org.apache.axiom.om.impl.dom.NodeImpl.serializeAndConsume(NodeImpl.java:844)
 at org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:74)
 ... 25 more
Caused by: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1509)
 at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1521)
 at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
 at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
 at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
 at org.apache.commons.httpclient.ChunkedOutputStream.flush(ChunkedOutputStream.java:191)
 at com.sun.xml.internal.stream.writers.UTF8OutputStreamWriter.flush(UTF8OutputStreamWriter.java:138)
 at com.sun.xml.internal.stream.writers.XMLStreamWriterImpl.close(XMLStreamWriterImpl.java:376)
 ... 29 more
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
 at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1917)
 at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:301)
 at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:295)
 at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1369)
 at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:156)
 at sun.security.ssl.Handshaker.processLoop(Handshaker.java:925)
 at sun.security.ssl.Handshaker.process_record(Handshaker.java:860)
 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1043)
 at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1343)
 at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:728)
 at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
 at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
 at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
 at org.apache.commons.httpclient.ChunkedOutputStream.flush(ChunkedOutputStream.java:191)
 at com.sun.xml.internal.stream.writers.UTF8OutputStreamWriter.flush(UTF8OutputStreamWriter.java:138)
 at com.sun.xml.internal.stream.writers.XMLStreamWriterImpl.flush(XMLStreamWriterImpl.java:397)
 at org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.flush(XMLStreamWriterWrapper.java:50)
 at org.apache.axiom.om.impl.MTOMXMLStreamWriter.flush(MTOMXMLStreamWriter.java:198)
 at org.apache.axiom.om.impl.dom.NodeImpl.serializeAndConsume(NodeImpl.java:842)
 ... 26 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
 at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
 at sun.security.validator.Validator.validate(Validator.java:260)
 at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
 at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
 at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
 at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1351)
 ... 41 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:145)
 at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131)
 at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
 at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
 ... 47 more
org.apache.axis2.AxisFault: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
 at org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:78)
 at org.apache.axis2.transport.http.AxisRequestEntity.writeRequest(AxisRequestEntity.java:84)
 at org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:499)
 at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114)
 at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
 at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
 at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
 at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
 at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:622)
 at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:193)
 at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:75)
 at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451)
 at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278)
 at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
 at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430)
 at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
 at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
 at org.apache.axis2.client.ServiceClient.sendReceive(ServiceClient.java:554)
 at org.apache.axis2.client.ServiceClient.sendReceive(ServiceClient.java:530)
 at SecurityClient.runSecurityClient(SecurityClient.java:99)
 at SecurityClient.main(SecurityClient.java:34)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:483)
 at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: javax.xml.stream.XMLStreamException: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at com.sun.xml.internal.stream.writers.XMLStreamWriterImpl.close(XMLStreamWriterImpl.java:378)
 at org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.close(XMLStreamWriterWrapper.java:46)
 at org.apache.axiom.om.impl.MTOMXMLStreamWriter.close(MTOMXMLStreamWriter.java:188)
 at org.apache.axiom.om.impl.dom.NodeImpl.serializeAndConsume(NodeImpl.java:844)
 at org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:74)
 ... 25 more
Caused by: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1509)
 at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1521)
 at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
 at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
 at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
 at org.apache.commons.httpclient.ChunkedOutputStream.flush(ChunkedOutputStream.java:191)
 at com.sun.xml.internal.stream.writers.UTF8OutputStreamWriter.flush(UTF8OutputStreamWriter.java:138)
 at com.sun.xml.internal.stream.writers.XMLStreamWriterImpl.close(XMLStreamWriterImpl.java:376)
 ... 29 more
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
 at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1917)
 at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:301)
 at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:295)
 at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1369)
 at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:156)
 at sun.security.ssl.Handshaker.processLoop(Handshaker.java:925)
 at sun.security.ssl.Handshaker.process_record(Handshaker.java:860)
 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1043)
 at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1343)
 at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:728)
 at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
 at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
 at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
 at org.apache.commons.httpclient.ChunkedOutputStream.flush(ChunkedOutputStream.java:191)
 at com.sun.xml.internal.stream.writers.UTF8OutputStreamWriter.flush(UTF8OutputStreamWriter.java:138)
 at com.sun.xml.internal.stream.writers.XMLStreamWriterImpl.flush(XMLStreamWriterImpl.java:397)
 at org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.flush(XMLStreamWriterWrapper.java:50)
 at org.apache.axiom.om.impl.MTOMXMLStreamWriter.flush(MTOMXMLStreamWriter.java:198)
 at org.apache.axiom.om.impl.dom.NodeImpl.serializeAndConsume(NodeImpl.java:842)
 ... 26 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
 at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
 at sun.security.validator.Validator.validate(Validator.java:260)
 at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
 at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
 at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
 at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1351)
 ... 41 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:145)
 at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131)
 at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
 at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
 ... 47 more
Exception in thread "main" java.lang.NullPointerException
 at SecurityClient.main(SecurityClient.java:38)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:483)
 at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

I'm assuming that you have parsed the certificate importing step which is the most common cause for this issue. You simply need to import the servers public certificate to the Java clients trust-store. To import a certificate you can use the following keytool commnad.

keytool -import -v -alias wso2 -file nginx.crt -keystore client-truststore.jks -storepass wso2carbon

Its important to know when the client is making  a SSL Connection what happens.
Following image depicts the SSL handshake process.

If you haven't enabled Mutual SSL the step 4 will be skipped in SSL handshake. When the server receives a client hello the server will reply with the servers public certificate and the client will validate whether this certificate is available in the clients trust-store to make sure the client is talking with the actual server. (To avoid Man in the Middle attack). This is where the above error will be thrown. If the client is not able to find the servers certificate in the trust-store it will break the handshake and will start complaining.


So How can we debug this issue. First let make sure that your trust-store has the actual certificate. To do that you can list all the ertificates in the client-trust store.

#If you do not know the alias

keytool -list -v -keystore keystore.jks

#If you know the alias

keytool -list -v -keystore keystore.jks -alias abc.com

If the certificate is not available we need to import the certificate. Also makesure you don't have multiple certificates with same CN  (Common Name) if you are using wildcard certificates.

So what if you have the certificate but you are still getting this issue. So lets make sure that the Server or Load Balancer is sending the correct certificate. In my case I have a NginX server running and my client is connecting through NginX.

To check the servers certificate you can use the openssl client. Simply execute the following in your terminlal.

openssl s_client -connect wso2.com:443

If everything is working correctly your certificates CN should match the servers Host name.

[yasassri@yasassri-device wso2esb-analytics-5.0.0]$ openssl s_client -connect wso2.com:443
CONNECTED(00000003)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 High Assurance Server CA
verify return:1
depth=0 C = US, ST = California, L = Palo Alto, O = "WSO2, Inc.", CN = *.wso2.com
verify return:1
---
Certificate chain
 0 s:/C=US/ST=California/L=Palo Alto/O=WSO2, Inc./CN=*.wso2.com
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
 1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFSTCCBDGgAwIBAgIQB1fk8mjmJAD836dv4rBT7zANBgkqhkiG9w0BAQsFADBw
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
d3cuZGlnaWNlcnQuY29tMS8wLQYDVQQDEyZEaWdpQ2VydCBTSEEyIEhpZ2ggQXNz
dXJhbmNlIFNlcnZlciBDQTAeFw0xNTEwMjYwMDAwMDBaFw0xODEwMjkxMjAwMDBa
MGAxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRIwEAYDVQQHEwlQ
YWxvIEFsdG8xEzARBgNVBAoTCldTTzIsIEluYy4xEzARBgNVBAMMCioud3NvMi5j
b20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDmRnXn8ez+xcD0f+x1
BF76v0SlKLb1KxjXTWZ9IPwUa9H6XxNbbIymxgFPrPitzL+JH6o90JW+BNqm1+Wk
MEhvDakuShA462vrrKKlj0S+wSecT/rbCJ/hZ9a5T8hRhLv75H8+7Kq3BYmPOryC
lalisdsvCM9yMzXxFmyCC2DHIvm4yhYl6jsuNirkw5WF6ep12ywPbRcKjU3YMBrG
khNtbIJLbHaR+JiziR3WlXR2R8nEmdeHs98p8YTVJH52ohCNrIEjHuDdOCE0nLg/
ZZqmO5PUKF3RE5s3Nqmoe7FFps3uDghdwhtqHQ4xsPAAZDflcpyov6dnjPDifa7P
K8S9AgMBAAGjggHtMIIB6TAfBgNVHSMEGDAWgBRRaP+QrwIHdTzM2WVkYqISuFly
OzAdBgNVHQ4EFgQUCobs4BBRc7f2I1GLS6XIOthCR+AwHwYDVR0RBBgwFoIKKi53
c28yLmNvbYIId3NvMi5jb20wDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsG
AQUFBwMBBggrBgEFBQcDAjB1BgNVHR8EbjBsMDSgMqAwhi5odHRwOi8vY3JsMy5k
aWdpY2VydC5jb20vc2hhMi1oYS1zZXJ2ZXItZzQuY3JsMDSgMqAwhi5odHRwOi8v
Y3JsNC5kaWdpY2VydC5jb20vc2hhMi1oYS1zZXJ2ZXItZzQuY3JsMEwGA1UdIARF
MEMwNwYJYIZIAYb9bAEBMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8vd3d3LmRpZ2lj
ZXJ0LmNvbS9DUFMwCAYGZ4EMAQICMIGDBggrBgEFBQcBAQR3MHUwJAYIKwYBBQUH
MAGGGGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBNBggrBgEFBQcwAoZBaHR0cDov
L2NhY2VydHMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0U0hBMkhpZ2hBc3N1cmFuY2VT
ZXJ2ZXJDQS5jcnQwDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQsFAAOCAQEAgx6w
WDDP3AMZ4Ez5TB/Tu57hVmaDZlMB+chV89u4ns426iQKIf82CBJ880R/R9adxfNn
kBuNF0mwF7BCzgp7R62L0PqLWB0cO7ExhixIPdXceH3T1x2Jsjnv+BiyO+HFdNbP
fhdbTmaEKehjWUwIA36QGi8AdG3FXEr1ijlilj3dYfgfm7qLAQIUEcf9ww12eeR3
far103txuZn3P5Lsc6aV8SZdMrlsdceCn+2EsK+Vf7PJBWfUkeXH3KGdXAlTHxSY
IodGC5B2ACFW2C2H69t4Ec+9FrFLPV8rWXxmBO+44t+opCHvqpZ3yBgFPhncE2Fy
ju9e8Gag5kRWanNQMw==
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Palo Alto/O=WSO2, Inc./CN=*.wso2.com
issuer=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3240 bytes and written 327 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 43BD18F9F2D84C05ECFF44189DBFA7E94D3FB569EDBABB79864BCE5E715698E3
    Session-ID-ctx: 
    Master-Key: 23934BED53F879565B01055F9C9FA98CF8DFA8E8E4F1C5FD07C5630D4A68C60CC7B3D15D2AC5E3DEFED7DC0A442BBEEC
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 71 59 c8 ea 79 a8 4e 76-65 1f ed ca 8d 71 3f d3   qY..y.Nve....q?.
    0010 - f7 cd 68 b8 03 75 6d b2-73 66 e1 90 2c 22 92 fd   ..h..um.sf..,"..
    0020 - 19 7d 98 c5 0a bb 82 b1-b0 84 3b 37 c0 72 57 c3   .}........;7.rW.
    0030 - c0 e1 9d d2 bf 7d 7d 8f-ce 3e af 5d 13 4d b9 c2   .....}}..>.].M..
    0040 - bd e0 8f c9 1a 58 d3 48-8e 04 96 5c c0 50 3a a6   .....X.H...\.P:.
    0050 - bc 74 18 89 95 49 e6 d9-7d 5d 7d 1a 0b 77 56 7b   .t...I..}]}..wV{
    0060 - f5 2b 87 6c af 4a 3d 16-61 a8 f9 b5 46 e6 c2 9f   .+.l.J=.a...F...
    0070 - cb 4f 11 52 d9 30 ea 62-d3 31 49 0e 8f 32 6b 58   .O.R.0.b.1I..2kX
    0080 - 9f 45 ab db 71 7b 29 7e-24 1d 0f d8 fa 67 59 39   .E..q{)~$....gY9
    0090 - 6f f3 23 1b 43 64 c9 45-c8 7f b7 33 2e 01 e8 0a   o.#.Cd.E...3....
    00a0 - f5 85 79 64 69 b9 3c af-33 63 26 2f 36 a2 5b 63   ..ydi.<.3c&/6.[c

    Start Time: 1484740335
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
closed

What if your certificate is different????? Why and How? In my case I had a similar issue, my NginX server was sending me the wrong certificate. After debuging a lot it turn out, that my client is using SSLv2. So let me explain this further.

In my NginX configurations I have configured multiple certificates for multiple servers. So I figured-out that the NginX sending me the certificate of a different server. So Why?  It turns out in older days it was not possible to add multiple certificates to same IP+PORT. In the SSL handshake level there is no way for the server to know whether you are calling foo.com or bar.com. But in later iterrations in SSL, in TLS 1.2+ there is a concept called SNI(Server Name Identifier) with SNI the client can send the servers hostname at the SSL handshake level. So since my client was using SSLv2, NginX didn't have a clue to send the correct certificate so it randomly sends the certificate which matches first. In my case it was done in alphabetical order.

So the correct fix for this is to use later SSL protocols like TLS. Or you can simply move different servers to different ports in NginX so nginX will always have a single certificate to deal with. Aother workaround is to import all the certificates to client-truststore.

In my case I moves some servers to different ports in NginX since I didn't have any control over the clients. So how can I use SNI when connecting with openssl client. You can simply use the following command for this.

openssl s_client -servername wso2.com -connect wso2.com:443

So hope this will help someone. Drop a comment if you have any queries.

Read More

Securing MySQL and Connecting WSO2 Servers

Settingup MYSQL

Generating the Keys and Signing them

Execute following commands to generate necessary keys and to sign them.

openssl genrsa 2048 > ca-key.pem
openssl req -new -x509 -nodes -days 3600 -key ca-key.pem -out ca.pem
openssl req -newkey rsa:2048 -days 3600 -nodes -keyout server-key.pem -out server-req.pem
openssl rsa -in server-key.pem -out server-key.pem
openssl x509 -req -in server-req.pem -days 3600 -CA ca.pem -CAkey ca-key.pem -set_serial 01 -out server-cert.pem
openssl req -newkey rsa:2048 -days 3600 -nodes -keyout client-key.pem -out client-req.pem
openssl rsa -in client-key.pem -out client-key.pem
openssl x509 -req -in client-req.pem -days 3600 -CA ca.pem -CAkey ca-key.pem -set_serial 01 -out client-cert.pem

Now open my.cnf and add the following configurations. Its located at /etc/mysql/my.cnf in Ubuntu.

[mysqld]
ssl-ca=/etc/mysql/ca.pem
ssl-cert=/etc/mysql/server-cert.pem
ssl-key=/etc/mysql/server-key.pem

An sample my.cnf would look like following.

Now restart mysql server.  You can use the following command to do this.

sudo service mysql restart

Now to check whether SSL certificates are properly set. Login to MySQL and execute the following query.

SHOW VARIABLES LIKE '%ssl%';

Above will give the below output.

+---------------+----------------------------+

| Variable_name | Value                      |

+---------------+----------------------------+

| have_openssl     | YES                                 |

| have_ssl             | YES                                  |

| ssl_ca                 | /etc/mysql/ca.pem         |

| ssl_capath         |                            |

| ssl_cert             | /etc/mysql/server-cert.pem |

| ssl_cipher         |                            |

| ssl_crl               |                                |

| ssl_crlpath        |                            |

| ssl_key              | /etc/mysql/server-key.pem  |

+---------------+----------------------------+

Now MYSQL configurations are done. Now lets configure WSO2 products to connect to MYSQL via SSL.

Connecting WSO2 Products to secured MySQL Server

1. First, we need to import client and server certificates to the client-truststore of WSO2 server. You can do this with following commands. (The certificates we created when configuring MySQL)

keytool -import -alias wso2qamysqlclient -file  /etc/mysql-ssl/server-cert.pem -keystore repository/resources/security/client-truststore.jks

keytool -import -alias wso2qamysqlserver -file  /etc/mysql-ssl/client-cert.pem -keystore repository/resources/security/client-truststore.jks

2. Now specify the SSL parameters in the connection URL. Make sure you specify both options useSSL and requireSSL.

jdbc:mysql://192.168.48.98:3306/ds21_carbon?autoReconnect=true&useSSL=true&requireSSL=true

The Full datasource will look like following.

<configuration>
    <url>jdbc:mysql://192.168.48.98:3306/ds21_carbon?autoReconnect=true&amp;useSSL=true&amp;requireSSL=true</url>
    <username>root</username>
    <defaultAutoCommit>false</defaultAutoCommit>
    <password>root</password>
    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
    <maxActive>80</maxActive>
    <maxWait>60000</maxWait>
    <minIdle>5</minIdle>
    <testOnBorrow>true</testOnBorrow>
    <validationQuery>SELECT 1</validationQuery>
    <validationInterval>30000</validationInterval>
</configuration>

3. Now you can start the server. If everything is set properly, the server should start without errors.

Read More

How to get rid of handshake alert: unrecognized_name

When I was trying to call a service via HTTPS I observed the below issue.

FATAL: Error verifying developer key: Failed to read server's response: handshake alert:  unrecognized_name
br.eti.kinoshita.testlinkjavaapi.util.TestLinkAPIException: Error verifying developer key: Failed to read server's response: handshake alert:  unrecognized_name
 at br.eti.kinoshita.testlinkjavaapi.MiscService.checkDevKey(MiscService.java:63)
 at br.eti.kinoshita.testlinkjavaapi.TestLinkAPI.<init>(TestLinkAPI.java:144)
 at hudson.plugins.testlink.TestLinkBuilder.getTestLinkSite(TestLinkBuilder.java:318)
 at hudson.plugins.testlink.TestLinkBuilder.perform(TestLinkBuilder.java:197)
 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:782)
 at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.build(MavenModuleSetBuild.java:906)
 at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:857)
 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
 at hudson.model.Run.execute(Run.java:1738)
 at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:529)
 at hudson.model.ResourceController.execute(ResourceController.java:98)
 at hudson.model.Executor.run(Executor.java:410)
Caused by: org.apache.xmlrpc.XmlRpcException: Failed to read server's response: handshake alert:  unrecognized_name
 at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:161)
 at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143)
 at org.apache.xmlrpc.client.XmlRpcSunHttpTransport.sendRequest(XmlRpcSunHttpTransport.java:69)
 at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56)
 at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167)
 at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:158)
 at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:147)
 at br.eti.kinoshita.testlinkjavaapi.BaseService.executeXmlRpcCall(BaseService.java:90)
 at br.eti.kinoshita.testlinkjavaapi.MiscService.checkDevKey(MiscService.java:60)
 ... 12 more
Caused by: javax.net.ssl.SSLProtocolException: handshake alert:  unrecognized_name
 at sun.security.ssl.ClientHandshaker.handshakeAlert(ClientHandshaker.java:1292)
 at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1952)
 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1077)
 at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
 at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
 at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
 at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563)
 at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
 at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091)
 at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
 at org.apache.xmlrpc.client.XmlRpcSunHttpTransport.writeRequest(XmlRpcSunHttpTransport.java:104)
 at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151)
 ... 20 more
ERROR: Error communicating with TestLink. Check your TestLink configuration.
Finished: FAILURE

The reason for this issue is the server sends an unrecognized host in the time of handshake. And java client fails due to this. Most other clients will ignore this alert and proceed but from JAVA 7 this doesn't work like that.

This can happen due to misconfigurations in the servers. If you come across the above issue there are several ways to get this resolved. You can use the method 1 if you do not have accesses to the remote server.

1. By disabling SNI verification.

This is one of the workarounds you can follow, you can simply set the following JVM property at the client side.

-Djsse.enableSNIExtension=false

e.g : In the below example I have set this property to TomCat run-time. So SNI will be disabled globally. You can do this in class level as well.

export CATALINA_OPTS="-Djsse.enableSNIExtension=false"

2. By modifying Apche2 configurations. 

In Apache2 server configurations make sure you have set the following parameters

ServerName testlinkstaging.wso2.com
ServerAlias testlinkstaging.wso2.com

Full configs will look like following.

<VirtualHost testlinkstaging.wso2.com:443>
    ServerName testlinkstaging.wso2.com
    ServerAlias testlinkstaging.wso2.com
    SSLEngine on
    SSLCertificateFile  /etc/apache2/ssl/apache.crt
    SSLCertificateKeyFile /etc/apache2/ssl/apache.key

    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/html/testlink

</VirtualHost>

That's it, If everything is set properly you should be able to access the SSL endpoint.

Please drop a comment if you have any queries.

Read More

How to Setup Simple SVN server, Expose it via DAV with Apache2 Over SSL in Ubuntu

In this Post I will explain you how to install subversion and expose it via Apache2 server while enabling SSL. So I have blogged on how to install Apache2 and enable SSL here. So As a prerequisite follow the above blog and install Apache2 server and configure SSL.

So lets install and configure subversion with Apache2.

Installing Subversion

First update Ubuntu repositories

sudo apt-get update
sudo apt-get upgrade

Now install Subversion

sudo apt-get install subversion

After installing you can verify the installed version by executing the following command

svn --version

Now create a directory and navigate to that directory

mkdir ~/svn
cd ~/svn

By executing following create a new SVN repository

svnadmin create svnqarepo

Now changer ownership of the directory

1	sudo chown -R www-data:www-data *

change permission of the directory

chmod 755 -R *

Configure Apache2 with SVN

First lets install SVN related packages to Apache2

sudo apt-get install libapache2-svn

Install extra Packages that are required

sudo apt-get install apache2-utils

Note: apache2-utills are required to htpasswd when adding users

Now Open /etc/apache2/mods-available/dav_svn.conf and add the following

1 2 3 4 5 6 7 8

<Location /svn> DAV svn SVNPath /home/ubuntu/svn/svnqarepo AuthType Basic AuthName "myproject subversion repository" AuthUserFile /etc/subversion/passwd Require valid-user </Location>

Now lets add users, to add initial users execute the following 

sudo htpasswd -c /etc/subversion/passwd wso2qauser

Execute following command and restart Apache server

sudo a2enmod dav_svn

sudo service apache2 restart

Type the password when password is asked. Now you should be able to access the SVN repository with the following URL

http://<HOST_NAME>/svn
https://<HOST_NAME>/svn

When you access the URL it will be like following, and username/password should be entered when accessing the repository.

Now you are done configuring the server, Please drop a comment if you have queries. :)

Read More

How to Install Apache2 server and enable SSL in Ubuntu

In this post I'll explain you how to install Apache2 Server and enable SSL for Apache2. Here I'm working on ubuntu 14.04 LTS. So lets get started.

Installing Apache2

First Lets update the Ubuntu repositories.

sudo apt-get install udgrade
sudo apt-get install update

Now Lets install apache2

sudo apt-get install apache2

Enable SSL in Apache Server

Create a directory as shown below,

sudo mkdir /etc/apache2/ssl

Now go to the created directory

cd /etc/apache2/ssl

Creating the certificates. 

Execute the following commands to create the key

sudo openssl genrsa -out "apache.key" 2048

Execute the the following to create the certificate sign request

sudo openssl req -new -key "apache.key" -out "apache.csr"

Fill information similar to following whencreating the CSR

Lets Sign the Key

sudo openssl x509 -req -days 3650 -in "apache.csr" -signkey "apache.key" -out "apache.crt"

Now Enable SSL module, if this is module is not installed you will have to install it :). Execute following to enable SSL in apache2

sudo a2enmod ssl

Now add the following to 000-default.conf in sites-enabled which is located at /etc/apache2/sites-enabled/000-default.conf

<VirtualHost *:443>
   
    SSLEngine on
    SSLCertificateFile  /etc/apache2/ssl/apache.crt
    SSLCertificateKeyFile /etc/apache2/ssl/apache.key

</VirtualHost>

Note : You can define server names and other parameters when configurin this further, above is the minimum content that is required.

Now restart the server.

sudo service apache2 restart

That is it, Now you are good to go :).

Please drop a comment if you have any Queries.

Read More