Scenario
It has been that situation again when I have spent a whole day trying to research and figure out the solution to a problem that has bugged me. Hopefully this will save researching time for people having the same issue.
If you, like me, are trying to get cURL, or rather, plugin such as WordPress OpenID (which uses PHP libcurl) to work with HTTPS sites that use self-signed certificates but do not wish to compromise the security by disabling CURLOPT_SSL_VERIFYPEER and/or CURLOPT_SSL_VERIFYHOST option in the code, the following might be the answer for you.
Some background
You can skip this part actually. But if you are interested in some technical details, you can continue reading. This post is written on the assumption that Ubuntu Linux is used. It should apply to other distros as well with minor differences.
When we are using cURL to retrieve a HTTPS site that is not using a CA-signed certificate, the following problem occurs. Of course, this can simply be overcome by using the -k option.
root@ubuntu:/etc# curl https://example.selfip.com curl: (60) SSL certificate problem, verify that the CA cert is OK. Details: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed More details here: http://curl.haxx.se/docs/sslcerts.html curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). The default bundle is named curl-ca-bundle.crt; you can specify an alternate file using the --cacert option. If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL). If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option.
However, in the case of applications such as WordPress OpenID plugin, we have to amend the code to disable both CURLOPT_SSL_VERIFYPEER and/or CURLOPT_SSL_VERIFYHOST. Not a very good idea as this will disable verification for authentic sites as well. If we know example.selfip.com, in this case is one of our own trusted test servers, we can add its certificate to the trusted list.
Now that we know what to do, we have to find out how to do it. It took me quite a while to look for the solution as I was looking in the wrong places. WordPress OpenID and cURL documentations/forums did not point me in the right direction. I decided to take a look at the source code in cURL and discover that ultimately, the responsibility of the verification lies with OpenSSL.
Arm with this and some right keyword searching, it has led me to this HOWTO, which provided some insight on how OpenSSL recognizes certificate authorities, and an important program. Right now, we are all set to add the new certificate so that our OpenSSL can recognize the HTTPS server.
The solution
Identify which directory your OpenSSL installation uses.
root@ubuntu:~# openssl version -d OPENSSLDIR: "/usr/lib/ssl"
Change to that directory.
root@ubuntu:~# cd /usr/lib/ssl
List the directory contents. You should see a directory called “certs”.
root@ubuntu:/usr/lib/ssl# ls -la total 24 drwxr-xr-x 4 root root 4096 2009-04-24 14:37 . drwxr-xr-x 50 root root 12288 2009-04-24 17:56 .. lrwxrwxrwx 1 root root 14 2009-04-24 14:37 certs -> /etc/ssl/certs drwxr-xr-x 2 root root 4096 2009-04-24 14:37 engines drwxr-xr-x 2 root root 4096 2009-04-24 14:37 misc lrwxrwxrwx 1 root root 20 2009-04-24 14:37 openssl.cnf -> /etc/ssl/openssl.cnf lrwxrwxrwx 1 root root 16 2009-04-24 14:37 private -> /etc/ssl/private
Change to that directory.
root@ubuntu:/usr/lib/ssl# cd certs
List the directory contents. You should see from the symlinks that the certificates are actually stored in “/usr/share/ca-certificates”.
root@ubuntu:/usr/lib/ssl/certs# ls -la total 648 drwxr-xr-x 2 root root 16384 2009-06-24 12:32 . drwxr-xr-x 4 root root 4096 2009-04-24 14:39 .. lrwxrwxrwx 1 root root 26 2009-06-24 12:32 00673b5b.0 -> thawte_Primary_Root_CA.pem lrwxrwxrwx 1 root root 59 2009-06-24 12:32 2edf7016.0 -> Verisign_Class_1_Public_Primary_Certification_Authority.pem lrwxrwxrwx 1 root root 61 2009-06-24 12:32 thawte_Primary_Root_CA.pem -> /usr/share/ca-certificates/mozilla/thawte_Primary_Root_CA.crt lrwxrwxrwx 1 root root 94 2009-06-24 12:32 Verisign_Class_1_Public_Primary_Certification_Authority.pem -> /usr/share/ca-certificates/mozilla/Verisign_Class_1_Public_Primary_Certification_Authority.crt . . .
Change to that directory.
root@ubuntu:/usr/lib/ssl/certs# cd /usr/share/ca-certificates
List the directory contents. As you can see, I have added “example.selfip.com.crt” here.
root@ubuntu:/usr/share/ca-certificates# ls -la total 56 drwxr-xr-x 11 root root 4096 2009-06-24 15:02 . drwxr-xr-x 98 root root 4096 2009-04-24 17:51 .. drwxr-xr-x 2 root root 4096 2009-04-24 14:35 brasil.gov.br drwxr-xr-x 2 root root 4096 2009-04-24 14:35 cacert.org drwxr-xr-x 2 root root 4096 2009-04-24 14:35 debconf.org -rw-r--r-- 1 root root 1220 2009-06-24 11:57 example.selfip.com.crt drwxr-xr-x 2 root root 4096 2009-04-24 14:35 gouv.fr drwxr-xr-x 2 root root 12288 2009-04-24 14:35 mozilla drwxr-xr-x 2 root root 4096 2009-04-24 14:35 quovadis.bm drwxr-xr-x 2 root root 4096 2009-04-24 14:35 signet.pl drwxr-xr-x 2 root root 4096 2009-04-24 14:35 spi-inc.org drwxr-xr-x 2 root root 4096 2009-04-24 14:35 telesec.de
Change to “/etc” directory and edit the file “ca-certificates.conf”.
root@ubuntu:/usr/share/ca-certificates# cd /etc root@ubuntu:/etc# nano ca-certificates.conf
Add “example.selfip.com.crt” to the file and save it.
------------------------------------------------------------------------------- GNU nano 2.0.9 File: ca-certificates.conf # This file lists certificates that you wish to use or to ignore to be # installed in /etc/ssl/certs. # update-ca-certificates(8) will update /etc/ssl/certs by reading this file. # # This is autogenerated by dpkg-reconfigure ca-certificates. # Certificates should be installed under /usr/share/ca-certificates # and files with extension '.crt' is recognized as available certs. # # line begins with # is comment. # line begins with ! is certificate filename to be deselected. # example.selfip.com.crt brasil.gov.br/brasil.gov.br.crt cacert.org/cacert.org.crt cacert.org/class3.crt . . . -------------------------------------------------------------------------------
Execute the program “update-ca-certificates –fresh”.
Note: You might like to backup /etc/ssl/certs before executing the command.
root@ubuntu:/etc# update-ca-certificates --fresh Clearing symlinks in /etc/ssl/certs...done. Updating certificates in /etc/ssl/certs....done. Running hooks in /etc/ca-certificates/update.d....done.
Test with curl on your target HTTPS site and it should work now.
root@ubuntu:/etc# curl https://example.selfip.com <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> . . .
Excellent References:
http://www.madboa.com/geek/openssl/#verify-system
http://manpages.ubuntu.com/manpages/karmic/man8/update-ca-certificates.8.html