Sunday, 12 March 2017

Let's Encrypt SSL Recipes -- MySQL with Lets Encrypt Certificates

It may be the case that you need to encrypt a connection to a MySQL server. You may want to do this if you are connecting to the server remotely, e.g. via a MySQL client across the internet or an ODBC connection. Normally this would not be required because you would be connected to a web server that would have a private network or other secure connection to the database server, but in some cases even that private network connection may not be secure -- hence the need for encryption.

If you need to secure a MySQL connection, you can use Let's Encrypt certificates to do so. MySQL is (like sendmail) fussy about the permissions on the certificates and keys, and so you should copy the Let's Encrypt files from their normal location to MySQL's directory and then set permissions that way.

Assuming that you have created certificates for the site, use this recipe:

cp /etc/letsencrypt/live/*.pem /var/lib/mysql
chown mysql.mysql /var/lib/mysql/*.pem
chmod 600 /var/lib/mysql/*.pem
Then you just need to add these lines to the [mysqld] section of your MySQL configuration file (this will be either /etc/my.cnf or /etc/mysql/my.cnf or /etc/my.cnf.d/server.cnf):

Restart your MySQL or MariaDB server :

service mariadb restart
Make a connection to the MySQL server using the --ssl option:

mysql --ssl
Then to check to see that you have installed everything correctly, use the following command:

SHOW STATUS LIKE 'Ssl_cipher';
You should get output that looks like this:

Variable_name       Value
Ssl_cipher DHE-RSA-AES256-GCM-SHA384

Tuesday, 7 March 2017

Let's Encrypt SSL Recipes -- 389 Directory Server with LDAPS


These instructions are aimed at installing 389 Directory Server with the latest release of Red Hat Enterprise Linux or CentOS (currently release 7).
I assume that you have already created SSL certificates using Let's Encrypt as per the first part of this series.

389 Directory Server Installation

To install the directory server:

yum -y install 389-ds-base 389-admin 389-admin-console 389-console 389-ds-console xauth

The 389 directory server home page is here: and the installation guide is here:

Directory Server Setup

To set up the 389 directory server, a script is provided:

Run this script once and only once! It will ask you for some port numbers and a directory manager password. If you forget these then the only option is to rebuild the directory server.

Note that administering the 389 directory server is done using the administration console. To run this remotely requires an authenticated remote X11 session, hence the installation of "xauth". The 389 administration console is a Java program so you can install it on any desktop PC and do the administration remotely if you don't have a local X11 server.

During the setup process you will be asked some questions. The defaults should mostly be sufficient. Some questions you need to pay attention to are as follows:
Please enter the administrator ID for the configuration directory
server.  This is the ID typically used to log in to the console.  You
will also be prompted for the password.

Configuration directory server
administrator ID [admin]: 
Password (confirm):

You should choose the default administrator ID here admin and also make sure that you remember this password, it will be important later.
Each instance of a directory server requires a unique identifier.
This identifier is used to name the various
instance specific files and directories in the file system,
as well as for other uses as a server instance identifier.

Directory server identifier [ldap]:

This instance name is important -- we will need it later. For the time being we will put that instance name into a shell variable. I stay with the default instance name ldap because that seems to be a pretty obvious choice:
The suffix is the root of your directory tree.  The suffix must be a valid DN.
It is recommended that you use the dc=domaincomponent suffix convention.
For example, if your domain is,
you should use dc=example,dc=com for your suffix.
Setup will create this initial suffix for you,
but you may have more than one suffix.
Use the directory server utilities to create additional suffixes.

Suffix [dc=mysite,dc=com]:

This suffix becomes the Base DN of your directory. You should use the format above -- e.g. becomes dc=mysite,dc=com. Of course substitute your own domain name in there for
Certain directory server operations require an administrative user.
This user is referred to as the Directory Manager and typically has a
bind Distinguished Name (DN) of cn=Directory Manager.
You will also be prompted for the password for this user.  The password must
be at least 8 characters long, and contain no spaces.
Press Control-B or type the word "back", then Enter to back up and start over.

Directory Manager DN [cn=Directory Manager]: 
Password (confirm):

This password is very important because it's the one that you will use to create directory entries, later.
The Administration Server is separate from any of your web or application
servers since it listens to a different port and access to it is

Pick a port number between 1024 and 65535 to run your Administration
Server on. You should NOT use a port number which you plan to
run a web or application server on, rather, select a number which you
will remember and which will not be used for anything else.

Administration port [9830]:

Again this port number is important. Picking the default of 9830 seems to be a good choice.

Firewall Rules

One thing that I always forget on a RHEL or CentOS server is to set the correct firewall rules. This recipe allows access to the firewall on the LDAP and LDAPS ports (we may disable LDAP later):

# Add rules to the public zone to allow LDAP and LDAPS
firewall-cmd --permanent --zone=public --add-service=ldap
firewall-cmd --permanent --zone=public --add-service=ldaps
firewall-cmd --permanent --zone=public --add-port=9830/tcp
firewall-cmd --reload

# Check the firewall rules created by firewalld
iptables -L -n

# Check the rules loaded into firewalld
firewall-cmd --state
firewall-cmd --zone=public --list-all

Note that the port for LDAP is 389 and the port for LDAPS is 636 so you should check that these are both enabled in your firewall rule set.

Certificate Installation

Firstly, here are some references that I read in order to create these recipes. You may also find them helpful:

PKCS12 file

There are many different certificate file formats. Let's Encrypt provides certificates in PEM format, while 389 Directory Server requires them in PKCS12 format. We can use OpenSSL to convert between the two. We will put the converted certificate into a temporary directory while we are working on it:

# Make the temporary directory to store the PKCS12 file
mkdir -p $TMPDIR
chmod 700 $TMPDIR

# Make the PKCS12 file from the letsencrypt PEM files
rm -f $TMPDIR/$SITE.p12
openssl pkcs12 -export -in /etc/letsencrypt/live/$SITE/cert.pem \
    -inkey /etc/letsencrypt/live/$SITE/privkey.pem \
    -nodes -passout pass: -name $SITE -out $TMPDIR/$SITE.p12

As in all of these recipes, replace with your actual server name, the name that you obtained the certificate for. This should be exactly the same hostname that you ask your LDAP clients to connect to the server with, otherwise certificate verification will fail.

Import the PKCS12 file into 389 Directory Server

Importing the PKCS12 file into 389 Directory Server is done using the pk12util command line utility program provided with the nss-tools package -- this should already have been installed as part of the 389 Directory Server installation. The first time you do this it will ask you for a password which will become the password for the NSS key store. You will need to remember this password (along with the admin and cn=Directory Manager passwords that you created when you set up the directory). A password manager program such as keepass may be useful at this point.
# Import the PKCS12 file into 389DS
# This will require the Password for the 389DS NSS Certificate DB
pk12util -d /etc/dirsrv/slapd-$LDAPINSTANCE -i $TMPDIR/$SITE.p12

Import the Certificates into 389 Directory Server

We can now separately import the certificates into 389 Directory Server. When we do this, we give the certificates some specific nicknames:
# Import the certs separately with standard certificate nicknames
certutil -A -d /etc/dirsrv/slapd-$LDAPINSTANCE/ -n "ca_cert" -t "C,," -i /etc/letsencrypt/live/$SITE/chain.pem
certutil -A -d /etc/dirsrv/slapd-$LDAPINSTANCE/ -n "Server-Cert" -t ",," -i /etc/letsencrypt/live/$SITE/cert.pem

Check the Keys and Certificates

To check that the keys and certificates have been imported correctly, you can list the contents of the key store and the certificate store using these commands. Note that to list the contents of the key store you will need to provide the key store password:

# Check on keys and certs imported into 389DS
# This will require the Password for the 389DS NSS Certificate DB
certutil -K -d /etc/dirsrv/slapd-$LDAPINSTANCE
certutil -L -d /etc/dirsrv/slapd-$LDAPINSTANCE
In case you ever need to delete keys or certificates from the site, you can use these commands:

# In case you need to delete keys or certificates, do this
certutil -d /etc/dirsrv/slapd-$LDAPINSTANCE -F -n 'key_nickname'
certutil -d /etc/dirsrv/slapd-$LDAPINSTANCE -D -n 'certificate_nickname'

Enable LDAP Security

Most of these recipes follow the guide here:

Before you start, shut down the directory server as follows:
systemctl stop dirsrv@ldap

You now need to edit the file /etc/dirsrv/slapd-ldap/dse.ldif. This is the raw configuration data that is loaded into your directory server when it boots.
The format of this file is:
dn: (some dn)
attribute1: value1
attribute2: value2
... etc
(blank line before the next dn)

So when you're editing this file, maintain the correct format. You can insert and remove lines but leave the blank lines in place, and don't add any extra blank lines. To turn on and enable LDAP security, you need to edit this file, and change some entries by adding or removing attributes.


Add this attribute:
nsslapd-security: on


Set all of these attributes:

nsSSLSessionTimeout: 0
nsSSLClientAuth: off
nsSSL3: off
nsSSL2: off
Remove the sslVersionMin, sslVersionMax and nsSSL3Ciphers attributes if they are present.


You will need to add this as a new entry. It should look like this:

dn: cn=RSA,cn=encryption,cn=config
objectClass: top
objectClass: nsEncryptionModule
nsSSLPersonalitySSL: Server-Cert
nsSSLActivation: on
nsSSLToken: internal (software)
cn: RSA
Note that Server-Cert matches the nickname of the certificate that you imported using certutil above (after the -n parameter). You don't need to specify the nickname of the CA signing certificate, Directory Server and NSS will work out this chain correctly provided all CA and chain elements are in the certificate database.

Specify the NSS Key Store Password

You need to put the password for the NSS key store into a file so that the server can read it when it boots.

The file name is /etc/dirsrv/slapd-ldap/pin.txt (in the same directory as dse.ldif) and it contains the password in this format:

Internal (Software) Token:MySecretPassword
This password is the same one as the password that you gave when you first imported the key into the key store using pk12util.

Restart the Directory Server

systemctl restart dirsrv@ldap

Check That Security Is Turned On

To check that security is turned on you can try these commands:

Search for the nssldap-security entry

ldapsearch -x -W -D 'cn=Directory Manager' -H ldap://localhost/ -b 'cn=config' -s base nsslapd-security

This should report:
nsslapd-security: on

Search for the Encryption Config

ldapsearch -x -W -D 'cn=Directory Manager' -H ldap://localhost/ -b 'cn=encryption,cn=config' -s base

This should report a large entry with lots of nssslenabledciphers attributes, and also the following:
nsSSL2: off
nsSSL3: off
nsTLS1: on
sslVersionMin: TLS1.0
sslVersionMax: TLS1.2

Search for the RSA config

ldapsearch -x -W -D 'cn=Directory Manager' -H ldap://localhost/ -b 'cn=RSA,cn=encryption,cn=config' -s base

This should report something like the following:
dn: cn=RSA,cn=encryption,cn=config
objectClass: top
objectClass: nsEncryptionModule
nsSSLPersonalitySSL: Server-Cert
nsSSLActivation: on
nsSSLToken: internal (software)
cn: RSA

Do an LDAPS search

Now that you have SSL/TLS configured on your LDAP server you should be able to do searches like this:

ldapsearch -x -H ldaps://localhost/ -s base -b '' namingContexts
Note the ldaps part of the URL (-H parameter) which specifies to connect on port 636 and set up an encrypted SSL session with the LDAP server.

Do the same search with the ldaps:// URL and then gain with the ldap:// URL (non-SSL mode) to check that you get the same results.


If you want to restrict your directory server to doing SSL sessions only, you could remove the open port 389 from the firewall configuration and leave only port 636 open.

firewall-cmd --permanent --zone=public --remove-service=ldap
firewall-cmd --reload
firewall-cmd --zone=public --list-all

Test the Certificate Using OpenSSL

OpenSSL has a useful tool for probing SSL enabled sites and checking their certificates. You should do this from outside of your new LDAP machine:

openssl s_client -connect -showcerts </dev/null
This command will establish a connection to your LDAP server on the LDAPS port (636) and print out the certificates. Because it reads input from /dev/null it will finish once the connection is established.

You should get 2 certificates printed out. One will be your server certificate and will begin like this:

 0 s:/
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
... the other will be the Let's Encrypt authority certificate and will begin like this:

 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
As long as you see both of those certificates you should be OK. Towards the end of the output there will be a block like this:

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
This is telling you about the encryption negotiated between s_client and your LDAP server. In this case it's showing a good level of security with TLS v1.2 being enabled as well as a cipher from the AES-256 cipher suite being negotiated with SHA384 as the signing hash algorithm -- all current industry standard security.

Using 389 Directory Server

There are plenty of articles about how to use the Directory Server console so I won't go into details here. This is a tool that you can use to manage the directory server.

The directory server can be used as an authentication method for many applications, including:
  • redmine and JIRA
  • gitlab
  • Many CRM systems
... etc. The usual information that you will need to provide for these programs when you are setting them up for LDAP access to your new directory server are:
  • Hostname -- or whatever your directory server host name is.
  • Port -- 636 and you will need to check the "SSL" or "LDAPS" options.
  • Base DN -- this was set by you when you ran the configuration script.
  • Search DN -- leave this empty.
  • Search Password -- leave this empty.
  • LDAP Filter -- you probably want to use something like objectClass=inetOrgPerson
  • Mapping attributes. You may or may not need to do this but some attributes that you may need to provide are:
    • Login Attribute: uid
    • First Name Attribute: givenName
    • Surname Attribute: sn
    • Email Attribute: mail
  • Active Directory Mode -- what this means may depend on your application but some applications use this to turn on a whole bunch of options. You should turn this off.

Extra Software

There are some extra applications that I find useful to install on a Directory Server. These are:
  • Self Service Password -- this application allows a simple interface for users to change their password.
  • phpLDAPAdmin -- a web application to add/update/remove directory entries, gives you full management over your directory server without having to use the directory server console. You can install this directly from the EPEL repository like this:
yum -y install phpldapadmin
After installing you may want to trim down the templates that are in /usr/share/phpldapadmin/templates/creation/ I generally remove all of them except the following:
  • alias.xml
  • inetOrgPerson.xml
  • organizationalRole.xml
  • ou.xml
  • posixAccount.xml
  • posixGroup.xml
  • simpleSecurityObject.xml

Sunday, 5 March 2017

Let's Encrypt SSL Recipes -- Mail Server (sendmail and dovecot)

In part 1 of this series I showed how to obtain a free SSL certificate from Let's Encrypt and use the certificate to secure a website using Apache.

The same certificate can be used to secure a mail server such as dovecot, and a mail transfer agent such as sendmail.  I will show dovecot first because it's the easiest.

The dovecot configuration files are usually in /etc/dovecot with the master configuration file being /etc/dovecot/dovecot.conf.  In fact on most distributions you shouldn't edit this file, instead you should edit the files in /etc/dovecot/conf.d/. The one file you need to edit is /etc/dovecot/conf.d/10-ssl.conf on Red Hat and CentOS distributions.

The lines that you need to set in this file are as follows:

ssl_cert = </etc/letsencrypt/live/
ssl_key =  </etc/letsencrypt/live/
ssl_protocols = !SSLv2 !SSLv3
ssl_cipher_list = ALL:!LOW:!SSLv2:!EXP:!aNULL:!RC4::!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS

As before, replace with the actual site name of your web site, the site name that has been used to obtain the SSL certificate.  Note that you need to give the path to the fullchain.pem file obtained by certbot and not just the cert.pem file.

The site name should be the same as server name that you configure in your mail client to access this server.  For example if you're configuring Thunderbird you should set this as the Server Name under Server Settings in the Account Settings. Don't use another alias or CNAME, use the exact server name as registered on the certificate.

The sendmail Mail Transfer Agent (MTA) usually used for sending mail is a bit fussier about the certificates.  It can't use the certificates from the Let's Encrypt path because sendmail insists on certain permissions on the key file -- it must not be world readable.  So therefore in my cron script to renew the certificates I add some extra lines to copy the certificate and key elsewhere, and put some permissions on it that sendmail will be happy with:

cat /etc/letsencrypt/live/ \
    /etc/letsencrypt/live/ > \
chmod 0600 /etc/pki/tls/certs/sendmail.pem
cat /etc/letsencrypt/live/ > /etc/pki/tls/certs/chain.pem

The above lines will put a copy of the combined key and certificate in /etc/pki/tls/certs as well as the CA chain file.

The next step is to configure sendmail.  I do this by editing the sendmail macro file ( from the sendmail-cf package on CentOS:

cd /usr/share/sendmail-cf/cf

Add these lines to the macro file:

DAEMON_OPTIONS(`Port=smtp, Name=MTA')dnl
DAEMON_OPTIONS(`Port=smtps, Name=TLSMTA, M=s')dnl

If you're configuring sendmail for TLS then it's probable that you also are going to use SMTP Authentication.  To do that you will need these additional lines in your macro file:

define(`confAUTH_OPTIONS', `A')dnl

Note that "LOGIN" and "PLAIN" authentications are mostly OK here -- because although it's plain text authentication it occurs in a TLS encrypted channel.  You will also need to start the SASL authentication daemon:

service saslauthd start
chkconfig saslauthd on

You will probably also want to set up Fail2Ban on your server as well because once you have sendmail authentication turned on you will receive many brute force attempts to hack your passwords.

Rebuild your macro file using make and copy this to your normal location.

I will update this for postfix at some point.

Saturday, 4 March 2017

Let's Encrypt SSL Recipes -- Renewing certificates

In the previous post I explained how to obtain a Let's Encrypt SSL certificate for free.

Let's Encrypt certificates, when they are issued, are only valid for 90 days.  However it costs nothing to renew them and the process for doing so is easy.

You can run the certbot program daily or weekly (your choice) to automatically renew the certificates when they are ready for renewal.  You can create a cron script to do this for you automatically:

cat << EOFRENEW > /etc/cron.weekly/certbot-renew

/root/bin/certbot-auto renew --quiet
chmod +x /etc/cron.weekly/certbot-renew

This will create a script that will run weekly and renew any certificates that need renewing.

Once the certificate has been renewed you will need to restart Apache or nginx (or any other programs using the certificate) so you need to plan a short outage window to do that.

Let's Encrypt SSL Recipes -- HTTPS with Apache or Nginx

This post is aimed at people who run their own web servers.  These days there is no real excuse for not using HTTPS instead of HTTP on a web site, and the reasons for switching from HTTP to HTTPS are many:
  • Traffic to and from HTTPS sites is encrypted, HTTP traffic is not encrypted.
  • Google and other search engines rank HTTPS web sites more highly than HTTP web sites.
  • New versions of Chrome and some other web browsers will soon start displaying warnings when accessing a HTTP web site.
With the advent of the Let's Encrypt service which offers free SSL certificates for all web sites, there is no no real excuse for using HTTP instead of HTTPS -- using HTTPS is free and easy.

I'll start by providing some simple recipes to obtain Let's Encrypt certificates and install them on a web server -- then I'll follow up with some more posts about using Let's Encrypt certificates for other reasons.

On a recent Linux server, it's now possible to use a simple recipe to install the Let's Encrypt certbot program, which creates certificates:

wget --no-check-certificate
chmod +x certbot-auto
mkdir -p /root/bin
mv certbot-auto /root/bin
You can run certbot and create the certificate using this recipe:

/root/bin/certbot-auto certonly --webroot --webroot-path /var/www/html

This should work on most recent Linux variants -- Ubuntu, Debia, and Red Hat or CentOS.  Replace /var/www/html in the above recipe with your actual web root.  Certbot will ask you some questions (including your site name) and then create the certificate after checking that the site actually points to the server that you're running on.

The next thing is to update your Apache configuration (the default web server on Red hat and CentOS -- I will get to nginx later).

In your Apache virtualhost configuration you will need some lines like this:

    SSLEngine on
    SSLCertificateFile /etc/letsencrypt/live/
    SSLCertificateKeyFile /etc/letsencrypt/live/
    SSLCACertificateFile /etc/letsencrypt/live/mysitecom/chain.pem
    SSLProtocol all -SSLv2 -SSLv3
    SSLHonorCipherOrder on
    SetEnvIf User-Agent ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0
    # HSTS
    Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains;" 

In the above recipe replace with your site name.

I have included a SSLCipherSuite line that should work for most people -- on reasonably recent Apache 2.2 or 2.4 versions.  If you want to be more specific about which cipher suites you want to support then try the Mozilla SSL Configuration Generator which will give you various options for most versions of Apache and nginx.

Note I have also included a line for HSTS which your site should support if you are running an HTTPS only site.

The next thing to do is to redirect HTTP to HTTPS.  This is done using the following lines in your Apache configuration:

    # Redirect HTTP to HTTPS
    RewriteEngine On
    RewriteCond %{HTTPS} !=on
    RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R=301,L]
For nginx there are actually a lot of recipes around on the internet about how to incorporate Let's Encrypt certificates.  Here are the nginx settings that I use.  All of this goes into a configuration file which on CentOS could be something like /etc/nginx/conf.d/ssl.conf:

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    # certs sent to the client in SERVER HELLO are concatenated in ssl_certificate
    ssl_certificate /etc/letsencrypt/live/;
    ssl_certificate_key /etc/letsencrypt/live/;
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets off;

    # Diffie-Hellman parameter for DHE ciphersuites, recommended 2048 bits
    ssl_dhparam /etc/ssl/certs/dhparam.pem;

    # intermediate configuration. tweak to your needs.
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;

    # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
    add_header Strict-Transport-Security max-age=15768000;

    # OCSP Stapling ---
    # fetch OCSP records from URL in ssl_certificate and cache them
    ssl_stapling on;
    ssl_stapling_verify on;

    ## verify chain of trust of OCSP response using Root CA and Intermediate certs
    ssl_trusted_certificate /etc/letsencrypt/live/;

    #resolver <IP DNS resolver>;

    # Load configuration files for the default server block.
    include /etc/nginx/default.d/*.conf;

    location / {
        ## Put location specific configuration here.

To generate the dhparam.pem file referred to in the configuration above, run this command:

openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048

Finally, you should test the HTTPS security component of your web site using the SSL Labs SSL Server Test tool.  Hopefully you should get an A+ rating on this tool, if not then note any issues that the tool raises and fix them.