Another Post About Top Password Mistakes

There are a lot of blog posts around about the top mistakes that people make while setting passwords and just to add myself to the mix I will make one more.  There are a lot of misconceptions around about what makes a good password and what does not, but here are some common mistakes including some that other "top 5 password mistakes" advise you to make! I'll explain my reasoning for each one.

1. Passwords Too Short

This has to be the most common one, and having been working with computers for some time I remember the days when a 4 or 6 character password was considered OK.  It's not.

Brute force attacks on passwords have become faster and faster, and now a 6 character password can be broken by an average computer in a fairly short period of time. In fact, assuming that a hacker has obtained the hashed or encrypted set of passwords from your database, many serious hackers have access to "botnets" of computers which they can harness to brute force crack these databases.  If the password database is at all vulnerable (and if you're logging on to someone else's website then you don't know how vulnerable it is) then a 12 character password should be considered a minimum, and I recommend 20 characters.

2. Assuming That Complex Passwords Are Safer

It's a common mistake amongst IT departments to make people use "complex" passwords. By "complex" it's meant that passwords should include upper and lower case characters, numbers, punctuation marks and special characters, etc.  Against a brute force attack a complex password is no harder than a simple password.  The main issue that prevents brute force attacks is password length, not complexity.

In fact, as demonstrated several times for example in this XKCD comic, complex passwords have no more entropy than simple passwords and in fact having a complex password just encourages users to write the password down somewhere, totally destroying any additional safety in having a password that is complex.

Updated NIST password guidelines totally destroy the myth that complex passwords are safer.  They are not.

Articles that recommend using complex passwords for extra safety can safely be ignored.  Advising someone to avoid password mistakes by making them make another mistake is no advice worth following at all.

3. Not Using a Password Safe

Many people use the same password for every website, because they only want to remember one password.  The problem is that once someone has one of your passwords they have them all.  To avoid this problem you need to have a list of long passwords, which are unique for each website or software program that you use.

A password safe is the best way of maintaining a list of long passwords.  I wrote a detailed article on that some time ago, but the rules are simple:

• Download a copy of KeepassX in which to store your passwords.
• Use one long password, generated by a site like Correct Horse Battery Staple to be your password safe key.  This is the one and only password that you have to remember.
• Use KeepassX to generate long, unique, random passwords for all of your websites.  Since you never have to remember any of these because you just cut and paste them from KeepassX, they can be as long as you like -- 20 characters of random junk is fine.
• Save your KeepassX file somewhere secure and give the password for that file to your lawyer to be used only in case of your death.

4. Writing Down Passwords

Up to 64% of people in a 2003 survey report having written down their passwords, and a more recent report indicates that in an average USA office up to 40% of passwords can be found on a yellow sticky note hidden under the keyboard or somewhere similar.

Seriously, don't do this.

5. Belief In Regular Password Changes

Another commonly held belief is that forcing users to change their passwords once per month makes the passwords more secure.  It does not, it just encourages users to write down passwords more often.

Again the recent NIST password guidelines blow this mistake out of the water.  The only reason to force users to change their passwords is if you believe the password database has become compromised somehow or for some other reason involving increased security (such as upgrading the hash algorithm in use, or forcing longer passwords to be used, etc). Forcing regular password changes on people just encourages them to follow less secure password practices.

Let's Encrypt SSL Recipes -- MySQL with Lets Encrypt Certificates

It may be the case that you need to encrypt a connection to a MySQL server. You may want to do this if you are connecting to the server remotely, e.g. via a MySQL client across the internet or an ODBC connection. Normally this would not be required because you would be connected to a web server that would have a private network or other secure connection to the database server, but in some cases even that private network connection may not be secure -- hence the need for encryption.

If you need to secure a MySQL connection, you can use Let's Encrypt certificates to do so. MySQL is (like sendmail) fussy about the permissions on the certificates and keys, and so you should copy the Let's Encrypt files from their normal location to MySQL's directory and then set permissions that way.

Assuming that you have created certificates for the site mysite.com, use this recipe:

cp /etc/letsencrypt/live/mysite.com/*.pem /var/lib/mysql
chown mysql.mysql /var/lib/mysql/*.pem
chmod 600 /var/lib/mysql/*.pem

Then you just need to add these lines to the [mysqld] section of your MySQL configuration file (this will be either /etc/my.cnf or /etc/mysql/my.cnf or /etc/my.cnf.d/server.cnf):

ssl_ca=/var/lib/mysql/chain.pem
ssl_cert=/var/lib/mysql/cert.pem
ssl_key=/var/lib/mysql/privkey.pem

Restart your MySQL or MariaDB server :

service mariadb restart

Make a connection to the MySQL server using the --ssl option:

mysql --ssl

Then to check to see that you have installed everything correctly, use the following command:

SHOW STATUS LIKE 'Ssl_cipher';

You should get output that looks like this:

Variable_name	Value
Ssl_cipher	DHE-RSA-AES256-GCM-SHA384

Let's Encrypt SSL Recipes -- 389 Directory Server with LDAPS

Assumptions

These instructions are aimed at installing 389 Directory Server with the latest release of Red Hat Enterprise Linux or CentOS (currently release 7).
I assume that you have already created SSL certificates using Let's Encrypt as per the first part of this series.

389 Directory Server Installation

To install the directory server:

yum -y install 389-ds-base 389-admin 389-admin-console 389-console 389-ds-console xauth

The 389 directory server home page is here: http://directory.fedoraproject.org/ and the installation guide is here: https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Installation_Guide/index.html

Directory Server Setup

To set up the 389 directory server, a script is provided:

setup-ds-admin.pl

Run this script once and only once! It will ask you for some port numbers and a directory manager password. If you forget these then the only option is to rebuild the directory server.

Note that administering the 389 directory server is done using the administration console. To run this remotely requires an authenticated remote X11 session, hence the installation of "xauth". The 389 administration console is a Java program so you can install it on any desktop PC and do the administration remotely if you don't have a local X11 server.

During the setup process you will be asked some questions. The defaults should mostly be sufficient. Some questions you need to pay attention to are as follows:

==============================================================================
Please enter the administrator ID for the configuration directory
server.  This is the ID typically used to log in to the console.  You
will also be prompted for the password.

Configuration directory server
administrator ID [admin]: 
Password: 
Password (confirm):

You should choose the default administrator ID here admin and also make sure that you remember this password, it will be important later.

==============================================================================
Each instance of a directory server requires a unique identifier.
This identifier is used to name the various
instance specific files and directories in the file system,
as well as for other uses as a server instance identifier.

Directory server identifier [ldap]:

This instance name is important -- we will need it later. For the time being we will put that instance name into a shell variable. I stay with the default instance name ldap because that seems to be a pretty obvious choice:

LDAPINSTANCE=ldap

==============================================================================
The suffix is the root of your directory tree.  The suffix must be a valid DN.
It is recommended that you use the dc=domaincomponent suffix convention.
For example, if your domain is example.com,
you should use dc=example,dc=com for your suffix.
Setup will create this initial suffix for you,
but you may have more than one suffix.
Use the directory server utilities to create additional suffixes.

Suffix [dc=mysite,dc=com]:

This suffix becomes the Base DN of your directory. You should use the format above -- e.g. mysite.com becomes dc=mysite,dc=com. Of course substitute your own domain name in there for mysite.com.

==============================================================================
Certain directory server operations require an administrative user.
This user is referred to as the Directory Manager and typically has a
bind Distinguished Name (DN) of cn=Directory Manager.
You will also be prompted for the password for this user.  The password must
be at least 8 characters long, and contain no spaces.
Press Control-B or type the word "back", then Enter to back up and start over.

Directory Manager DN [cn=Directory Manager]: 
Password: 
Password (confirm):

This password is very important because it's the one that you will use to create directory entries, later.

==============================================================================
The Administration Server is separate from any of your web or application
servers since it listens to a different port and access to it is
restricted.

Pick a port number between 1024 and 65535 to run your Administration
Server on. You should NOT use a port number which you plan to
run a web or application server on, rather, select a number which you
will remember and which will not be used for anything else.

Administration port [9830]:

Again this port number is important. Picking the default of 9830 seems to be a good choice.

Firewall Rules

One thing that I always forget on a RHEL or CentOS server is to set the correct firewall rules. This recipe allows access to the firewall on the LDAP and LDAPS ports (we may disable LDAP later):

# Add rules to the public zone to allow LDAP and LDAPS
firewall-cmd --permanent --zone=public --add-service=ldap
firewall-cmd --permanent --zone=public --add-service=ldaps
firewall-cmd --permanent --zone=public --add-port=9830/tcp
firewall-cmd --reload

# Check the firewall rules created by firewalld
iptables -L -n

# Check the rules loaded into firewalld
firewall-cmd --state
firewall-cmd --zone=public --list-all

Note that the port for LDAP is 389 and the port for LDAPS is 636 so you should check that these are both enabled in your firewall rule set.

Certificate Installation

Firstly, here are some references that I read in order to create these recipes. You may also find them helpful:

• https://labs.par-tec.it/certutil-cheatsheet/
• https://www.centos.org/docs/5/html/CDS/ag/8.0/Managing_SSL-Using_certutil.html
• http://directory.fedoraproject.org/docs/389ds/howto/howto-ssl.html

PKCS12 file

There are many different certificate file formats. Let's Encrypt provides certificates in PEM format, while 389 Directory Server requires them in PKCS12 format. We can use OpenSSL to convert between the two. We will put the converted certificate into a temporary directory while we are working on it:

TMPDIR=/root/keys
SITE=mysite.com

#
# Make the temporary directory to store the PKCS12 file
#
mkdir -p $TMPDIR
chmod 700 $TMPDIR

#
# Make the PKCS12 file from the letsencrypt PEM files
#
rm -f $TMPDIR/$SITE.p12
openssl pkcs12 -export -in /etc/letsencrypt/live/$SITE/cert.pem \
    -inkey /etc/letsencrypt/live/$SITE/privkey.pem \
    -nodes -passout pass: -name $SITE -out $TMPDIR/$SITE.p12

As in all of these recipes, replace mysite.com with your actual server name, the name that you obtained the certificate for. This should be exactly the same hostname that you ask your LDAP clients to connect to the server with, otherwise certificate verification will fail.

Import the PKCS12 file into 389 Directory Server

Importing the PKCS12 file into 389 Directory Server is done using the pk12util command line utility program provided with the nss-tools package -- this should already have been installed as part of the 389 Directory Server installation. The first time you do this it will ask you for a password which will become the password for the NSS key store. You will need to remember this password (along with the admin and cn=Directory Manager passwords that you created when you set up the directory). A password manager program such as keepass may be useful at this point.

#
# Import the PKCS12 file into 389DS
# This will require the Password for the 389DS NSS Certificate DB
#
pk12util -d /etc/dirsrv/slapd-$LDAPINSTANCE -i $TMPDIR/$SITE.p12

Import the Certificates into 389 Directory Server

We can now separately import the certificates into 389 Directory Server. When we do this, we give the certificates some specific nicknames:

#
# Import the certs separately with standard certificate nicknames
#
certutil -A -d /etc/dirsrv/slapd-$LDAPINSTANCE/ -n "ca_cert" -t "C,," -i /etc/letsencrypt/live/$SITE/chain.pem
certutil -A -d /etc/dirsrv/slapd-$LDAPINSTANCE/ -n "Server-Cert" -t ",," -i /etc/letsencrypt/live/$SITE/cert.pem

Check the Keys and Certificates

To check that the keys and certificates have been imported correctly, you can list the contents of the key store and the certificate store using these commands. Note that to list the contents of the key store you will need to provide the key store password:

#
# Check on keys and certs imported into 389DS
# This will require the Password for the 389DS NSS Certificate DB
#
certutil -K -d /etc/dirsrv/slapd-$LDAPINSTANCE
certutil -L -d /etc/dirsrv/slapd-$LDAPINSTANCE

In case you ever need to delete keys or certificates from the site, you can use these commands:

#
# In case you need to delete keys or certificates, do this
#
certutil -d /etc/dirsrv/slapd-$LDAPINSTANCE -F -n 'key_nickname'
certutil -d /etc/dirsrv/slapd-$LDAPINSTANCE -D -n 'certificate_nickname'

Enable LDAP Security

Most of these recipes follow the guide here: http://directory.fedoraproject.org/docs/389ds/howto/howto-ssl.html

Before you start, shut down the directory server as follows:

systemctl stop dirsrv@ldap

You now need to edit the file /etc/dirsrv/slapd-ldap/dse.ldif. This is the raw configuration data that is loaded into your directory server when it boots.
The format of this file is:

dn: (some dn)
attribute1: value1
attribute2: value2
... etc
(blank line before the next dn)

So when you're editing this file, maintain the correct format. You can insert and remove lines but leave the blank lines in place, and don't add any extra blank lines. To turn on and enable LDAP security, you need to edit this file, and change some entries by adding or removing attributes.

cn=config

Add this attribute:

nsslapd-security: on

cn=encryption,cn=config

Set all of these attributes:

nsSSLSessionTimeout: 0
nsSSLClientAuth: off
nsSSL3: off
nsSSL2: off

Remove the sslVersionMin, sslVersionMax and nsSSL3Ciphers attributes if they are present.

cn=RSA,cn=encryption,cn=config

You will need to add this as a new entry. It should look like this:

dn: cn=RSA,cn=encryption,cn=config
objectClass: top
objectClass: nsEncryptionModule
nsSSLPersonalitySSL: Server-Cert
nsSSLActivation: on
nsSSLToken: internal (software)
cn: RSA

Note that Server-Cert matches the nickname of the certificate that you imported using certutil above (after the -n parameter). You don't need to specify the nickname of the CA signing certificate, Directory Server and NSS will work out this chain correctly provided all CA and chain elements are in the certificate database.

Specify the NSS Key Store Password

You need to put the password for the NSS key store into a file so that the server can read it when it boots.

The file name is /etc/dirsrv/slapd-ldap/pin.txt (in the same directory as dse.ldif) and it contains the password in this format:

Internal (Software) Token:MySecretPassword

This password is the same one as the password that you gave when you first imported the key into the key store using pk12util.

Restart the Directory Server

systemctl restart dirsrv@ldap

Check That Security Is Turned On

To check that security is turned on you can try these commands:

Search for the nssldap-security entry

ldapsearch -x -W -D 'cn=Directory Manager' -H ldap://localhost/ -b 'cn=config' -s base nsslapd-security

This should report:

nsslapd-security: on

Search for the Encryption Config

ldapsearch -x -W -D 'cn=Directory Manager' -H ldap://localhost/ -b 'cn=encryption,cn=config' -s base

This should report a large entry with lots of nssslenabledciphers attributes, and also the following:

nsSSL2: off
nsSSL3: off
nsTLS1: on
sslVersionMin: TLS1.0
sslVersionMax: TLS1.2

Search for the RSA config

ldapsearch -x -W -D 'cn=Directory Manager' -H ldap://localhost/ -b 'cn=RSA,cn=encryption,cn=config' -s base

This should report something like the following:

dn: cn=RSA,cn=encryption,cn=config
objectClass: top
objectClass: nsEncryptionModule
nsSSLPersonalitySSL: Server-Cert
nsSSLActivation: on
nsSSLToken: internal (software)
cn: RSA

Do an LDAPS search

Now that you have SSL/TLS configured on your LDAP server you should be able to do searches like this:

ldapsearch -x -H ldaps://localhost/ -s base -b '' namingContexts

Note the ldaps part of the URL (-H parameter) which specifies to connect on port 636 and set up an encrypted SSL session with the LDAP server.

Do the same search with the ldaps:// URL and then gain with the ldap:// URL (non-SSL mode) to check that you get the same results.

Firewall

If you want to restrict your directory server to doing SSL sessions only, you could remove the open port 389 from the firewall configuration and leave only port 636 open.

firewall-cmd --permanent --zone=public --remove-service=ldap
firewall-cmd --reload
firewall-cmd --zone=public --list-all

Test the Certificate Using OpenSSL

OpenSSL has a useful tool for probing SSL enabled sites and checking their certificates. You should do this from outside of your new LDAP machine:

openssl s_client -connect mysite.com:636 -showcerts </dev/null

This command will establish a connection to your LDAP server on the LDAPS port (636) and print out the certificates. Because it reads input from /dev/null it will finish once the connection is established.

You should get 2 certificates printed out. One will be your server certificate and will begin like this:

 0 s:/CN=mysite.com
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
-----BEGIN CERTIFICATE-----

... the other will be the Let's Encrypt authority certificate and will begin like this:

 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
-----BEGIN CERTIFICATE-----

As long as you see both of those certificates you should be OK. Towards the end of the output there will be a block like this:

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384

This is telling you about the encryption negotiated between s_client and your LDAP server. In this case it's showing a good level of security with TLS v1.2 being enabled as well as a cipher from the AES-256 cipher suite being negotiated with SHA384 as the signing hash algorithm -- all current industry standard security.

Using 389 Directory Server

There are plenty of articles about how to use the Directory Server console so I won't go into details here. This is a tool that you can use to manage the directory server.

The directory server can be used as an authentication method for many applications, including:

• redmine and JIRA
• gitlab
• Many CRM systems

... etc. The usual information that you will need to provide for these programs when you are setting them up for LDAP access to your new directory server are:

• Hostname -- mysite.com or whatever your directory server host name is.
• Port -- 636 and you will need to check the "SSL" or "LDAPS" options.
• Base DN -- this was set by you when you ran the configuration script.
• Search DN -- leave this empty.
• Search Password -- leave this empty.
• LDAP Filter -- you probably want to use something like objectClass=inetOrgPerson
• Mapping attributes. You may or may not need to do this but some attributes that you may need to provide are:
  • Login Attribute: uid
  • First Name Attribute: givenName
  • Surname Attribute: sn
  • Email Attribute: mail
• Active Directory Mode -- what this means may depend on your application but some applications use this to turn on a whole bunch of options. You should turn this off.

Extra Software

There are some extra applications that I find useful to install on a Directory Server. These are:

• Self Service Password -- this application allows a simple interface for users to change their password.
• phpLDAPAdmin -- a web application to add/update/remove directory entries, gives you full management over your directory server without having to use the directory server console. You can install this directly from the EPEL repository like this:

yum -y install phpldapadmin

After installing you may want to trim down the templates that are in /usr/share/phpldapadmin/templates/creation/ I generally remove all of them except the following:

• alias.xml
• inetOrgPerson.xml
• organizationalRole.xml
• ou.xml
• posixAccount.xml
• posixGroup.xml
• simpleSecurityObject.xml

Let's Encrypt SSL Recipes -- Mail Server (sendmail and dovecot)

In part 1 of this series I showed how to obtain a free SSL certificate from Let's Encrypt and use the certificate to secure a website using Apache.

The same certificate can be used to secure a mail server such as dovecot, and a mail transfer agent such as sendmail.  I will show dovecot first because it's the easiest.

The dovecot configuration files are usually in /etc/dovecot with the master configuration file being /etc/dovecot/dovecot.conf.  In fact on most distributions you shouldn't edit this file, instead you should edit the files in /etc/dovecot/conf.d/. The one file you need to edit is /etc/dovecot/conf.d/10-ssl.conf on Red Hat and CentOS distributions.

The lines that you need to set in this file are as follows:

ssl_cert = </etc/letsencrypt/live/mysite.com/fullchain.pem
ssl_key =  </etc/letsencrypt/live/mysite.com/privkey.pem
ssl_protocols = !SSLv2 !SSLv3
ssl_cipher_list = ALL:!LOW:!SSLv2:!EXP:!aNULL:!RC4::!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS

As before, replace mysite.com with the actual site name of your web site, the site name that has been used to obtain the SSL certificate.  Note that you need to give the path to the fullchain.pem file obtained by certbot and not just the cert.pem file.

The site name should be the same as server name that you configure in your mail client to access this server.  For example if you're configuring Thunderbird you should set this as the Server Name under Server Settings in the Account Settings. Don't use another alias or CNAME, use the exact server name as registered on the certificate.

The sendmail Mail Transfer Agent (MTA) usually used for sending mail is a bit fussier about the certificates.  It can't use the certificates from the Let's Encrypt path because sendmail insists on certain permissions on the key file -- it must not be world readable.  So therefore in my cron script to renew the certificates I add some extra lines to copy the certificate and key elsewhere, and put some permissions on it that sendmail will be happy with:

cat /etc/letsencrypt/live/mysite.com/privkey.pem \
    /etc/letsencrypt/live/mysite.com/cert.pem > \
    /etc/pki/tls/certs/sendmail.pem
chmod 0600 /etc/pki/tls/certs/sendmail.pem
cat /etc/letsencrypt/live/mysite.com/chain.pem > /etc/pki/tls/certs/chain.pem


The above lines will put a copy of the combined key and certificate in /etc/pki/tls/certs as well as the CA chain file.

The next step is to configure sendmail.  I do this by editing the sendmail macro file (sendmail.mc) from the sendmail-cf package on CentOS:

cd /usr/share/sendmail-cf/cf
vi sendmail.mc

Add these lines to the macro file:

define(`confCACERT_PATH',`/etc/pki/tls/certs')dnl
define(`confCACERT',`/etc/pki/tls/certs/chain.pem')dnl
define(`confSERVER_CERT',`/etc/pki/tls/certs/sendmail.pem')dnl
define(`confSERVER_KEY',`/etc/pki/tls/certs/sendmail.pem')dnl
DAEMON_OPTIONS(`Port=smtp, Name=MTA')dnl
DAEMON_OPTIONS(`Port=smtps, Name=TLSMTA, M=s')dnl

If you're configuring sendmail for TLS then it's probable that you also are going to use SMTP Authentication.  To do that you will need these additional lines in your macro file:

define(`confAUTH_OPTIONS', `A')dnl
define(`confAUTH_MECHANISMS', `LOGIN PLAIN')dnl
TRUST_AUTH_MECH(`LOGIN PLAIN')dnl

Note that "LOGIN" and "PLAIN" authentications are mostly OK here -- because although it's plain text authentication it occurs in a TLS encrypted channel.  You will also need to start the SASL authentication daemon:

service saslauthd start
chkconfig saslauthd on

You will probably also want to set up Fail2Ban on your server as well because once you have sendmail authentication turned on you will receive many brute force attempts to hack your passwords.

Rebuild your macro file using make sendmail.cf and copy this to your normal sendmail.cf location.

I will update this for postfix at some point.

Let's Encrypt SSL Recipes -- Renewing certificates

In the previous post I explained how to obtain a Let's Encrypt SSL certificate for free.

Let's Encrypt certificates, when they are issued, are only valid for 90 days.  However it costs nothing to renew them and the process for doing so is easy.

You can run the certbot program daily or weekly (your choice) to automatically renew the certificates when they are ready for renewal.  You can create a cron script to do this for you automatically:

cat << EOFRENEW > /etc/cron.weekly/certbot-renew
#!/bin/sh

/root/bin/certbot-auto renew --quiet
EOFRENEW
chmod +x /etc/cron.weekly/certbot-renew

This will create a script that will run weekly and renew any certificates that need renewing.

Once the certificate has been renewed you will need to restart Apache or nginx (or any other programs using the certificate) so you need to plan a short outage window to do that.

Let's Encrypt SSL Recipes -- HTTPS with Apache or Nginx

This post is aimed at people who run their own web servers.  These days there is no real excuse for not using HTTPS instead of HTTP on a web site, and the reasons for switching from HTTP to HTTPS are many:

• Traffic to and from HTTPS sites is encrypted, HTTP traffic is not encrypted.
• Google and other search engines rank HTTPS web sites more highly than HTTP web sites.
• New versions of Chrome and some other web browsers will soon start displaying warnings when accessing a HTTP web site.

With the advent of the Let's Encrypt service which offers free SSL certificates for all web sites, there is no no real excuse for using HTTP instead of HTTPS -- using HTTPS is free and easy.

I'll start by providing some simple recipes to obtain Let's Encrypt certificates and install them on a web server -- then I'll follow up with some more posts about using Let's Encrypt certificates for other reasons.

On a recent Linux server, it's now possible to use a simple recipe to install the Let's Encrypt certbot program, which creates certificates:

wget --no-check-certificate https://dl.eff.org/certbot-auto
chmod +x certbot-auto
mkdir -p /root/bin
mv certbot-auto /root/bin

You can run certbot and create the certificate using this recipe:

/root/bin/certbot-auto certonly --webroot --webroot-path /var/www/html

This should work on most recent Linux variants -- Ubuntu, Debia, and Red Hat or CentOS.  Replace /var/www/html in the above recipe with your actual web root.  Certbot will ask you some questions (including your site name) and then create the certificate after checking that the site actually points to the server that you're running on.

The next thing is to update your Apache configuration (the default web server on Red hat and CentOS -- I will get to nginx later).

In your Apache virtualhost configuration you will need some lines like this:

    SSLEngine on
    SSLCertificateFile /etc/letsencrypt/live/mysite.com/cert.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/mysite.com/privkey.pem
    SSLCACertificateFile /etc/letsencrypt/live/mysitecom/chain.pem
    SSLProtocol all -SSLv2 -SSLv3
    SSLHonorCipherOrder on
    SSLCipherSuite "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH EDH+aRSA !RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS" 
    SetEnvIf User-Agent ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0

    # HSTS
    Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains;" 

In the above recipe replace mysite.com with your site name.

I have included a SSLCipherSuite line that should work for most people -- on reasonably recent Apache 2.2 or 2.4 versions.  If you want to be more specific about which cipher suites you want to support then try the Mozilla SSL Configuration Generator which will give you various options for most versions of Apache and nginx.

Note I have also included a line for HSTS which your site should support if you are running an HTTPS only site.

The next thing to do is to redirect HTTP to HTTPS.  This is done using the following lines in your Apache configuration:

    # Redirect HTTP to HTTPS
    RewriteEngine On
    RewriteCond %{HTTPS} !=on
    RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R=301,L]

For nginx there are actually a lot of recipes around on the internet about how to incorporate Let's Encrypt certificates.  Here are the nginx settings that I use.  All of this goes into a configuration file which on CentOS could be something like /etc/nginx/conf.d/ssl.conf:

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    # certs sent to the client in SERVER HELLO are concatenated in ssl_certificate
    ssl_certificate /etc/letsencrypt/live/mysite.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mysite.com/privkey.pem;
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets off;

    # Diffie-Hellman parameter for DHE ciphersuites, recommended 2048 bits
    ssl_dhparam /etc/ssl/certs/dhparam.pem;

    # intermediate configuration. tweak to your needs.
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS';
    ssl_prefer_server_ciphers on;

    # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
    add_header Strict-Transport-Security max-age=15768000;

    # OCSP Stapling ---
    # fetch OCSP records from URL in ssl_certificate and cache them
    ssl_stapling on;
    ssl_stapling_verify on;

    ## verify chain of trust of OCSP response using Root CA and Intermediate certs
    ssl_trusted_certificate /etc/letsencrypt/live/mysite.com/chain.pem;

    #resolver <IP DNS resolver>;

    # Load configuration files for the default server block.
    include /etc/nginx/default.d/*.conf;

    location / {
        ## Put location specific configuration here.
    }
}

To generate the dhparam.pem file referred to in the configuration above, run this command:

openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048

Finally, you should test the HTTPS security component of your web site using the SSL Labs SSL Server Test tool.  Hopefully you should get an A+ rating on this tool, if not then note any issues that the tool raises and fix them.

Crash Handling

I wrote a thing for the company blog:

Crash Handling

Seriously, though, displaying a stack trace to your users is poor form. It's the web site equivalent of the Windows "Blue Screen of Death".  However stack traces are useful diagnostics tools for the developer or DevOps team.

Frequently they occur in unexpected circumstances -- database down, disk full, disk not writeable, disk error, network failure, etc. Even in the most trying of circumstances try to log something useful somewhere (be aware that your normal logging procedure may not work, so have a fallback), and try to display something less unfriendly to the user, such as redirecting them to a "maintenance" page or server status blog.  Try to keep your customers informed -- don't just leave them hanging. Status blogs are good practice because it may be some time before your networks, systems or DevOps teams have the problem diagnosed and resolved.