Configuring Sendmail for STARTTLS

By Weldon Whipple <weldon@whipple.org>


Contents

  1. Introduction
  2. Prerequisites
  3. Preparing to Generate Certificates
  4. Issuing Certificates for the Sendmail Server(s)
  5. Placing the Certificates and Keys on the Sendmail Server(s)
  6. Modifying the sendmail.cf File
  7. Verifying the Configuration
  8. Interpreting Mail Log Entries
  9. Selected Resources

1. Introduction

This document tells how I currently implement STARTTLS on my sendmail servers. Hopefully it will give you ideas of things you can do (and maybe shouldn't do). Feel free to send comments, suggestions and corrections--and to set me straight if you think I don't understand something.

After investigation, thought and some trial and error, I have arrived at the following goals for my STARTTLS setup:

  1. Provide encryption of plaintext usernames and passwords sent during an SMTP AUTH negotiation for relaying e-mail through my server. Be able to require AUTH encryption--either through an encrypted AUTH mechanism or through a STARTTLS "tunnel."
    I use Cyrus SASL version 2 (from Carnegie Mellon University) to control relaying with AUTH.
  2. Facilitate the encryption of incoming mail submissions from remote clients to the sendmail server.

My setup doesn't accomplish the following:

  1. It does not use commercial SSL/TLS certificates (certs) to verify the identity of sendmail servers.
    I have had only limited success getting e-mail clients and other servers to verify the one commercial SSL certificate I purchased last year. Investigation seems to indicate that the purchased certificate's "purpose" is specifically for https transactions. In conversations between two sendmail servers, I have never seen TLS verification succeed with the commercial cert.

    Many (user) e-mail clients don't recognize commercial certs. (Those clients that do recognize commercial certificates are generally closely associated with a web browser that also recognizes the certificate.) Even with commercial certificates, it is likely--when the client sends TLS-encrypted e-mail to the STARTTLS-enabled sendmail server for the first time--that the client software will display a dialog that reports an unknown TLS certificate.

    If you decide to use commercial certificates, you can skip sections 3 and 4 of this document.

  2. It does not guarantee that e-mail opened by the recipient has been encrypted at every step of its transmission from its origin.
    This is a limitation of the SMTP protocol when used on the Internet at large. In fact, since only a miniscule number of public mail servers implement STARTTLS, it is highly unlikely that any given e-mail will remain encrypted during its journey from sender to recipient. (Configured appropriately, private networks can be configured to enforce encryption at every "hop" from sender to recipient.)

This document focuses on basic STARTTLS configuration. It does not (for example) discuss the role of the access database in STARTTLS negotiations, DH parameter tweaks, and other STARTTLS sendmail topics.

2. Prerequisites

These instructions assume that

  1. you are running a recent version of sendmail on Linux, FreeBSD, or some other Unix-like operating system.
    All of my sendmail servers are running sendmail 8.12.11 or 8.13.0, on FreeBSD 4.x or 5.2.1.
  2. you have installed the latest OpenSSL on your server.
    At the time of this writing, I use OpenSSL 0.9.7d.
  3. you have installed Cyrus SASL version 2.
    On FreeBSD, you can install from ports with the following command sequence:
    
    # cd /usr/ports/security/cyrus-sasl2
    # make
    # make install
    # cd /usr/ports/security/cyrus-sasl2-saslauthd
    # make
    # make install
    
  4. Your sendmail executable is compiled to support both SASLv2 and STARTTLS.
    To verify that your sendmail supports SASLv2 and STARTTLS, issue the command: "sendmail -d0.1 < /dev/null" from the command line. The "Compiled with" section of the output should include both "SASLv2" and "STARTTLS".

A future version of this document might include more information on how to achieve the prerequisites. For the present, let's proceed with configuration.

3. Preparing to Generate Certificates

Before creating certificates for your sendmail servers, you need to become a certificate authority (CA). Becoming a certificate authority means creating a special certificate authority certificate, which asserts that you have the authority to issue certificates for others (your e-mail server(s) in this case.)

As root, issue the following commands. (Many thanks to Gregory Neil Shapiro, whose instructions are the basis of the following steps.)

If you are generating certs for multiple servers and/or e-mail clients, you should do it on a computer that is fairly secure--some go so far as to say that the computer shouldn't be connected to a network at all. After generating the certs for each computer, I use the scp (secure copy) command to encrypt and copy the certificates and keys to each mail server. (When I am finished creating and distributing certificates to the sendmail servers, I write the certificates, keys, and other working files to a CD-R, then delete all but the files needed for daily operation.)
  1. cd
    This command changes to root's home directory, beneath which the generated certificates and keys will be created.
  2. mkdir CA
  3. cd CA
    We will be working in the CA ("certificate authority") directory, beneath root's home directory.
  4. mkdir certs crl newcerts private
  5. echo "01" > serial
  6. cp /dev/null index.txt
    The above create some working directories and initialize some files that we will use in generating certs. The "serial" file contains the serial number of the next certificate to be generated. (The openssl command to sign a cert reads the serial number, then uses that number to create an "archive" of certs signed by the certificate authority, stored in the newcerts directory.)
  7. cp /usr/local/openssl/openssl.cnf.sample openssl.cnf
    This copies the sample OpenSSL configuration file (shipped with the OpenSSL distribution) to the current directory. If OpenSSL is installed somewhere else on your server, change the first argument of the cp command appropriately.
  8. vi openssl.cnf

    Edit the openssl configuration as indicated in the table below.

    (Actually, I prefer to edit with emacs, but use the more ubiquitous vi editor for this example.)
    AttributeOriginal ValueChange to
    dir ./demoCA .
    Under the [ policy_match ] section:
    stateOrProvinceName match optional
    organizationalUnitName optional match
    commonName supplied optional
    Under the [ req_distinguished_name ] section:
    commonName Common Name (eg, YOUR name) Common Name (eg, host/domain name)

  9. openssl req -new -x509 -keyout private/cakey.pem -out cacert.pem -days 1825 -config openssl.cnf
    The above command is what makes you a certificate authority. The CA certificate will end up in the file cacert.pem; the CA's private key will be in private/cakey.pem. I chose to make the certificate good for about 5 years (1825 days).

    Here is my interaction with the openssl req command (my input in bold italics):

    # openssl req -new -x509 -keyout private/cakey.pem -out cacert.pem -days 1825 -config openssl.cnf
    Generating a 1024 bit RSA private key
    ..++++++
    .............................................................++++++
    writing new private key to 'private/cakey.pem'
    Enter PEM pass phrase:[Type pass phrase here]
    Verifying - Enter PEM pass phrase:[Type pass phrase here]
    -----
    You are about to be asked to enter information that will be incorporated
    into your certificate request.
    What you are about to enter is what is called a Distinguished Name or a DN.
    There are quite a few fields but you can leave some blank
    For some fields there will be a default value,
    If you enter '.', the field will be left blank.
    -----
    Country Name (2 letter code) [AU]:US
    State or Province Name (full name) [Some-State]:. [i.e., type a period]
    Locality Name (eg, city) []:
    Organization Name (eg, company) [Internet Widgits Pty Ltd]:Whipple Certificate Authority
    Organizational Unit Name (eg, section) []:
    Common Name (eg, host/domain name) []:
    Email Address []:
    #
    
    Note that I intentionally omitted the Common Name, as I have noticed that commercial CA's generally don't have common names. Some SSL/TLS verification routines object to CA certificates that have common names.

    If you want to see what your new CA cert looks like, issue this command:

    openssl x509 -noout -text -in cacert.pem
    (If you want to see what it really looks like, I suppose you could try:
    less cacert.pem
    :-)

4. Issuing Certificates for the Sendmail Server(s)

Now that you are a Certificate Authority, you are prepared to issue certificates for your sendmail servers.

Continuing as root (in the same directory as above), issue the following command sequence for each sendmail server:

  1. openssl req -nodes -new -x509 -keyout servername.pem -out servername.pem -days 365 -config openssl.cnf

    Here is my interaction with the openssl req command for one of my own servers (my input in bold italics):

    # openssl req -nodes -new -x509 -keyout gabriel.pem -out gabriel.pem -days 365 -config openssl.cnf
    Generating a 1024 bit RSA private key
    ..................++++++
    .....++++++
    writing new private key to 'gabriel.pem'
    -----
    You are about to be asked to enter information that will be incorporated
    into your certificate request.
    What you are about to enter is what is called a Distinguished Name or a DN.
    There are quite a few fields but you can leave some blank
    For some fields there will be a default value,
    If you enter '.', the field will be left blank.
    -----
    Country Name (2 letter code) [AU]:US
    State or Province Name (full name) [Some-State]:. [i.e., type a period]
    Locality Name (eg, city) []:
    Organization Name (eg, company) [Internet Widgits Pty Ltd]:Whipple Website
    Organizational Unit Name (eg, section) []:
    Common Name (eg, host/domain name) []:gabriel.whipple.org
    Email Address []:hostmaster@whipple.org
    #
    
    The above generates a certificate request, storing the unsigned certificate and private key in the same file, named servername.pem (Substitute your server's name for servername in the command.)

    If you were purchasing a certificate from a commercial certificate authority, they would have you do something like the above to generate a certificate "request." They you would send them the certificate request. They would sign it and return it to you.

    If you prefer, you can specify different files for the -keyout and -out options. (If you were generating the certificate request for a commercial CA, you would want to send them only the request--you wouldn't want them to get their hands on your private key!) In our case, it doesn't hurt to put both the request and the private key in the same file: OpenSSL "tags" the sections of the file, so that commands that use the file for input can find what they are looking for (e.g., the request or the key).

    In the following commands, your certificate authority signs the above certificate (for free!).

  2. openssl x509 -x509toreq -in servername.pem -signkey servername.pem -out tmp.pem
  3. openssl ca -config openssl.cnf -policy policy_anything -out servername-cert.pem -infiles tmp.pem

    The above command will prompt you for the CA's pass phrase (the one you entered twice when you created the CA certificate earlier--to make you a certificate authority). Then it will display the certificate that you are signing and ask you if you want to sign the certificate. Respond with 'y'. When it asks you if you want to commit the certified request, respond with 'y'.

    The files that sendmail will need (later on in these instructions) are servername.pem (for the private key that it contains) and servername-cert.pem (the server's signed certificate).

    Notice that we specify "policy_anything" when we sign the certificate, so that it can be used for e-mail. (Commercial CAs generally use a more restrictive policy--which prevents sendmail and many e-mail clients from using them for identity verification.)

When you finish generating and signing certs for all your servers, remove the file tmp.pem:

rm -f tmp.pem
If you want to see what a server cert looks like, issue this command:
openssl x509 -noout -text -in servername-cert.pem

5. Placing the Certificates and Keys on the Sendmail Server(s)

Now that you have signed certificates and keys for your sendmail servers, place them on the mail servers, as follows.

  1. Each sendmail server will need the following files (for starters):
    cacert.pem
    This is your certificate authority's certificate. (Its public key is imbedded in the certificate. Do not store the CA's private key--private/cakey.pem in the above instructions--on the sendmail server!)
    servername-cert.pem
    (servername is replaced by your server's name that you used when generating the server cert above.) This is your sendmail server's certificate (including its public key).
    servername-key.pem
    This is the file servername.pem that you generated above when you generated a certificate request. It includes two parts: the sendmail server's private key and the original (unsigned) certificate request. I generally rename it to servername-key.pem, then use my favorite editor to delete all but the private key.
  2. The above files are commonly stored in the directory /etc/mail/certs. Make sure that the directory is owned by root. The files themselves (within /etc/mail/certs) must be owned by root and accessible only to root.
    The following commands will ensure that the above requirements are met:
    
    # cd /etc/mail/certs
    # chown root *
    # chmod 0400 *
    
  3. Create a hashed symbolic link to the CA certificate by issuing the following commands:
    
    # cd /etc/mail/certs
    # ln -s cacert.pem `openssl x509 -noout -hash < cacert.pem`.0
    
    The above reads the cacert.pem file and creates an 8-character cryptographic hash of its contents, which is used as the filename (with '.0' appended) that links to the CA certificate. The "ls -l" command (in the /etc/mail/certs directory) will now show something like:
    
    # ls -l
    total 8
    -r--------  1 root  wheel  1139 Mar 15 15:17 cacert.pem
    lrwxr-xr-x  1 root  wheel    18 Mar 16 16:42 c567b670.0 -> cacert.pem
    -r--------  1 root  wheel  3462 Mar 15 15:13 gabriel-cert.pem
    -r--------  1 root  wheel  2018 Mar 15 15:13 gabriel-key.pem
    #
    

    During an SSL handshake's certificate exchange, sendmail will compute the the hash of the received CA cert's public key, append '.0' to it, then compare it to its own copy of the CA cert's public key. (This is probably an over simplification, but you get the idea.)

6. Modifying the sendmail.cf File

With certificates and keys in place, you are ready to modify the sendmail.cf file. Using your favorite editor, open the macro configuration (mc) file you use to generate your server's sendmail.cf. Add the following directives (in the area of the file that has other "define" statements):

define(`CERT_DIR', `/etc/mail/certs')
define(`confCACERT_PATH', `CERT_DIR')
define(`confCACERT', `CERT_DIR/cacert.pem')
define(`confSERVER_CERT', `CERT_DIR/servername-cert.pem')
define(`confSERVER_KEY', `CERT_DIR/servername-key.pem')
define(`confCLIENT_CERT', `CERT_DIR/servername-cert.pem')
define(`confCLIENT_KEY', `CERT_DIR/servername-key.pem')

The defined macros have the following significance:

  1. CERT_DIR. A "convenience" definition referenced by subsequent lines.
  2. confCACERT_PATH. Sendmail looks in this path for the CA Cert hashed file names.
    When sendmail attempts to verify the host at the other end of an SMTP connection, it creates a hash of the CA information in the presented cert, then checks for a file matching that name in confCACERT_PATH to verify the authenticity of the host's certificate.
  3. confCACERT. The file containing the certificate of the Certificate Authority that issued this sendmail server's certificate.
  4. confSERVER_CERT and confCLIENT_CERT typically refer to the same cert--this server's certificate used when acting as a server and when acting as a client.
  5. confSERVER_KEY and confCLIENT_KEY are the corresponding private keys.

If your sendmail server relays e-mail for users who authenticate with SMTP AUTH, you can require that their user name and password be encrypted by including the following line in the mc file:

define(`confAUTH_OPTIONS',`A,p,y')dnl
If you support both encrypted and plaintext SMTP authentication mechanisms, the above will cause the server to offer only encrypted authentication mechanisms until after STARTTLS encryption is negotiated. After STARTTLS has been negotiated, the server will add PLAIN and LOGIN to the AUTH mechanisms it offers in response to EHLO.

By default, sendmail will ask e-mail clients for their SSL/TLS certificates. Since almost no clients have personal TLS certificates, you can tell sendmail to skip the request with the line:

define(`confTLS_SRV_OPTIONS', `V')

After editing your mc file, generate a sendmail.cf. With the new sendmail.cf in place, restart the sendmail daemons.

The sendmail distribution says to use "./Build <cfname>" to generate a cf file from an mc file. On FreeBSD, the command sequence "cd /etc/mail; make; make install; make restart" will generally suffice. The exact command used to build a sendmail.cf file is beyond the scope of this document.

7. Verifying the Configuration

After restarting the sendmail daemons with the new sendmail.cf file, be sure to check the mail log and look for error messages from sendmail, indicating that sendmail might not be working properly. You will also want to try sending some test e-mail and verify that it is delivered.

If you configured STARTTLS properly, you should see mention of STARTTLS in some of the sendmail log entries. It is probably a good idea to telnet to port 25 or 587 of the server just configured, and make sure that it offers STARTTLS in response to the EHLO command. Here is a sample telnet session (my input in bold italics):


weldon@zadkiel:~% telnet gabriel.whipple.org 587
Trying 166.70.98.35...
Connected to gabriel.whipple.org.
Escape character is '^]'.
220 gabriel.whipple.org ESMTP Sendmail 8.12.11/8.12.11; Thu, 18 Mar
2004 13:30:08 -0700 (MST)
ehlo zadkiel.whipple.org
250-gabriel.whipple.org Hello zadkiel.whipple.org
[166.70.98.36], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-STARTTLS
250-DELIVERBY
250 HELP
quit
221 2.0.0 gabriel.whipple.org closing connection
Connection closed by foreign host.
weldon@zadkiel:~%

If you see the line 250-STARTTLS in response to the SMTP EHLO command, it indicates that STARTTLS is probably configured correctly.

8. Interpreting Mail Log Entries

With STARTTLS configured, you will probably see lines like the following in the mail log file:

Mar 18 00:24:55 gabriel sm-mta[19377]: STARTTLS=server, relay=[204.228.146.20], 
version=TLSv1/SSLv3, verify=NO, cipher=DHE-RSA-AES256-SHA, bits=256/256

If you see the cipher and bits equates, it indicates that encryption is occuring.

The verify equate caused some confusion the first time I saw it. Here are possibly values that you might see for the verify equate:

OK
Verification succeeded.
OK is quite common when two of my sendmail servers--each configured with certificates signed by my private certificate authority--are communicating with each other. (This happens when my secondary MX server relays mail to my primary MX server.)
NO
No cert presented.
This occurs when my client--which doesn't have a private certificate--submits e-mail to my sendmail server. It has no certificate to present, so it doesn't present one.
NOT
No cert requested.
One reason for this to occur is the inclusion of a line define(`confTLS_SRV_OPTIONS', `V') in the sendmail macro configuration file (which prevents sendmail from asking for client certificates).
FAIL
Cert presented but could not be verified, e.g., the cert of the signing CA is missing.
This value might be the most common value for the verify equate (?). As long as the cipher and bits equates have meaningful values, encryption is still occuring.
NONE
STARTTLS has not been performed.
TEMP
Temporary error occurred.
PROTOCOL
Protocol error occurred (SMTP level).
I don't recall ever seeing PROTOCOL or SOFTWARE values in the log file.
SOFTWARE
STARTTLS handshake failed.

9. Selected Resources

Bryan Costales with Eric Allman, Sendmail, 4th ed., (O'Reilly, 2007).
The latest edition of the "Bat Book"--the most authoritative source of information on sendmail. Updated to cover sendmail version 8.14. STARTTLS coverage starts on p. 202.
SMTP STARTTLS in sendmail/Secure Switch
This was my first source of information for STARTTLS. Much of my tutorial is based on ideas from this earlier one.
Very brief introduction to create a CA and a CERT
The very brief introduction was the basis of most of sections 3 and 4 of my tutorial.
Craig Hunt, Sendmail Cookbook (O'Reilly, 2004).
Chapter 8 of this recent book has 13 STARTTLS "recipes" that go beyond the basics of this tutorial.
John Viega, Matt Messier & Pravir Chandra, Network Security with OpenSSL (O'Reilly, 2002).
The best source information I know of on OpenSSL's implementation of TLS and SSL (except, perhaps for http://www.openssl.org and Google).
Eric Rescorla, SSL and TLS: Designing and Building Secure Systems (Addison-Wesley, c2001).
Probably the single best source of information on SSL and TLS. The code examples in "C" use OpenSSL. Those in Java use PureTLS.
http://groups.google.com
Google is your friend! Use it to search for STARTTLS in the sendmail newsgroup. Most of your questions have been answered before.
Stephen Thomas SSL and TLS Essentials: Securing the Web (Wiley, 2000).
Gives a good overview of SSL and TLS. This book is probably OK if you don't want to get into implementation or configuration of SSL and TLS.
http://www.sendmail.org
Everything about sendmail is here. Don't forget the search the FAQs. Claus Assmann's tilde directory is loaded with useful information.

Don't overlook the "Sendmail Configuration Files" README (in the sendmail distribution at ./cf/README) or the Sendmail Installation and Operation Guide. The latter is in the sendmail distribution at ./doc/op.

If you can't read the Operation Guide, try issuing the command (with ./doc/op as your current directory): make op.pdf to create a nicely formatted PDF file. If that doesn't work, then try make op.txt for a text version of the file.