My Experiences (So Far) with STARTTLS and Sendmail

By Weldon Whipple <>


1. Prolog

As a software engineer assigned to sendmail at a large ISP, I decided last year to get serious about authentication, encryption and e-mail security in general.

With the help of the sendmail newsgroup (comp.mail.sendmail), the documentation that comes with the sendmail distribution, and Google, I was able to understand SMTP AUTH enough to roll out a limited implementation in 2002. Our default rollout uses AUTH's PLAIN and LOGIN mechanisms--probably the least desirable--primarily because Microsoft's Outlook Express supports only the LOGIN mechanism.

"Real" security gurus frown on PLAIN and LOGIN, because they send usernames and passwords in clear text. (Well, almost clear text--they use the trivial base64 encoding to convert them to a transmission-safe subset of ASCII characters.) SMTP AUTH supports encrypted authentication mechanisms (including CRAM-MD5, DIGEST-MD5, Kerberos, etc.), but it is risky for a large, general-audience ISP to require them.

2. STARTTLS to the Rescue?

Sending user names and passwords across the Internet in plain text is bad. Fortunately, sendmail provides another method of encrypting not only the user name and password, but the entire mail transmission, using Transport Layer Security (TLS)--which is only slightly different from its predecessor, Secure Socket Layer (SSL) version 3. Sendmail uses the STARTTLS command within an SMTP conversation to signal the beginning of an encrypted TLS conversation. Better still, Microsoft's Outlook Express (and many other clients) support STARTTLS.

STARTTLS can do many things, including:

  1. Verify the identity of the client and/or server in an e-mail transmission.
  2. Encrypt mail transmissions (even when the identity can't be verified).
  3. Authenticate a user for relaying through a mail server (similar to the functionality of SMTP AUTH).
  4. Encrypt transmissions between two mail servers on the Internet. (This is really the same as 2 above, but I mention it again so you won't overlook it.)

Although sendmail's STARTTLS can do the above, RFCs state that no "public" e-mail server should require the above as a prerequisite for delivering mail. (Private mail servers--within a corporate network, for example--can tighten encryption and verification requirements and refuse connections or mail to/from any client or server that can't prove its identity with a certificate.) Because the SMTP protocol is a "store and forward" protocol and e-mail delivery can involve multiple servers, even if a sending client can authenticate and encrypt e-mail transmission to the "first-hop" mail server, it is almost impossible for the recipient to trust the sender's authenticity and assume the untampered delivery of the e-mail.

Although SMTP's TLS implementation cannot always provide the kind of verification that HTTP servers can, SMTP's STARTTLS can safely encrypt a clear text user name and password sent in an SMTP AUTH conversation between a client and a mail server.

3. How STARTTLS Works

3.1. When Sendmail is a Server

Clients (which can be other mail servers or a person sending mail to a sendmail server) can issue the EHLO command during an SMTP session to determine whether the server supports STARTTLS. If the server supports STARTTLS, it will include it in the list of commands the client can issue.

If the server supports STARTTLS and the client wants to use it, the client next issues the STARTTLS command (without arguments). Sendmail responds that it is ready to start TLS. Negotiations proceed.

In the rare case that a client presents an SSL certificate, sendmail compares the certificate's certificate authority (CA--the issuer of the certificate, who can vouch for the client's identity) with the CAs it knows about. It may also try to match the certificate's common name with the client's host name (or other identifier). Sendmail might also check its access datbase to see if the client certificate is listed there. If acceptable to the server, sendmail will allow the conversation to proceed. Actually, most mail servers will allow the conversation to proceed even if unable to verify the client's certificate. (In the mail log entry for the TLS exchange, it will include something like "verify=FAIL" or "verify=NO"--or one of several other alternatives--to indicate that verification wasn't OK. The relatively rare "verify=OK" indicates that the client's identity could be verified.)

Even if the client's identity can't be verified, sendmail will attempt to encrypt the transmission--assuming that the client is capable. In that case, the mail log (and probably a Received: header) will mention something like "cipher=DHE-RSA-AES256-SHA, bits=256/256" to indicate that encryption occurred.

After an encrypted session has been negotiated, a client that wants to relay through the server can use SMTP AUTH to verify its identity. It doesn't matter if the PLAIN or LOGIN authentication mechanisms are used--STARTTLS will encrypt the user name and password!

3.2. When Sendmail is a Client

As sendmail receives mail from its originator, it will often need to forward it to the recipient's mail server. As it contacts the "next" mail server, sendmail assumes the role of client.

If the "next" mail server offers "STARTTLS" as one of its allowed commands (highly unlikely at present!), our sendmail (which has been compiled to support STARTTLS) will always accept the offer and issue a STARTTLS command--even if it has no certificates configured! When this occurs, it is very likely that encryption will occur.

The log entry for the STARTTLS exchange will include either "STARTTLS=client" or "STARTTLS=server" (in addition to the version, verify, cipher and bits equates) to show sendmail's role during the transaction.

4. Configuring Sendmail to Use STARTTLS

4.1 The file

Assuming that the sendmail binary has been compiled to support STARTTLS, the following directives (in the mc file used to generate a file) tell sendmail to use STARTTLS:

define(`CERT_DIR', `/etc/mail/certs')
define(`confCACERT_PATH', `CERT_DIR')
define(`confCACERT', `CERT_DIR/whipple.orgCA-cert.pem')
define(`confSERVER_CERT', `CERT_DIR/')
define(`confSERVER_KEY', `CERT_DIR/')
define(`confCLIENT_CERT', `CERT_DIR/')
define(`confCLIENT_KEY', `CERT_DIR/')
  1. CERT_DIR. The choice of /etc/mail/certs as the directory for storing sendmail certs is a convention used in sendmail documentation. (It is a subdirectory of the current default /etc/mail directory that stores, databases, etc.) The files in the CERT_DIR should not be group or other readable or writable. (A rights mask of 0600 or 0400 is appropriate.) You can place certs in some other directory, as long as their rights mask is set appropriately.
    Hint: If you have just configured sendmail to use STARTTLS, but sendmail doesn't offer STARTTLS during the SMTP conversation, double check the rights masks of the certificates. If they are group or other readable (e.g., their rights mask is 0644 or 0444), try issuing the command chmod 0600 or chmod 0400 and restart sendmail.
  2. confCACERT_PATH. The path that stores the certificates of all the Certificate Authorities a sendmail server knows about. The file name that stores each certificate is an SSL hash of the certificate itself, with a ".0" suffix. The file containing the certificate for "C=US, O=VeriSign, Inc., OU=Class 1 Public Primary Certification Authority" (for example) is stored in a file named 2edf7016.0.

    (In practice, 2edf7016.0 is a symbolic link to the file vsign1.pem, as shipped in OpenSSL.)

    When sendmail attempts to verify the host at the other end of an SMTP connection, it creates a hash of the CA information in the presented cert, then checks for a file matching that name in confCACERT_PATH to verify the authenticity of the host's certificate. (Note: sendmail also checks the access database to see if it should verify the cert of the connecting host.)

  3. confCACERT. The file containing the certificate of the Certificate Authority that issued this sendmail server's certificate.
  4. confSERVER_CERT and confCLIENT_CERT typically refer to the same cert--this server's certificate used when acting as a server and when acting as a client.
  5. confSERVER_KEY and confCLIENT_KEY are the private keys that correspond to the sendmail server's certificates.

If your sendmail server will relay e-mail for users who authenticate with SMTP AUTH, you can require that the user name and password be encrypted by including the following line in the mc file:


(If you support both encrypted and plaintext SMTP authentication mechanisms, the above will cause the server to offer only encrypted authentication mechanisms until after STARTTLS encryption is negotiated. After STARTTLS has been negotiated, the server will add PLAIN and LOGIN to the AUTH mechanisms it offers.)

Almost no e-mail clients have SSL/TLS certificates. You can tell sendmail not to bother to ask for client certificates by including the line:

define(`confTLS_SRV_OPTIONS', `V')

4.2 The Access Database

Sendmail's access database supports numerous prefixes related to STARTTLS. Some of things you can do with access entries include:

These and other capabilities of the access database (relating to STARTTLS) are detailed on pages 422-28 of the 3rd edition of the O'Reilly sendmail book by Bryan Costales with Eric Allman.

4.3 DH Parameters

I've never had occasion to use the DHParameters option in a file. (If omitted, sendmail automatically handles the parameters, as far as I can tell.) The Diffie-Hellman algorithm (DH--introduced by Whitfield Diffie and Martin Hellman in 1976) is used by SSL to generate a "shared" (as opposed to public-key) secret from two numbers typically referred to as p and g (the "DH parameters"):

A prime number at least 512 or 1024 bits in length, with the property that (p-1)/2 is also prime.
The "generator"--either 2 or 5 in OpenSSL

If you want to specify DH parameters yourself, you can create a DH parameters file for sendmail with a command something like:

$ openssl dhparam -out dhparam.pem -2 

which will write the DH parameters in PEM format (the format needed by sendmail) to the file dhparam.pem. The example uses a generator (the g parameter) of 2 and generates a random 512-bit prime number. (If you add "1024" as a final argument to the command shown above, it will generate a random 1024-bit prime number.)

You can view the DH parameters as C code by issuing the command

$ openssl dhparam -in dhparam.pem -noout -C 

If you want (for some reason) to have sendmail use your dhparam.pem file, add a line like the following to the mc file you use to generate your


(Note: I haven't actually tried the above in my I hope it works!)

5. Certificate Authority Certificates

I purchased a GeoTrust certificate through for $39.00 and installed it on my web and sendmail servers. It has the following certificate authority:

C=US, O=Equifax, OU=Equifax Secure Certificate Authority

My browser accepts the cert if offered by apache, and my e-mail client software accepts it when I use it to connect to an outgoing sendmail server (and subsequently use SMTP AUTH to relay through the server). Both the browser and the mail clients I use have the Equifax certificate "compiled in."

I have been unsuccessful in getting other mail servers to verify the purchased certificate. I configured two sendmail servers to use the CA certificates that are shipped with OpenSSL (which sendmail uses for its TLS/SSL functionality):

  1. C=US, O=VeriSign, Inc., OU=Class 1 Public Primary Certification Authority
  2. C=US, O=VeriSign, Inc., OU=Class 2 Public Primary Certification Authority
  3. C=US, O=VeriSign, Inc., OU=Class 3 Public Primary Certification Authority
  4. C=AU, ST=Queensland, O=CryptSoft Pty Ltd, CN=Test PCA (1024 bit)
  5. C=ZA, ST=Western Cape, L=Cape Town, O=Thawte Consulting cc, OU=Certification Services Division, CN=Thawte Premium Server CA/
  6. C=ZA, ST=Western Cape, L=Cape Town, O=Thawte Consulting cc,
  7. OU=Certification Services Division, CN=Thawte Server CA/
  8. C=US, O=RSA Data Security, Inc., OU=Secure Server Certification Authority

Then I downloaded the Equifax CA cert from GeoTrust's web site and added it (with hashed symlinks) to the CACERT_PATH of both servers. Even so, I have not yet seen "verify=OK" appear in the maillog of either server when communicating between themselves using the Equifax certificate.

On the other hand, I have created my own CA cert, then generated and signed certs for both mail servers (following the instructions at "Very brief introduction to create a CA and a CERT"). Using those certs, the servers have reported "verify=OK".

(Note: With either purchased or self-signed certs, it is possible to add entries to the access database of the individual servers to explicitly list certificates and enable verification to occur. I have been able to get the Equifax cert to verify by specifying the CERTISSUER and CERTSUBJECT in the access database of the second sendmail server.)

6. "Purposeful" Purchased Certs

Why doesn't my second sendmail server recognize my first sendmail server's Equifax certificate? I think the answer is in certificate extensions.

6.1. Certificate Extensions ... Briefly

The primary standard for certificates is X.509 (described in RFC 2459), dating from 1988. Version 3 of X.509--introduced in 1996--added support for "extensions," which can indicate the certificate's "purpose(s)." Web browsers (and most e-mail client software as well, apparently) don't require extensions, but if certificates have any (indicating that they are version 3 X.509 certificates), browsers check them (each in their own way ...).

The X.509v3 standard defines 14 extensions and allows developers to define (essentially) unlimited special-purpose extensions for certificates. (See John Viega et al., Network Security with OpenSSL (O'Reilly, 2002), p. 49ff.)

Dr. Stephen N. Henson, Core Developer of the OpenSSL project, writes:

"Purpose checking is a critical aspect of certificate security. It determines if CAs are valid and if a certificate is being used for an authorised purpose.

"Without it anyone could use their user certificate as a CA or one of those "no checking freeware email" for something like object signing.

"X509_PURPOSE_ANY actually does disable purpose checking and is a big security hole unless used with extreme caution: it's there because there are some highly broken CAs about which it is unfortunately occasionally necessary to tolerate."

6.2 My Purchased Cert's Purposes

My RackShack cert's failure to verify between servers is likely related to its certificate extensions. Here is what I have observed so far.

  1. Increasing sendmail's log level on the second server to 15 causes the following message to appear in the maillog: "unsupported certificate purpose".
  2. OpenSSL's "verify" command (invoked with 7 different "purpose" values):
       # openssl verify -verbose -CAfile /etc/mail/certs/Equifax_Secure_CA.pem -purpose

    yields these results for a purchased vs. private cert:

    Purpose-value RackShack Cert Private Cert
    any OK OK
    sslserver OK OK
    sslclient unsupported OK
    nssslserver OK OK
    smimesign unsupported OK
    smimeencrypt unsupported OK
    crlsign unsupported OK
  3. The OpenSSL "x509" command confirm's the above:
       # openssl x509 -noout -purpose -in  [Private cert]
       # openssl x509 -noout -purpose -in      [Purchased]
    Certificate purpose Private Cert Purchased Cert
    SSL client : Yes No
    SSL client CA : No No
    SSL server : Yes Yes
    SSL server CA : No No
    Netscape SSL server : Yes Yes
    Netscape SSL server CA : No No
    S/MIME signing : Yes No
    S/MIME signing CA : No No
    S/MIME encryption : Yes No
    S/MIME encryption CA : No No
    CRL signing : Yes No
    CRL signing CA : No No
    Any Purpose : Yes Yes
    Any Purpose CA : Yes Yes
    OCSP helper : Yes [?]
    OCSP helper CA : No [?]

It appears that sendmail needs a certificate with one or more of the following purposes: sslclient, smimesign, smimeencrypt, or crlsign ... or possibly some other extension (in addition to [ns]sslserver and any). (I need to look at this some more.)

6.3 Should I Purchase a Certificate?

Probably not--at least not if it is to be used exclusively for a single SMTP server. It you need to purchase a certificate for a web server, you might be able to make a single purchased cert work for both the web server and your mail server.

7. Choosing a Common Name

If you use the command-line "openssl req" command to request a certificate for your mail server (see Very brief introduction to create a CA and a CERT), OpenSSL will prompt you for (among other things) a "Common Name." Most sources say to "embed the server['s] ... fully qualified domain name (FQDN) in the ... certificate's commonName field." (See John Viega et al., Network Security with OpenSSL (O'Reilly, 2002), p. 124.)

RFC 2487 ("SMTP Service Extension for Secure SMTP over TLS") offers the following advice:

A SMTP client would probably only want to authenticate an SMTP server whose server certificate has a domain name that is the domain name that the client thought it was connecting to.

All my initial tests used (and similar fully-qualified host names) as common names for server certificates. Most mail clients (the ones that aren't other servers) are able to verify those certs when issuing SMTP's STARTTLS command--assuming that the CA cert is generated "correctly."

There might be reasons to consider other common names, however:

7.1 Based on Left-Hand-Side of MX Record?

In chapter 10 of his SSL and TLS: Designing and Building Secure Systems, Eric Rescorla includes a discussion of MX records and TLS. At the bottom of p. 356 he writes:

Unfortunately, this sort of mail relaying [i.e. relaying from secondary MX servers to primary MX servers] presents a security challenge. Because we can't trust the DNS we can't know that the MX records we're getting haven't been forged. Thus, just as with CNAMEs, we need to check the recipient's domain (e.g., rather than the name of the mail exchanger (e.g., against the server's certificate when we connect.

This implies that it might be preferable to have's common name be "", rather than "" (More on this later.)

7.2 Wildcard Certs?

RFC 2818 ("HTTP over TLS") discusses the use of wildcarded DNs (section 3.1) with HTTP. Specifically,

Names may contain the wildcard character * which is considered to match any single domain name component or component fragment. E.g., * matches but not f*.com matches but not

Rescorla (pp. 308-9) notes that wildcarded domains "only work well if you have a number of virtual servers run by the same organization. You can't use it to represent (say) and because any wildcard that matched both would also match other domains not owned by the certificate-holder."

Wildcarded domains are legal, and are (apparently) recognized by web clients. They also work for several mail clients I have tried.

7.3 What Common Names Work for Clients?

(This section is still in progress.) It appears that many mail clients support all three of the naming patterns mentioned above:

Client Common Name in Mail Server Certificate *
Mozilla Verifies Verifies Verifies
Outlook Express Verifies Verifies Verifies
pine [future test] [future test] [future test]
Eudora [future test] [future test] [future test]

8. Clients

This section discusses e-mail clients that aren't also mail servers. The clients that your users use can affect your decisions when configuring STARTTLS.

8.1 Is STARTTLS Even Necessary?

If your clientele all use Eudora, and all you want is encrypted authentication with a user name and password, STARTTLS probably isn't necessary: Eudora's implementation of SMTP AUTH prefers CRAM-MD5 if the server offers it. Many other (non-Microsoft) e-mail clients also support encrypted authentication methods.

Although not a particularly popular (or secure) method of controlling relaying, some ISPs still offer POP before SMTP relaying. For those customers--especially if they don't care about encryption--STARTTLS might not be necessary.

Many ISPs allow e-mail relay based on IP addresses or blocks of addresses. (Sendmail's access database allows this, for example.) Clients in this type of environment also might not need STARTTLS.

If your clients want to send encrypted e-mail (and have it safe from intermediate "sniffers" along the say), they are probably better off with Pretty Good Privacy (PGP) or GnuPG. The more recent S/MIME also allows encryption of e-mail before it is submitted to a mail server. (It requires that both the sender and recipient have "user" SSL/TLS certificates.)

8.2 My Current Client Setup

This describes how I use Mozilla from WindowsXP to access the latest sendmail running on a FreeBSD 4.8 server. (Actually, I don't use Windows very often--I generally use ssh to access my mail server, and check/send mail directly with pine or mutt. Since most of the world uses Windows, however, this is a fairly typical setup. I use Mozilla because it doesn't run viruses automatically, and I am able to eliminate pop-up browser windows ...)

8.2.1 On Mozilla

On Mozilla version 1.3.1:

Edit > Mail & Newsgroup Account Settings > Outgoing Server (SMTP):
Server name:
Port: 587
Use name and password: [checked]
User name: weldon
Use secure connection (SSL): When available

Rather than set up DNS for my internal home network, I edited c:\windows\system32\etc\hosts and added a mapping from to my mail server's internal address.

I added my own certificate authority to Mozilla's list: Edit > Preferences > Privacy & Security > Certificates > Manage Certificates > Authorities > Import

8.2.2 My

I have abandoned (for the moment, at least) my purchased cert. Here are the pertinent entries in the mc file used to generate the for

dnl # SASL stuff

dnl # Offer SMTP AUTH only after encryption (STARTTLS) has been negotiated

dnl # Don't ask for client cert(s)
define(`confTLS_SRV_OPTIONS', `V')

define(`CERT_DIR', `/etc/mail/certs')dnl
define(`confCACERT_PATH', `CERT_DIR')dnl
define(`confCACERT', `CERT_DIR/WhippleCA.pem')dnl
define(`confSERVER_CERT', `CERT_DIR/')dnl
define(`confSERVER_KEY', `CERT_DIR/')dnl
define(`confCLIENT_CERT', `CERT_DIR/')dnl
define(`confCLIENT_KEY', `CERT_DIR/')dnl

My access database includes no TLS-related entries at the moment.

8.2.3 My Certificates

Earlier attempts at being my own CA caused Mozilla (and other clients) to complain because the CN (Common Name) of the Issuer matched the CN of the Subject. Noticing that commercial certificate authorities don't even have a CN in their Subject or Issuer fields, I decided to try to generate a WhippleCA cert that includes the fields:

Issuer: C=US, O=Whipple, OU=Whipple Secure Certificate Authority
Subject: C=US, O=Whipple, OU=Whipple Secure Certificate Authority

and a cert with the following fields:

Issuer: C=US, O=Whipple, OU=Whipple Secure Certificate Authority
Subject: C=US, O=The Whipple Website,

Note that since the CA's DN (Distinguished name) doesn't even include a CN component, it doesn't match the "" component of's cert.

There are many ways to prevent OpenSSL from including the CN component in the CA cert. I accomplished it by making the following changes to the openssl.cnf file (compared to the sample shipped with OpenSSL):

AttributeValue in SampleValue in Mine
dir ./demoCA .
Under the [ policy_match ] section:
stateOrProvinceName match optional
organizationalUnitName optional match
commonName supplied optional
Under the [ req_distinguished_name ] section:
commonName Common Name (eg, YOUR name) Common Name (eg, host/domain name)

Then I followed Gregory Neil Shapiro's instructions at Very brief introduction to create a CA and a CERT. I renamed the resulting files as follows:

This is the certificate for the Certificate Authority, resulting from the first call to openssl (in Shapiro's instructions). I copied the file to /etc/mail/certs/WhippleCA.pem and reference it in the confCACERT definition in the sendmail mc file.
This is the Certificate Authority's private key, corresponding to the certificate above. It isn't mentioned in the file. (It--together with the CA certificate above--is used to sign the server cert mentioned below.) If you were a commercial certificate authority, it would be critical that this file be kept secret and never be lost. Since you are using it to sign your server's certificate (below), you should still keep it safe and "secret," but if you lose it, you can just repeat Shapiro's instructions and generate a complete set of certs and keys ...
This is my mail server's certificate, resulting from the fourth (and final) call to openssl (in Shapiro's instructions). I copied the file to /etc/mail/certs/ and reference it in both the confSERVER_CERT and confCLIENT_CERT definitions.

If you browse the file, you will see that the top part of the file is readable by a human, and the bottom part is formated for the computer to transmit. If you wanted to (I didn't with my certificate, though), you could edit that file and delete the top part. To view the original top part, use the command:

$ openssl x509 -noout -text -in
To see the original file (both parts), omit the "-noout" option
This is the output of the second and third openssl commands. If you browse the file, you will see that it has two parts: an RSA private key and a certificate. The private key is your server's private key. I copied newreq.pem to /etc/mail/certs/ Then I edited the copied file and deleted the second part (the certificate).

If you compare the certificate in newreq.pem to the one in newcert.pem, you will notice that they are different. The newreq.pem cert is the unsigned (by the Certificate Authority) version; the newcert.pem cert is the signed version.

When I purchased my RackShack cert, I generated a certificate and key--probably corresponding roughly to the contents of newreq.pem. Then I copied the certificate portion (only) into a form on their web site. I did not send my key to RackShack. When the Certificate Authority signs your certificate, it doesn't affect your original key--the one that is in the newreq.pem file. (In fact, the Certificate Authority doesn't--must not, actually--know what your server's key is.)

Claus Assmann's SMTP STARTTLS in sendmail/Secure Switch was also very helpful.

8.3 Earlier Observations about Clients

I tried several other clients with an earlier set of private certificates--before I created a CA without a CN. Here are some of those (possibly out-of-date) findings. (More recent commentary on my previou writings are in [square brackets].)

8.3.1 Outlook Express

In one experiment, I had Outlook Express (OE) attempt to relay through a mail server that had a certificate signed by my own CA cert [which had a CommonName--CN]. OE displayed a dialog, informing me that it couldn't verify the security certificate, but allowing me to send the mail anyway. (The subject name in this case matched the name of the server exactly: [The reason the verification failed might be that the CN of the certificate issuer (my CA) matched the CN of's certificate??]

I repeated the above with my certificate from RackShack--the one that can't be verified when the server communicates with other mail servers. In this case, Outlook Express accepted the certificate and sent the LOGIN authentication (SMTP AUTH). [I wonder if importing the Whipple CA certificate into Internet Explorer/Outlook Express might make the private cert behave like the purchased cert ...?]

In both cases above, the maillog showed "verify=NO", indicating that the client presented no certificate--I have neither purchased nor generated a certificate for my mail clients. (I don't plan to in the near future, and doubt that most mail clients would either ...) [As I learn more about S/MIME encryption, however, I might very well have WhippleCA generate a private user cert ...]

8.3.2 Netscape

I haven't actually tried this one yet, but noticed a statement at that might apply to Netscape:

Chris Dagdigian sent me a note with a good bit of info, newer versions of Mozilla and Netscape won't accept the server certificate if it is self- signed. I.e. if the CN of the CA certificate and CN of the server certificate are the same. So make sure you use something like "Joe's Certificate Authority" for the CN of the CA certificate and use the server's hostname as the CN of the server cert.

8.3.3 Pine

Pine must be compiled with TLS/SSL support in order to support encryption. In my tests (where sendmail requires STARTTLS encryption before offering plaintext SMTP AUTH), Pine without SSL support flatly fails to send mail through the server, reporting that SMTP AUTH is not supported.

Here is how Pine behaves with TLS/SSL compiled into the executable:

With Cert Purchased from RackShack:

Pine is able to connect using STARTTLS, authenticate with plaintext SMTP AUTH [through the encrypted STARTTLS tunnel], and relay through the remote sendmail server. It displays no warnings about unverifiable certificates. Here is the setup I used:

With Certs Signed by Whipple CA:

I repeated the above, this time using self-signed certs. [In this earlier CA cert, the certificate "Issuer" and "Subject" were identical. I should repeat this test with my new CA Cert.] This time, pine displayed a warning:

There was a failure validating the SSL/TLS certificate for the server


The reason for the failure was

                      self signed certificate in certificate chain (details)

We have not verified the identity of your server. If you ignore this certificate validation
problem and continue, you could end up connecting to an imposter server.

If the certificate validation failure was expected and permanent you may avoid seeing this
warning message in the future by adding the option


to the name of the folder you attempted to access.

I entered "Y[es]", responded to the prompt for my password, and the relay worked properly.

[Note: I need to try the above with my new CA cert, where the CN of the issuer and are not the same.]

9. SMTP Auth and STARTTLS Encryption without Certificates?

Anonymous Diffie-Hellman (ADH) encryption does not require certificates. Used by itself, it leaves connections open to man-in-the-middle attacks, since there is no way of verifying the machine at the other end of the connection. Used with SMTP Authentication, ADH might be suitable for encrypting plaintext user names and passwords sent by clients to servers.

9.1. Server-to-Server Encryption with ADH

I have successfully configured two sendmail servers to encrypt using ADH, without certificates:

  1. Enable the #define _FFR_TLS_1 (FFR = "For Future Release") code path when building sendmail. (The sendmail that ships with FreeBSD comes built that way). You can verify that sendmail (version 8.12.9, at least) was compiled with _FFR_TLS_1 by issuing the following command in the directory that has the sendmail executable:
    # strings sendmail | grep CipherList

    (If the string "CipherList" is displayed as a result, then sendmail probably supports _FFR_TLS_1.)

    Note: Be sure you are looking at the actual sendmail executable. In a "standard" FreeBSD installation, for example, the sendmail executable is at /usr/libexec/sendmail/sendmail. The executable at /usr/sbin/sendmail is a "mail wrapper" that calls the real sendmail indirectly.
  2. Add the following to your sendmail mc file:
    O CipherList=ALL:ADH:@STRENGTH
    The CipherList values are used to call OpenSSL's SSL_CTX_set_cipher_list. You can see an example on p. 147 of Network Security with OpenSSL by John Viega et al. (O'Reilly, 2002). In my LOCAL_CONFIG example, "ADH" probably isn't necessary, since it is already included by the "ALL" keyword. The @STRENGTH keyword says to list the "strongest" ciphers first.

    The only TLSSrvOptions value documented anywhere else is "V", which tells the server not to ask the client for a certificate. The "C" option (mentioned in the source file readcf.c) probably stands for "CipherList", meaning that Anonymous DH needs to be enabled with the CipherList option.

  3. Repeat the above for the second sendmail server.

9.2 Client-to-Server Encryption with ADH

The real usefulness of ADH encryption would be to provide an encryption tunnel for SMTH AUTH user names and passwords. I tried relaying from a Mozilla e-mail client through a sendmail server with ADH encryption and no certificate. Unfortunately, Mozilla displayed the following message:

Mozilla and cannot communicate securely because they have no common encryption algoriths.

Apparently the Mozilla developers have decided that because Anonymous Diffie-Hellman is bad for HTTP, it is also bad for e-mail.

I tried the same experiment with Microsoft's Outlook Express. The text of the resulting message was different, but it said essentially the same thing. For now, it appears that Anonymous Diffie-Hellman is not a viable alternative for e-mail.

10. What Appears in Logs and Headers?

As STARTTLS-enabled mail servers communicate among themselves, it is almost certain (from my experience so far) that purchased certificates will fail to verify. ("verify=FAIL" will appear in the maillog.) If the server acting as a client doesn't offer a certificate, then "verify=NO" will appear in the mail log. (Clients acting as clients--which rarely have certificates at all--will also cause "verify=NO" to appear in the log.)

11. Conclusions

I'm not quite ready to make final conclusions yet. So far, I have decided that using SMTP AUTH is my favorite way of controlling relaying. Additionally, STARTTLS seems perfect for encrypting PLAIN and LOGIN user names and passwords required by SMTP AUTH. Using encrypted SMTP AUTH, there is no need for client SSL/TLS certificates for verification--the user name and password verify the client's identity.

If e-mail clients supported Anonymous Diffie-Hellman, SSL encryption for SMTP AUTH would be possible without any certificates at all. Since clients don't support ADH, sendmail servers will need an SSL certificate in order to encrypt e-mail.

12. Selected Resources

SMTP STARTTLS in sendmail/Secure Switch
If you follow the above blindly, it might work for you. In the process, you might even come to understand what is going on!

The instructions assume that you are using bash or some sort of Bourne shell. If you are using tcsh (which I use on FreeBSD), type "bash" before issuing the command:

C=FileName_of_CA_Certificate so that your environment is set up correctly ...

Very brief introduction to create a CA and a CERT
I followed this exactly, and ended up with my private certs. You need to use the first section only once (to create your own Certificate Authority). Repeat sections 2 and 3 for each server certificate you want to generate. (Adjust the output file names newcert.pem and newkey.pem, using a different name for each server ...). See My Certificates (above) for changes I made to openssl.cnf.
Google is your friend! Use it to search for STARTTLS in the sendmail newsgroup. Most of your questions have been answered before.
Bryan Costales with Eric Allman, Sendmail, 3rd ed., (O'Reilly, 2003).
The latest edition of the "Bat Book"--the most authoritative source of information on sendmail. Updated to cover sendmail version 8.12.x. STARTTLS coverage starts on p. 415.
John Viega, Matt Messier & Pravir Chandra, Network Security with OpenSSL (O'Reilly, 2002).
The best source information I know of on OpenSSL's implementation of TLS and SSL (except, perhaps for and Google).
Eric Rescorla, SSL and TLS: Designing and Building Secure Systems (Addison-Wesley, c2001).
Probably the single best source of information on SSL and TLS. The code examples in "C" use OpenSSL. Those in Java use PureTLS.
Stephen Thomas SSL and TLS Essentials: Securing the Web (Wiley, 2000).
Gives a good overview of SSL and TLS. This book is probably OK if you don't want to get into implementation or configuration of SSL and TLS.
Everything about sendmail is here. Don't forget the search the FAQs. Claus Aßmann's tilde directory is loaded with useful information

Don't overlook the "Sendmail Configuration Files" README (in the sendmail distribution at ./sendmail/cf/README) or the Sendmail Installation and Operation Guide. The latter is in the distribution at ./sendmail/doc/op. If you can't read anything there, try issuing the command (with ./sendmail/doc/op as your current directory): make op.pdf for a nicely formatted PDF file. If that doesn't work, then try make op.txt for a text version of the file.