Mac OS X Panther Server and SSL

by Joel Rennich, mactroll@afp548.com

The purpose of this article is to give you an idea of what you can do with SSL in Mac OS X Server and how you can use that to secure as many services as possible. So, first we’ll talk some about SSL in general and how to create the certificates, then we’ll discuss what to do with those certificates.

SSL certificate creation

Before we start I’d like to point out that we are going to be creating home-rolled SSL certificates here. As such you will run into problems when connecting to your Server using applications like a Web browser. Most applications will allow you to ignore the fact that your certificate hasn’t been validated by one of the internationally recognized certificate authorities, but it’s still a pain.

I’ll show you how to get around that by importing in your own certificate authority onto your client machines. This presumes that you have control over all of your clients, so for internal use where you control both the server and the client setups being your own certificate authority is great. If you plan on doing business with the general public, such as using it for credit card processing on a Web page, I would strongly recommend that you invest the money in a “real” certificate.

If you do buy one at least take a look at www.qualityssl.com. They have really good prices and are Mac-based, so you can keep it in the family.

Also all of the openSSL work, such as generating and signing certificates, can be done on Mac OS X client.

1. Make a certificate authority (CA).

This should be done in a secure place, since if your CA gets compromised then all of your security goes out the window. A decent place for this would be on your most secure server or on your own machine.

It doesn’t matter where on the filesystem you do this; however, I personally prefer to create a CA directory in /etc:

sudo mkdir -p /etc/certs
cd /etc/certs

Right now this folder has fairly relaxed permissions on it. As soon as were done we’ll change that to greatly limit access to the folder. Now that we have our place we need to begin generatng the CA. We do this by making a certificate signing request (CSR). This example will generate a 3DES encrypted 2048 bit key. This is a rather high security key which means it takes longer to process. So if you feel the need you can scale it down to 1024 bits if you like. Although I haven’t had any problems using this with Mac OS X 10.2 and Windows 2000.

openssl genrsa -des3 -out ca.key 2048

You will be asked for a passphrase for this key. You need to both remember this phrase and keep it secure. Your entire SSL system will depend on this passphrase being secure.

Now that you have the request you can sign it into a CA.

openssl req -new -x509 -days 4096 -key ca.key -out ca.crt

You’ll be asked for the passphrase that you just set up. After that your certificate authority will be valid for 4096 days.

You now have a full blown certificate authority for your machine. From this we will base all of your other certificates from it.

2. Generate a certificate for your server.

You will need one for each domain that you have; i.e., mail.afp548.com and http://www.afp548.com will each need one if you want to secure both sites.

So first we will generate a new private key.

openssl genrsa -des3 -out server.key 1024

You will be prompted for a password here also. This should be different from the password for the CA. Just remember it because you will need to enter it into Server Admin to get SSL running.

Now you need to generate a request with the private key.

openssl req -new -key server.key -out server.csr

Again you will be asked for a password. This is the one you entered in the step above. Then you will get a bunch of questions. They all really don’t matter except for common name. This needs to be the fully qualified name of your Web server, like http://www.afp548.com. If this is wrong you will get errors in the browser. Also: leave the challenge password blank.

Now we need to set up a few folders so that we can actually sign the certificate.

mkdir -p demoCA/private
cp ca.key demoCA/private/cakey.pem
cp ca.crt demoCA/cacert.pem
mkdir demoCA/newcerts
touch demoCA/index.txt

            echo “01” > demoCA/serial

You can now actually sign the server certificate with your newly minted CA.

openssl ca -policy policy_anything -in server.csr -out server.crt

The password you are prompted for is the password you assigned to the CA, the first one, not to the certificate itself. If you need to create more certificates you will only need to do the last three steps for each.

Finally to keep things secret and to keep things safe, change the permissions on this folder.

sudo chmod 700 /etc/certs

Now you can take all of your pieces and make the sites secure.

3. Securing your web site.

Go into Server Admin and make sure that the SSL module is enabled in the modules pane under settings.

Then go to the site that you want to secure. Change the port to 443, click on the security button, and enable SSL by checking the box at the top. Then you need to open up some of the files that you have created in TextEdit, or any text editor, and copy and paste them into the three appropriate spots. Copy server.crt into “Certificate File.” Copy server.key into “Key File,” and copy ca.crt into “CA File.”

Finally, you’ll want to enter the passphrase for the server certificate into the “Pass Phrase” field or else you’ll have to be at the server everytime it starts up.

A few parting thoughts about securing Web connections. You will need a separate IP address for every SSL site that you have. There’s a complicated reason for this, but it involves how SSL connections begin and I don’t know of any way around this. In 10.2 you had to edit the httpd_macosxserver.conf file to get higher level encryption. This requirement seems to be gone in 10.3 as it defaults to using all ciphers.

When you are done your certificates will be stored in /etc/httpd/ssl.crt and /etc/httpd/ssl.key. Your site’s specific config is stashed in /etc/httpd/sites/your site’s name. So look in there for any specific info. Also the passphrase that you used is stashed in /etc/httpd/servermgr_web_httpd_config.plist, which is root-readable only.

4. Securing LDAP

We run into a bit of a problem here. OpenLDAP doesn’t like a server key that has a passpharse associated with it. Postfix and Cyrus are going to be the same way. So remove the passphrase.

openssl rsa -in server.key -out serverno.key

Now go back into Server Admin. Select the Open Directory settings and go to the “Protocols” tab. Check the “Use SSL” box and then put the path to your certificates in the three fields.

Certificate: /etc/certs/server.crt
SSL Key: /etc/certs/serverno.key
CA Certificate: /etc/certs/ca.crt

OpenSSL runs as root, so it will be able to get into /etc/certs without any issues. As soon as you save this config Server Admin will restart OpenLDAP with SSL support.

The SSL configuration for OpenLDAP is stored in /etc/openldap/slapd_macosxserver.conf.

5. Securing SMTP

Postfix can be setup to use the same certificate as the one you established for openLDAP. However, it wants to have both the key and the certificate in the same file. This is easily done.

sudo cat /etc/certs/serverno.key /etc/certs/server.crt > /etc/certs/server.pem

Now link that file to what Postfix is looking for.

ln -s /etc/certs/server.pem /etc/postfix/

Now reload Postfix through the GUI or by doing this from the command line.

sudo postfix reload

And start using encrypted SMTP services.

The SSL configuration for Postfix is kept in /etc/postfix/main.cf.

6. Securing POP/IMAP

Cyrus can use the same certificate as Postfix, but it needs to be accessible by the cyrus user. That requires relaxing the permission a bit on the certificate store.

sudo chown :mail /etc/certs
sudo chmod 750 /etc/certs
sudo chmod -R 700 /etc/certs/demoCA

Now you can link the server.pem file into where Cyrus POP and IMAP want to find it.

ln -s /etc/certs/server.pem /var/imap/server.pem

Now go into Server Admin and set up POP/IMAP to use SSL in the Advanced button of the Mail Server settings.

Set your mail client accordingly and securely read your mail.

The SSL configuration for Cyrus is stored in /etc/imap.conf.

7. Enable your clients

Since your CA is self-signed all of your Mac OS X applications and services will yell at you for using it. You can get around this by adding the cert to the client’s x509 Anchors keychain. Essentially this is the root CA file for your machine.

Do this by copying over to the client machine the ca.crt file that you created in the first step. Then install it by doing

sudo certtool i ca.crt v k=/System/Library/Keychains/x509Anchors

Your client will now trust certificates that you have signed into being with this CA. If you do this right, you’ll use the same CA for all of your servers and their services. That way you’ll only have to import one file into the clients x509Anchors.

8. E-mail certs

This bit is for bonus points, but all the cool kids are doing it and so should you. Mail.app in 10.3 allows the use of s/mime certificates. These are PKI certificates that act similar to SSL certificates and can sign and or encrypt e-mail.

The easiest way for a personal user to get a certificate is to head over to www.thawte.com and sign up for their free community mail certificate. Really good instructions for this can be found here:

http://joar.com/certificates/

However, if for some reason you feel like making your own, read on. Note that this is mostly an exercise in what you can do with OpenSSL. Since the Thawte certificates are free and easily available you’re probably better off using them. However, if you want to outfit your entire organization with home rolled certificates, well here you go. Just be careful to only use this between users that have imported your root CA that you created.

To do this you need to first generate a certificate for your e-mail user. This is pretty much the same thing as generating one for a server.

openssl genrsa -des3 -out mail.key 1024

Give it a pass phrase to lock it up.

openssl req -new -key mail.key -out mail.csr

Here, you’ll want to use your real name for the Common Name. Joel Rennich is what I would use. Then make sure that you fill out the e-mail field with what you have set up in Mail.app as your e-mail address. Capitalization is important here. I would use “mactroll@afp548.com”.

Now sign this cert with your CA.

openssl ca -policy policy_anything -in mail.csr -out mail.crt

You’ll enter in your CA password and then commit the signature.

Finally you can convert the signed certificate into the format that is used for s/mime. When you do this it will first ask you for your mail certificate password that you set up a few commands before. Then it will ask you for an export password. This can be the same of different, it doesn’t matter, but you will need to use the export password when importing this certificate into your Keychain so you can use it with mail.

openssl pkcs12 -export -inkey mail.key -certfile mail.crt -in mail.crt -out mactroll.p12

This is your “official” e-mail certificate. Copy this over to your client machine and double-click. Keychain Access should launch and ask you for your export password. The certificate will then be imported into your keychain and immediately usable by Mail.app for the account that you specified in the e-mail field when you generated it.

9. Other odds and ends

When you sign your certificates with your CA openssl uses a default config file which can be found at /System/Library/OpenSSL/openssl.cnf. If you want to change any of the defaults go here. For example, certificates that you sign will only be valid for 1 year, unless you edit this file to change that.