vSPhere 6.7 – Custom SSL Certificates

Note: There are 20 tutorials for installing custom SSL certificates out there on the net. I’m not going to cover that in detail. What I will cover here is all of the little things that you will need in order to be successful following one of those tutorials. They aren’t comprehensive and everything goes according to plan. In my experience that just isn’t a fair representation of what happens, especially after upgrades or not getting it right the first time. So read this first and then go try one of the tutorials. Or if you’re stuck on one of those tutorials, hopefully this will get you out of the muck. I’ll post links to some useful tutorials and documentation at the end of this post.

Custom SSL certificates prior to vSphere 6.x used to be a frustrating proposition. It wasn’t particularly easy to do and once done, it caused a lot of misc operational issues. It almost always made troubleshooting more difficult and it could cause communication issues between 3rd party components. The way its done now is consistent and convenient. It’s even easy once its in place.

In 6.x, the Platform Services Controller issues SSL certificates to every component participating in the SSO domain. Even 3rd party plug-ins are issued an SSL cert directly from the PSC. This is done by default with a self-signed root CA certificate. All we need to do in order to have the PSC issue valid SSL certificates for our own environment is to authorize it as a signing authority in our SSL signing chain.

Check out my other blog posts and security pages on SSL basics and 80/20 rules for success.

The process is the same for 6.5 and 6.7. I have had more success with these tools working the further along I am in versions. If you’re considering an upgrade to 6.7 then do that first. If you’re on 6.5 or 6.7, install all available updates and patches first.

First of all, before you start or if you’re having trouble, install the latest patches. Before you get into the meat of installing your own certificates, install the latest versions, updates, and patches. Did I say that enough times? There are some bugs in the SSL tools, pretty much in every version. One of them is even related to creating your CSRs so upgrade before you even get started.

I am only covering the process for the VCSA and not a Windows VC server. I have a distributed VCSA environment in my lab with multiple sites, PSCs, and VCs. The important pieces covered here should apply equally to a Windows install, however, some of the commands and paths referenced will be different on Windows.

Step 0: Install all available upgrades and patches. (One last time).
Then make sure that the DNS CNAME and PTR records for these things are correct:
vCenter
PSC
Every ESXi host

Step 1: Verify that you have an Enterprise Certificate Authority in your environment, that you are able to request certificates, and that you know how to contact the CA administrator. Also make sure that your CA configuration is up to date and using SHA256 instead of SHA1. SHA1 signed certificates will not be considered valid by most clients. This shouldn’t be an issue unless your CA has been around a long time or unless you’re starting out with an older OS to provide your CA, like Server2012. Note that even SHA256 certificates will have an SHA1 thumbprint. Don’t worry about that while troubleshooting, its normal.

Step 2: Follow the instructions (in other tutorials) for creating a VCSA Signing Certificate Template for 6.5 and higher. You will need this template to correctly fill your CSR.

Step 3: Download and install OpenSSL on your workstation, whether Windows or Linux. And look up the instructions for converting PKCS12 certificates to PEM format. You might need this for Step 8.

Step 4: If you have already done custom SSL certificates on your VCSA and are having trouble, or if you are replacing existing custom SSL certificates (If you don’t have an SSL blank slate) then you will want to follow this tutorial to reset your SSL environment back to zero.

Step 5: Login to your VCSA all-in one, or your external Platform Services Controller and run the certificate manager program.

Step 6: Choose “Replace SSL Certificate with Custom Signing Certificate and replace certificates.” The actual number for this varies depending on vSphere version and install type. You will need to enter an SSO admin credential.
Then choose to create a Certificate Signing Request.
When asked if you want to replace all certs, answer yes.
When asked if you want to configure the SSL configuration file, choose yes.
For this step, what you enter here is important. Enter the typical answers for Location, State, etc.
When asked about the Common Name, DO NOT ENTER THE FQDN OF YOUR PSC HOST. If you do, the whole process will fail several steps later.
This will be the name of the CA that is created. Ask your CA admin or examine an existing SSL certificate to determine if there is a naming convention. If you’re not sure, use HOSTNAME-CA.
Don’t enter an IP Address for your PSC, its unnecessary. The last question asking the name of your CA, use what you entered for the CN above. I will refer to this a CA_NAME for the rest of the guide.
Save your CSR and key someplace that is easy to find like /root or /tmp.

Step 7: Get your CSR signed with the VCSA Template created in Step 2 and export it as Base64.

Step 8: Add the whole certificate chain to your certificate.
check out my guide for doing this here

Step 9: Copy your new certificate chain file back to the PSC host

Step 10: If Certificate Manager is still up on your PSC then continue to import Custom SSL Certificate. If it’s not, rerun Certificate Manager. Choose the custom SSL option again, and then choose to import your custom certificate chain.
Provide the full path to your certificate
Provide the full path to your key file
Watch the prompts on the above two lines carefully. Notice if it accepts your cert before entering the key.

If you immediately encounter an error with the certificates. Then check Steps 0, 4, 6 and 8.

It will go through a lengthy process of generating and replacing keys and restarting services. If this process fails, refer to Steps 0, 4, and 6 as the problem will almost always be in one of those places.

Step 11: If you are on 6.7u1, this step is optional. Run certificate manager again. Choose to replace the Machine Certificate.
When asked if you want to reconfigure the SSL configuration, choose yes.
When asked for the CN, go ahead and use the FQDN now.
When asked for the CA name(last question) use CA_NAME identified in step 6.

Step 12: If you are on 6.7u1, this step is optional. Run certificate manager again. Choose to replace the Web Services Certificate.
When asked if you want to reconfigure the SSL configuration, choose yes.
When asked for the CN, use something different than you used for step 11. Maybe web-FQDN.
When asked for the CA name(last question) use CA_NAME identified in step 6.

Step 13: If you have an external PSC, then login to your vCenter Server and perform steps 11 and 12 for the vCenter Server.

Step 14: Restart the services on your vCenter Server (especially if you have an external PSC)

Step 15: Navigate to to FQDN of your vCenter Server. If you don’t have a clean SSL state, then inspect the site’s SSL certificate. If you see the whole chain, then its probably a caching / cookie issue. Clear your cookies or restart your browser. If it still isn’t working, look at the specific error message in your browser for a clue to the problem.

Step 16: Navigate to the VM Admin URL for your PSC and VSCA appliance(s). You MIGHT find that you have a valid SSL certificate there. You MIGHT find that you don’t.
If you don’t, then follow these instructions to fix it.

Step 17: ESXi Host Certificates. Your ESXi hosts won’t accept a new certificate from the PSC until that certificate is 24 hours old. If you don’t want to wait for 24 hours, you can adjust this setting. It is an advanced vCenter Server setting and is configured in minutes. Just change this setting to something like 5 minutes.
vpxd.certmgmt.certs.minutesBefore
When the hosts are done adding you can change it back to the default.

Step 18: Re-register your 3rd-party plug-ins. You may have had to disable or remove 3rd-party plug-ins in order to get this far. If you did, now is when you can re-register them. Keep in mind that the certificates issued by the PSC are only for inter-service communication. The actual server management URLs of your extension servers will need their own Custom SSL certificates to secure front-end management traffic. I will hopefully be providing enough examples of those to make securing whichever ones you have a piece of cake.

If you found this article helpful, take a look at my other vSpehre or SSL related posts and pages. Especially Breaking Bad SSL Habits.

I wrote this up a bit after I did the process so if anything isn’t quite right, feel free to let me know and I’ll fix it.

VMware Documentation
Creating a Signing CA Template
Replacing Default SSL Certificates with Custom Certificates

Excellent Example Video

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Blog at WordPress.com.

Up ↑

%d bloggers like this: