vSphere 6.7 – Resetting the SSL state back to zero

I have a few other tutorials on here regarding vSphere SSL certificates. I found that there were a variety of issues which led to a problematic SSL state that was difficult to recovery from.

The guide will show you how to get back to a stable starting point so that once you understand the process, you can install custom SSL certificates without any problems.

This guide only covers a VCSA environment. If you have a Windows environment, the tools and paths will be different, however, the concepts are the same.

I have created some scripts to make this process simpler. When I have a moment, I will upload them to github and link it HERE. (If you would find these helpful before I get that done, please contact me and I’ll get them uploaded sooner).

Step 1:
Unregister any 3rd-Party Extensions. These will often block successfully installing / updating the PSC certificates. Here are a couple of useful example links, or refer to the documentation for your 3rd-Party Extension Provider.
Remove Extensions using SSH
Remove Extensions using the MOB Browser

Step 2:
Attempt to use Certificate Manager to revert to default / self-signed certificates. This may not work if you are having other SSL related issues but try.

Step 3:
Identify and remove all non-VMware Root CA’s registered in the certificate store. This can feel complicated the first time. You will need to get familiar with a few tools, hopefully you are comfortable with the Linux CLI. This was tedious enough for me that I wrote some scripts which I will reference in addition to showing you the command line utilities. The instructions for doing this will be included below.

Step 4:
Assuming that Step 2 was not successful before, attempt Step 2 again. If you can’t get Step 2 working then installing your own certs won’t go any better.

Step 5:
If you can’t get Step 2 working then you are going to have to parse through your logs for warnings or errors. I recommend backing up or deleting your Certificate Manager Log file and running Step 2 over again. This way you will only have to parse data from one run at the process.

rm /var/log/vmware/vmafd/certificate-manager.log
/usr/lib/vmware-vmca/bin/certificate-manager
Or use my gencerts.sh script
cat /var/log/vmware/vmafd/certificate-manager.log |grep -i 'warning \|error \|fail' |more

I recommend starting with your favorite search engine for errors, but feel free to reach out to me if you can’t find a solution.

Step 6:
If you can’t get to the bottom of this, then I recommend upgrading to the latest update available and installing all available patches then trying again from Step 2.

If this doesn’t work then you might need to reinstall your PSC. This isn’t too difficult actually. Just do a backup, run the installer (Install the latest version) and redeploy using the backup files.

Viewing the Contents of the Root Certificate Store

vecs-cli usage:
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text

This will make it easier to read
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text |grep -i 'Alias\|Subject:\|Before\|After\|issuer'

Or use my listcerts.sh script

Example Output
Alias : f052cf63552bc9cb365c199b3320fa383415979f
        Issuer: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=wdl-psc-00.lab.local, OU=VMware Engineering
            Not Before: Feb 21 20:31:25 2019 GMT
            Not After : Feb 18 20:31:25 2029 GMT
        Subject: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=wdl-psc-00.lab.local, OU=VMware Engineering
Alias : 5ab252164061b935c22128f875a264fec8efd1d0
        Issuer: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=pdx-psc-00.lab.local, OU=VMware Engineering
            Not Before: Feb 27 16:20:59 2019 GMT
            Not After : Feb 24 16:20:59 2029 GMT
        Subject: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=pdx-psc-00.lab.local, OU=VMware Engineering
Alias : ff1f984a104a7c265ab6a3bd98c5b9a22c809b70
        Issuer: DC=local, DC=lab, CN=lab-PDX-DC-01-CA
            Not Before: Feb 20 17:08:18 2019 GMT
            Not After : Feb 20 17:18:18 2039 GMT
        Subject: DC=local, DC=lab, CN=lab-PDX-DC-01-CA
Alias : e6575bb7c6e3486bd4355e236e8dbefb0ddfb013
        Issuer: DC=local, DC=lab, CN=lab-PDX-DC-01-CA
            Not Before: Mar  2 18:51:15 2019 GMT
            Not After : Mar  2 19:01:15 2021 GMT
        Subject: C=US, ST=OR, L=PDX, O=Local Lab, OU=Engineering, CN=PDX-PSC-00-CA
                CA Issuers - URI:ldap:///CN=lab-PDX-DC-01-CA,CN=AIA,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=lab,DC=local?cACertificate?base?objectClass=certificationAuthority
Alias : e56a6f43a38003e101c2abfc35f0ad50de7218b9
        Issuer: DC=local, DC=lab, CN=lab-PDX-DC-01-CA
            Not Before: Feb 25 05:23:12 2019 GMT
            Not After : Feb 25 05:33:12 2021 GMT
        Subject: C=US, ST=WA, L=WDL, O=Lab.local, OU=Engineering, CN=WDL-PSC-00-CA
                CA Issuers - URI:ldap:///CN=lab-PDX-DC-01-CA,CN=AIA,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=lab,DC=local?cACertificate?base?objectClass=certificationAuthority

In the case above you would want to identify (and delete) the aliases for everything that isn’t a VMware self-signed cert.
ff1f984a104a7c265ab6a3bd98c5b9a22c809b70
e6575bb7c6e3486bd4355e236e8dbefb0ddfb013
e56a6f43a38003e101c2abfc35f0ad50de7218b9


The process is below or you can use my deletecert.sh script.

Backing up the Aliases
Backing up the aliases is part of deleting them. I will assume that you have a folder called /certs on your PSC host.

vecs-cli usage:
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias $ALIAS --output /certs/$ALIAS.crt

Example:
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias ff1f984a104a7c265ab6a3bd98c5b9a22c809b70 --output /certs/ff1f984a104a7c265ab6a3bd98c5b9a22c809b70.crt

Un-publishing the Alias
The alias needs to be unpublished before it is deleted or there is some risk that the certificate will be restored to the certificate store. The backup copy of the cert is used for this process.

dir-cli usage:
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert "/certs/$ALIAS.crt"

Example:
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert "/certs/ff1f984a104a7c265ab6a3bd98c5b9a22c809b70.crt"

Deleting the Alias

vecs-cli usage:
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias $ALIAS

Example:
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias ff1f984a104a7c265ab6a3bd98c5b9a22c809b70

Afterward list the aliases again to make sure the one you deleted is gone.

Advertisements

Comments are closed.

Blog at WordPress.com.

Up ↑

%d bloggers like this: