I think that most of us are familiar with the error above. We deploy something new into the environment and we know we will get around to adding an SSL certificate EVENTUALLY. In the meantime we buzz right along, logging in, providing root passwords, and desensitizing ourselves to what this really is. A BIG RED DANGER SIGN TELLING YOU NOT TO PUT IN YOUR ROOT PASSWORD. But we see this and we go ahead and put in that password anyway.
Over time they add up on the to-do list until we have 20 of these in flight and then it’s just standard operating procedure. The reality is that this big red warning sign is there for a reason. There’s a reason its big, there’s a reason it’s red, there’s a reason it’s right there in your face, and there’s a reason they made it so you have to click on several things to bypass it.
If you’re reading this, you know it’s true. The trouble is that it’s just so much work to do this right. I mean you have to build a Certificate Signing Request, fill out a form, put it in a queue, wait for it to be justified, approved, issued, and then it has to be installed. Heck if you manage to get a certificate at all you might not even have time to get around to installing it. When you do, crap, one of those new required fields gets missed and you have to do some research to figure out how to use it correctly only to start the whole process over. If you’re in an environment without the bureaucracy of somebody doing it for you then it’s even more work.
I really do understand why so few people exercise this discipline in their internal environments. Besides, it’s internal, right? No pen test will hit it and that’s the only reason we do security right? It to pass audits? Unfortunately that is not the case, these sort of vulnerabilities are actually very significant and can completely compromise your entire environment.
So what’s to be done. I’ll give you some pointers and most of you will say they are wrong, poor answers, bad discipline, or maybe even just as bad as ignoring the warnings. I don’t agree. I think the worst part of all of this is the lack of hygiene and the complacency around the big red triangle. If you plan to do things 100% right, then great, go for it! If not then these 80/20 compromises will be a worthwhile use of your time and earn you a Christmas card from your CISO.
Securing Virtual Infrastructure
First of all the low-hanging fruit. Most of us use VMware at least for something (usually for the important stuff) and even though the platform is generally considered secure, that only applies if we don’t go around handing out our passwords to anyone on the network. In fact if you are going to incur risk from a single compromised system, your Virtual Environment hosts take the cake, big time. I mean with the root password to an ESXi or KVM host, I can download a copy of your mail server, a domain controller and just about anything else. I can take those things home to hack in my pajamas safely away from any detection or alerting systems right? Right. I can then come back in with those credentials after I’ve read everyone’s emails and log in to everything unobstructed without triggering any alarms right? Right. So. The low hanging fruit. If you’re not seeing this little lock icon right here, then you have problems.
I’ve installed hundreds of Virtual Centers and ESXi hosts over the years. Installing custom certificates has always been a chore. But the worst part has been updating them, dealing with all of the caveats they introduce, fixing expired certs, making plug-ins compatible, and troubleshooting why random small things have stopped working only to trace it back to changing the SSL certificates. Historically it has been painful and hardly worth the pain. Luckily as of vSphere 6.0, the Platform Services Controller can act as an Intermediate CA signing authority. What this means is that any and all vSphere related things will request a signed SSL certificate from vCenter when it is added or connected. These certs will be valid in your organization and get rid of those pesky red triangles. A few years ago, I might have agreed with you if you said it’s not worth the effort. Now I will say there’s no excuse not to do this.
All I did to get this shiny lock symbol for these hosts was add them to Virtual Center. That’s it. Getting that set up was a bit of work, sure but in the 80/20 scheme of things, it was worth every second.
Next up Wildcard Certs…
What is a wild card certificate? I’m sure you know, its a certificate that we put on a large number of hosts that all participate in the same application. Effectively they are all the “same” so one manner or other has been used to indicate that ALL of these hosts, or EACH of X,Y,Z hosts are all valid for this same certificate. “Okay, but I’m not hosting a web app here so how does that help me?” you might ask. I’ll tell you how.
Create ONE wildcard certificate. Yup, that’s it. Create ONE certificate for your whole internal domain. Put it on everything. “That’s not right! That’s not what they’re for!” they’ll say. “That defeats the whole point of everything!” they’ll say. Maybe they are right but I say that the whole point of everything is hygiene and discipline. It completely solves the problem of alert fatigue. It completely eliminates people habitually ignoring the big red warning label and it creates an environment where the warning messages actually mean something.
I didn’t say this was perfect, I said it was 80/20. 80% of the reward for 20% of the effort. Is this the 100% best way to solve the problem? Maybe not. But unless you’re going to go the full 100% and do it 100% right every time, all the time, then I think this way takes the cake.
Next time on my Security Hygiene series, bastions.