This post is a part of automated deployment story

One of the blockers to widespread adoption of automated deployment tools is the fear that they cause greater security risk than humans executing a written deployment procedure. The greatest of fears is that an attacker would be able to push a malicious version of binaries to the production environment.

If you want to succeed implementing an automated deployment solution you must ensure at least same level of security as when humans manually copy binaries.

Who can deploy?

When there is no automation, the deployment is performed by designated humans. For testing environment it can be the QA manager, for production it is usually an operations team. They (and only they) are in possession of credentials required to log on to the target environment machines to copy the binaries and perform other deployment-related tasks.

People who have just begun automating the deployment tend to think that, since everything can be automated, the deployment process can be started straight from the build agent without any human action. This was also true in my case. Fortunately some wiser people reminded me about how important is to be able to name the person (not a machine) who was responsible for particular deployment.

The rule of thumb is, you can (and should) automate everything, but two things. First is starting the whole procedure. There should be a person that has to hit ENTER key in order to start deployment. We’ll be the one to blame if something goes wrong (for example upgrading the production site during peak usage hours). The second thing not to be automated is authentication. You should never store production (or even testing) environment credentials in order to be automatically used by the script. Whenever credentials are required, the script should ask the user to provide them. Sometimes it is possible to store them temporarily in memory so that user does not get prompted for same credentials twice.

What can be deployed?

Usually the more layers of security, the better. Should one get compromised, the other can prevent the disaster. Beside controlling who can deploy, it is also worth controlling what can be deployed. The easiest thing (and this is what we have actually implemented) is digital signature-based verification. A binary package is signed using a certificate stored on the build agent. The corresponding public key along with a gateway script is securely installed on all machines of the environment.

The only remotely accessible endpoint that is exposed by target environment machines is the aforementioned gateway script. This ensures that an attacker can’t bypass the security measures put in place. One can only trigger the deployment on the machine via this script and the script ensures that deployment package signature is verified against stored public key of the build agent.

Tools

Unfortunately there is no command-line tool to sign and verify signature built into Windows. That’s why PackMan was born. I mentioned this tool before in context of building a package. When building a package, PackMan also calculates the hash of the data and encrypts it using private key from provided certificate. It can be done with a command like this (MSBuild variable syntax)

PackMan.exe -i d:$(PackageDir) -o $(OutDir)\Package-$(BuildVersion).zip --cn DeployerCert -a create

When asked to unpack, PackMan first verifies the signature using a provided public key (PowerShell variable syntax)

PackMan.exe -p ${package_store}\${package_file} --vcn DeployerCert -a unpack -d ${tmp_package_dir}

This ensures that package is authentic and not corrupted.

VN:F [1.9.22_1171]
Rating: 5.0/5 (1 vote cast)
Automated deployment security concerns, 5.0 out of 5 based on 1 rating