Secure Credential and Certificate Management for Data Pipelines

Authors: Matthew Hall, Brian Waskiewicz, and Mike Forgione

Threat intelligence feeds and other sensitive data pipelines found in many contexts make use various different Authentication methods to protect data when transferred or at rest.

Some examples of these methods are:

1. Totally Unauthenticated
2. Unique / Secret HTTP file path
3. Proprietary HTTP Headers
4. Proprietary Login APIs
5. HTTPS with HTTP Basic Authentication
6. HTTPS / SSL/TLS with Client Certificate (and Private Key)
7. SSH / SCP / SFTP / RSYNC Private Keys

The wide range of authentication complicates the secure software development process. One must consider the TLP (Traffic Light Protocol) restrictions, cryptographic best practices (such as PCI or FIPS requirements), and system hardening (such as NIST recommendations), to create a secure solution.

Secure Credential Storage

The most secure methods for accessing sensitive data, such as HTTP Basic Authentication and TLS Client Certificates, or SSH infrastructure, require secure credential storage to allow automated access to a data pipeline.

Secure credential storage consists of:

1. a keystore, which is an encrypted file or database managed by a security library. A FIPS-compliant keystore does not allow direct access to keying material to protect the credentials during use.
2. a passphrase, which is stored in a separate location from the keystore, itself, to protect the credentials at rest.

When these requirements are met, it greatly complicates the process of credential exposure and/or exfiltration, or in the case of a hardware implementation, can potentially guarantee exfiltration is impossible.

Common Implementation Issues

Sadly, most popular software development reference sites, such as StackOverflow, Github projects, open-source code, programing language API guides, etc. recommend some or all of a series of common unsafe practices, which will increase the risk of compromise as well as making it difficult to certify for PCI and FIPS compliance:

1. Storing credentials in source control or source code repositories: adversary groups regularly take advantage of credentials and certificates located in source code checked out on compromised endpoints, and use it to fan out and infiltrate critical security and operations infrastructure at large target organizations.
2. Database credentials were stolen from a GitHub repository and used to exfiltrate personal information regarding 50,000 Uber drivers:
3. Uber Database Credential Theft
4. An Adobe code signing certificate was stolen and used to sign malware. Users were tricked into installing the malware, because it appeared to come from Adobe:
5. Adobe Code Signing Certificate Theft
6. Passing credentials as command line arguments: credentials in command lines are accessible to anyone who can execute "ps auxww" on a system, including the cracker that happened to find a shell-injection attack against your web server or web stack, and any other unexpected unprivileged users.
7. There are a few programs which overwrite the arguments or retitle themselves with setproctitle() after reading the credentials, but it isn't a complete guarantee since a patient adversary can still catch them in the clear early in the process's life, or by disrupting or slowing the process during launch.
8. Storing certificates and private keys in unsafe formats: unprotected certificates and private keys are easily accessible to anyone who can read them from the filesystem. Many daemons load them as root before dropping privileges, but often retain access after privileges are dropped, to use them during processing, where an attacker can access them if the daemon is exploitable.
9. This happens often in HTTPS servers and SSH environments.
10. Most of the classic HTTPS servers don't support keystores, or require entering passwords during boot, which is cumbersome. SSH agents are hard to configure, and don't have good support for headlessly entering the passphrases needed for automated access.
11. For SSH at least, one can use some special extensions to the authorized_keys file format to restrict the purpose and scope of headless SSH keys to prevent misuse. This is explained in the sshd(8) man page section, AUTHORIZED_KEYS FILE FORMAT.
12. sshd(8) man page
13. Disabling critical SSL/TLS protections for data in transit: most widely-available example code and a lot of open-source code disables SSL/TLS hostname verification and SSL/TLS certificate path validation, as defined in RFC 5280, which should never be disabled in production systems. Data that transfers through unvalidated connections is vulnerable to MITM attacks, thus it cannot be trusted anymore:
14. RFC 5280: PKIX (Public Key Infrastructure for X.509)

Mitigating the Risks

Running software that relies upon unsafe code is a security risk and a liability risk, as well as a business risk. It also cannot be sold to the US Government and many highly-security-conscious organizations. Fortunately, the industry developed a small number of platform-independent techniques for secure credential storage to mitigate these risks:

1. libnss with nss-db and libsecmod: this is the only popular open-source multi-language multi-platform cryptography and SSL/TLS library with an integrated FIPS-compliant keystore. OpenSSL, GnuTLS and most other libraries do not offer much assistance with these requirements. Because of this libnss is widely used in browsers.
2. NSS Library Home Page
3. Several technology vendors have banded together to ensure libnss is FIPS-validated:
4. NSS Library FIPS Validation NSS Library FIPS Certificate
5. Developers can also use NSS bindings available for Java (see below) and Python. We would love to know if anyone is aware of additional bindings.
6. Java PKCS#11 provider (includes support for NSS and smartcard / HSM): this is the gold standard for high-security operations in Java. This includes any of the following:
   • Tomcat, JBoss, Glassfish, Grizzly, or other J2EE servers
   • JCE, JSSE, Bouncy Castle and other popular cryptographic frameworks
   • any other JVM-based environments (Groovy, Scala, Jython, JRuby, etc.)
   Oracle Java 8 PKCS#11 GuideDevelopers can use this adapter with NSS or another FIPS-certified PKCS#11 device to get a FIPS-certified solution.
7. PKCS#12 cryptographic archive file format: this format is not necessarily as heavily-secured and feature-rich as the other formats, but still provides great protection and works in the widest range of environments, because it has been around the longest.

Practical Example

Recently, we needed to access a data pipeline which used a PKCS#12 containing an SSL/TLS Client Certificate and Private Key, from inside of a Python application.

We decided to use pycurl because we had existing products that used it, so it was simpler than using NSS, and it provided sufficient security for the type of data pipeline we were building.

Fortunately, pycurl can be configured to access a PKCS#12 keystore with a passphrase protecting the Client Certificate and Private Key. However the configuration and troubleshooting process was somewhat difficult due to a usability bug in libcurl itself, which we fixed and released to the community:

curl pull request to improve PKCS#12 errors

When troubleshooting our code sample sample below, we found that three different failure points in the PKCS#12 load process used the same non-specific error message, unable to use client certificate (no key found or wrong pass phrase?). This made it difficult to determine the true cause of a failure in the PKCS#12 initialization process. In our patch we added separate messages for each, and logic to retrieve a more detailed error from inside OpenSSL itself for more detail.

This code sample loads a PKCS#12 file in pycurl, and uses the credentials to access a target URL:

import cStringIO import pycurl request = pycurl.Curl() response = cStringIO.StringIO() request.setopt(pycurl.URL, request_url) request.setopt(pycurl.SSLCERTTYPE, "P12") request.setopt(pycurl.SSLCERT, pkcs12_file_path) request.setopt(pycurl.SSLKEY, pkcs12_file_path) request.setopt(pycurl.SSLKEYPASSWD, pkcs12_file_passphrase) request.setopt(pycurl.WRITEFUNCTION, response.write) request.perform()

Conclusion

We encourage everyone to take a little time to think about the sensitive data in a system while it is in transit, in use, or at rest. We enjoy working with the security and technology industry to set good examples, and educate developers about safe ways to handle sensitive credentials. We will keep encouraging everyone to build data pipelines that are secure by default.