As the fourth quarter of 2020 draws to a close many security teams are planning their next year of initiatives. Weighing the pros and cons of how to spend the most valuable resource we have available; staff time. In industries regulated by compliance, one of the activities you must do in order to verify that you are secure is to perform a penetration test. In a traditional penetration test, hackers hired by your company attempt to thwart existing security controls to go after a prize.
CISOs the world over would generally agree that contracting that activity out to a third party pays dividends. Those dividends come in the form of time saved, speed to deliver, and often quality of work. It can be a challenge for teams working on a piece of cloud software to consistently think of new and interesting ways to pivot through the architecture.
Before you go signing those contracts and scheduling your cloud pentest for Q1 2021 you may want to take a few proactive steps in order to help focus your penetration testers’ time. The goal of a preparatory effort is to ensure that the individual performing your assessment can focus on vulnerabilities that are unique to your use case. Leverage their expert skills to the fullest without becoming mired in top 10 type findings.
Prepare your team
An often overlooked key factor in penetration testing is the psychological impact of testing the system on your team. Even the most grizzled engineers take pride in their work and may feel uneasy about bringing in a third party to determine if the cloud platform infrastructure can be breached. Of course, this does become easier over time. Engagements tend to fall into two sets of scenarios that require different levels of handling:
- Full knowledge scenario: The entire team knows that the test is occurring and when to expect it. The team actively may participate in developing the requirements and process.
- Scopeless scenario: Only key players on the team have knowledge that a pentest is occurring. Generally there are a set of assets commonly referred to as crown-jewels that the test is designed to acquire. If your pentest is scopeless, decide well ahead of time what the criteria will be in order to ready team members as needed. (Example: They catch the test in progress using detection.) Nothing is more demoralizing than missing out on personal life due to a paid exercise.
If possible, have the team create a roadmap for a “Security Sprint” or “Technical Debt Paydown” to share with your testing team. After all, you don’t need to waste cycles on finding out things you already know to be problems.
Your cloud provider is part of your team as well. Each cloud provider has their own process for sending a notification that a pentest will take place. This is especially important if testing the provider’s underlying APIs is in scope. Customers that fail to notify are often taken offline while the provider determines if it is a legitimate attack. Avoid this at all costs by communicating early and often with your friendly cloud support team.
Focus on Excellent Onboarding
Regardless of the type of scenario and size of the team you should onboard the folks as if they were a member of your internal team. This should include things like how they will get access to your application and cloud, invoice, and communicate with you. Who are the emergency contacts and “get out of jail free” contacts in your Security organization? An additional consideration in a scopeless scenario is also leveraging contact channels outside of your corporate control for things like invoicing and billing meetings. Using corporate owned messaging mediums may very well leak the red team’s identity to your internal security team.
Have an Inventory of Assets
You can not decide what to test if you don’t know what you own. Cloud is more flexible than on premise infrastructure. The environment changes minute to minute. Having an event aggregator or inventory tool like AWS Config, Google Cloud Asset Inventory, and Azure Security: Inventory and Asset Control are all Cloud Native methods to gather a list.
A number of open and free inventory and asset tools exist in the AWS Ecosystem as well like CloudMapper, Prowler, Scout2, and Capital One’s Cloud Custodian if you prefer to use tooling from the open ecosystem.
These tools are only as good as the tagging system driving them. These are the key/value combinations like, “env:production, cost_center:1440, product_line:donut”. The historical driver for tagging has been accounting but tags are advantageous for detection and response, compliance, and security as well. Almost every major cloud now has a way to enforce a minimum level of tagging. When they do not you can address that with a tool like Cloud Custodian.
Be sure to define out of scope assets as well—after all you don’t want to disrupt the business if you know that can be avoided by scoping out or scheduling.
Extract Value Wherever Possible
Most companies that are mature enough to hire out a pentest also have systems that perform some type of detection and response. Part of the scoping with your pentester(s) should be to determine if they will be exercising detection and response capabilities of these tools. This could also mean giving a position of privilege in the infrastructure by adding them to bypass or allow lists.
When attacks or techniques are not detected and stopped ensure that your test will deliver a set of “observable artifacts”. The internal team can then use that timeline and artifact list to write new detections after the test is over.
Observable artifacts might come from the application logs themselves. But in the case that you are testing the “Security of the Cloud” or control plane these artifacts will come from cloud native logs like AWS CloudTrail, GCP StackDriver Logging, or Azure Log Analytics. Take advantage of features that exist in products like AWS Organizations. Using Organizations enables you to enforce logging for 100% of API Calls at a global level, across 100% of your managed accounts. Writing logs once and reading many for forensics has become the gold-standard for these types of logs. If possible, sequester those API Call logs in an account that is separate from your cloud platform’s primary billing heirarchy.
Patch where Practical and Possible
A penetration test leverages vulnerabilities. The purpose should not be to spend time aggregating lists of unpatched systems. If you do not have a regular release, patch process, or patch procedure, now is a good time to develop one and practice it prior to the start of a red team exercise. If you are using a platform like Amazon EC2, patching might take place by regularly rolling out new AMIs or patching using the Systems Manager Agent. If you use containers or serverless services, user patching will consist of regular rebuilds of lambda layers, dependencies, and containers (as appropriate). Ensure that this becomes a part of your regular operations pipeline, otherwise one-time patch management could skew your results towards false positives.
Identity and Access Management
One of the most nuanced and overlooked services in cloud infrastructure is Identity and Access Management:
- How do users access your cloud platform?
- How do applications authenticate in the cloud platform?
- How do you manage access across account boundaries and across clouds?
As an industry we know that cloud control plane access is most secure when users are authenticating using single sign-on. Additionally, that authentication should take place with more than a single factor. Machine-to-machine authentication should use strong cryptographic controls as proof of identity. Despite their efficacy, these steps are often overlooked or under-resourced due to the fact they are not required in most compliance standards.
Taking basic steps to implement a single sign-on pattern pays huge dividends. Most cloud data breaches start with a misconfiguration, access key leak, or phishing campaign. Moving to a model that enforces a single way that developers and machines assert strong identity, benefits detection and defensive posture.
As you move forward towards pentests of the cloud becoming more normative this will get easier. Don’t worry if this feels daunting at first. You will get better with time! Whether this is your first or fiftieth time performing a pentest now is a great time to level set with your team, schedule a security or tech debt paydown sprint, patch where possible, and evaluate your IAM perimeter to make your pentest as productive and fruitful as possible.
Andrew Krug @andrewkrug will speak on Pentest Preparation next week at the Wild West Hackin’ Fest Cloud Roundup December 10, 2020 2:00PM EST. Bring your questions and your cowboy hats to this live presentation. https://wildwesthackinfest.com/the-roundup/cloud-pentesting/