How Netflix Secures AWS Cloud Credentials

LAS VEGAS – Netflix has long been the poster child for being an “all-in-the-cloud” organization. The streaming media service relies on Amazon Web Services (AWS) for infrastructure and computing resources that it uses to operate.

With AWS being a public cloud, resources can be shared and there are also multiple sets of credentials and access for resources that could potentially be a risk for a large user like Netflix, making the company a pioneer in cloud security too.

In a session at Black Hat USA, Will Bengtson, senior software security engineer on Netflix’s security tools and operations team, explained some of the steps the streaming media giant takes to identify potentially compromised or unauthorized credentials.

Bengtson said Netflix has hundreds of thousands of virtual server instances running on AWS, and the company needs to know when a credential is being used that should not be. He noted that there are multiple tools on AWS, including GuardDuty, that provide continuous scanning for threats. Still, Bengtson said the attack landscape is large and credentials in the cloud are created and used very rapidly.

The core issue that Bengston and his team were concerned about was how tokens from the AWS Security Token Service (STS) were being used. STS provides credentials for AWS Identity and Access Management (IAM).

CloudTrail security

Among the primary services offered on AWS is the CloudTrail logging service, which Netflix uses to gain insight into how things are running.

Bengston said CloudTrail can be used to track event history for AWS account activity. Netflix has CloudTrail configured to send logs to Amazon S3 storage buckets so further analysis can be performed.

Figuring out if something has been misused starts with a requirement of knowing what is in use. For Netflix, that means being aware of all the IP addresses that are in use, a challenging task given the scale and ephemeral nature of some of the cloud services.

By comparing each IP found in CloudTrail to the list of IPs that Netflix knows it has, it can spot potential anomalous behavior and credential misuse. While that approach can work, Bengston said Netflix is now taking additional steps to identify credential misuse.

One such step is to look through CloudTrail logs to identify the “GetCallerIdentity” function, which is similar to the Linux “whoami” command. Bengston said an attacker would use that function to know what account they are in. He added that Netflix’s legitimate systems never need to use that function as the credentials and access are already known.

Trailblazer open source log security

Going line by line through CloudTrail logs is not a scalable task. As such, Bengston has built a new open-source tool called Trailblazer?that can help determine which AWS API calls are logged by CloudTrail and what they are logged as.

Bengston has also made code publicly available?on GitHub to help organizations hunt for potential compromised credentials in AWS.

Sean Michael Kerner is a senior editor at eSecurityPlanet and InternetNews.com. Follow him on Twitter @TechJournalist.