Continuous integration and development (CI/CD) pipelines are the most dangerous potential attack surface of the software supply chain, according to NCC researchers.
The presentation at last week’s Black Hat security conference by NCC’s Iain Smart and Viktor Gazdag, titled “RCE-as-a-Service: Lessons Learned from 5 Years of Real-World CI/CD Pipeline Compromise,” builds on previous work NCC researchers have done on compromised CI/CD pipelines.
A CI/CD pipeline is fundamentally remote code execution (RCE), and there are many paths to compromise it regardless of a company’s size.
Because organizations trust their pipelines too much, compromising these channels is usually a good path for hackers, as they would likely gain unauthorized access to critical data and even elevate their privileges to perform further attacks.
See the Best Third-Party Risk Management (TPRM) Tools
CI/CD Approaches Create Risk
CI stands for “continuous integration” and CD means “continuous delivery.” The combination of the two practices allows for automating and monitoring development throughout the entire lifecycle.
Continuous integration means developers can automate processes when code changes. Typically, scripts will run when a modification is merged into specific branches of the project to test, build, and deploy the code.
The idea is to reduce the efforts needed to deploy new code across all environments, which not only impacts developers but also business teams. As enhanced deployment is part of the Twelve-factor app, a set of principles that drive modern SaaS apps, approaches such as CI/CD are supposed to make the process less risky.
The Jenkins open source automation server, for example, is one the most popular solutions on the market. Many companies, including major ones, use it to handle DevOps operations. During their tests, the researchers managed to compromise Jenkins environments in various ways, leveraging misconfiguration in S3 Buckets or in the environment itself.
Indeed, organizations use third-party solutions like Jenkins to speed up their processes, but too permissive configurations ultimately lead to privilege escalation.
Also read: How Hackers Compromise the Software Supply Chain
Third-party Solutions Make CI/CD Pipelines Vulnerable
Researchers found attack paths in popular platforms that provide advanced CI/CD functionalities, such as GitLab. They took advantage of sensitive features in GitLab Runners to escalate privileges and grab secrets.
Again, configurations that are too permissive allow any user with rights to commit code to access passwords and secrets, for example, when the data is stored in plain text in environment variables.
Researchers also exploited privileged processes and containers (e.g., in Docker and Kubernetes) that run, for example, as root, while some solutions support rootless build.
These attacks may look quite classic, and robust configuration would not allow such escalations, but in reality, many organizations have poor practices and choose convenience over security. Some teams might not even know there are additional security settings or find them incompatible with deadlines.
Researchers reported interesting results in their 10 scenarios. The final one consisted of pretending “you have compromised a developer’s laptop.” After chaining some exploits, they managed to access the Jenkins master node and dump all variables, which ultimately gave them full access to the production environment, as the deployment pipeline was given too many permissions.
Also read: New Open-source Security Initiative Aimed at Supply Chain Attacks
How to Secure your CI/CD Pipeline
CI/CD pipelines are critical environments hackers will attack if given a chance. Because of their complexity and the time needed to configure them properly, many teams give full permissions to third-party tools that are not supposed to get so many rights.
Here are some practical security steps:
- The least privilege approach (everywhere in the chain) is the right direction here. Assign the minimum necessary rights to people and third-party tools. It certainly takes more time to configure and can even trigger some bugs in tests and elsewhere, but it’s worth it.
- Enable two-factor (2FA) and multi-factor authentication (MFA) whenever it’s possible, especially for administrator accounts.
- Never ever use environment variables for credentials, or at least encrypt them.
- Identify the weakest points in your pipelines that require additional security measures.
- Implement security checks, especially for committers.
- Have a strong vendor policy. Continuous deployment delivers code your teams do not maintain in production, making supply chain attacks attractive for hackers.
The setup phase is highly critical. Be extra vigilant during the first stages of your projects, as it’s usually at this point that you find misconfigured instances that can lead to privilege escalation.
Least privilege, or zero trust principles, figure prominently in Smart and Gazdag’s work, but they also recommend additional controls like network segmentation and patch management.
Further reading:
