Preventing Supply Chain Attacks like SolarWinds

In late 2020, it was revealed that the SolarWinds Orion software, which is in use by numerous US Government agencies and many private organizations, was severely compromised. This was an incredibly dangerous set of supply chain compromises that the information technology community (including the Open Source community) needs to learn from and take action on.

The US Cybersecurity and Infrastructure Security Agency (CISA) released an alert noting that the SolarWinds Orion software included malicious functionality in March 2020, but it was not detected until December 2020. CISA’s Emergency Directive 21-01 stated that it was being exploited, had a high potential of compromise, and a grave impact on entire organizations when compromised. Indeed, because Orion deployments typically control networks of whole organizations, this is a grave problem. The more people look, the worse it gets. As I write this, it appears that a second and third malware have been identified in Orion.

Why the SolarWinds Attack Is Particularly Noteworthy

What’s especially noteworthy is how the malicious code was inserted into Orion: the attackers subverted something called the build environment. When software is being developed it is converted (compiled) from source code (the text that software developers update) into an executable package using a “build process.” For example, the source code of many open source software projects is then used in software that is built, compiled, and redistributed by other organizations, so that it is ready to install and run on various computing platforms. In the case of SolarWinds’ Orion, CrowdStrike found a piece of malware called Sunspot that watched the build server for build commands and silently replaced source code files inside the Orion app with files that loaded the Sunburst malware. The SolarWinds Orion compromise by Sunspot isn’t the first example of these kinds of attacks, but it has demonstrated just how dangerous they can be when they compromise widely-used software.

Unfortunately, a lot of conventional security advice cannot counter this kind of attack: 

• “Only install signed versions” doesn’t help because this software was signed.
• “Update your software to the latest version” doesn’t help because the updated software was the subverted one. 
• “Monitor software behavior” eventually detected the problem, but the attack was quite stealthy and was only detected after tremendous damage was done. 
• “Review source code” is not a certain defense either. It’s not certain that the source code changes were visible to developers. The changes were carefully written to look like expected code, so developers might not have noticed even if they were visible. However, the attackers had control of the build environment, so they could have inserted the attack without it being visible to software developers.

SolarWinds’ Orion is not open source software. Only the company’s developers can legally review, modify, or redistribute its source code or its build system and configurations. If we needed further evidence that obscurity of software source code doesn’t automatically provide security, this is it.

Recommendations from The Linux Foundation 

Organizations need to harden their build environments against attackers. SolarWinds followed some poor practices, such as using the insecure ftp protocol and publicly revealing passwords, which may have made these attacks especially easy. The build system is a critical production system, and it should be treated like one, with the same or higher security requirements as its production environments. This is an important short-term step that organizations should already be doing. However, it’s not clear that these particular weaknesses were exploited or that such hardening would have made any difference. Assuming a system can “never be broken into” is a failing strategy.

In the longer term, I know of only one strong countermeasure for this kind of attack: verified reproducible builds. A “reproducible build” is a build that always produces the same outputs given the same inputs so that the build results can be verified. A verified reproducible build is a process where independent organizations produce a build from source code and verify that the built results come from the claimed source code. Almost all software today is not reproducible, but there’s work to change this. The Linux Foundation and Civil Infrastructure Platform has been funding work, including the Reproducible Builds project, to make it possible to have verified reproducible builds.

The software industry needs to begin shifting towards implementing and requiring verified reproducible builds. This will not be easy. Most software is not designed to be reproducible in their build environments today, so it may take years to make software reproducible. Many changes must be made to make software reproducible, so resources (time and money) are often needed. And there’s a lot of software that needs to be reproducible, including operating system packages and library level packages. There are package distribution systems that would need to be reviewed and likely modified. I would expect some of the most critical software to become reproducible first, and then less critical software would increase over time as pressure increases to make more software verified reproducible. It would be wise to develop widely-applicable standards and best practices for creating reproducible builds. Once software is reproducible, others will need to verify the build results for given source code to counter these kinds of attacks. Reproducible builds are much easier for open source software (OSS) because there’s no legal impediment to having many verifiers. Closed source software developers will have added challenges; their business models often depend on hiding source code. It’s still possible to have “trusted rebuilders” worldwide to verify closed source software, even though it’s more challenging and the number of rebuilders would necessarily be smaller.

The information technology industry is generally moving away from “black boxes” that cannot be inspected and verified and towards components that can be reviewed. So this is part of a general industry trend; it’s a trend that needs to be accelerated.

This is not unprecedented. Auditors have access to the financial data and review the financial systems of most enterprises. Audits are an independent entity verifying the data and systems for the benefit of the ecosystem. There is a similar opportunity for organizations to become independent verifiers for both open source and closed source software and build systems. 

Attackers will always take the easiest path, so we can’t ignore other attacks. Today most attacks exploit unintentional vulnerabilities in code, so we need to continue to work to prevent these unintentional vulnerabilities. These mitigations include changing tools & interfaces so those problems won’t happen, educating developers on developing secure software (such as the free courses from OpenSSF on edX), and detecting residual vulnerabilities before deployment through various detection tools. The Open Source Security Foundation (OpenSSF) is working on improving the security of open source software (OSS), including all these points.

Applications are mostly reused software (with a small amount of custom code), so this reused software’s software supply chain is critical. Reused components are often extremely out-of-date. Thus, they have many publicly-known unintentional vulnerabilities; in fact, reused components with known vulnerabilities are among the topmost common problems in web applications. The LF’s LFX security tools, GitHub’s Dependabot, GitLab’s dependency analyzers, and many other tools & services can help detect reused components with known vulnerabilities.

Vulnerabilities in widely-reused OSS can cause widespread problems, so the LF is already working to identify such OSS so that it can be reviewed and hardened further (see Vulnerabilities in the Core Preliminary Report and Census II of Open Source Software).

The supply chain matters for malicious code, too; most malicious code gets into applications through library “typosquatting” (that is, by creating a malicious library with a name that looks like a legitimate library). 

That means that Users need to start asking for a software bill of materials (SBOM) so they will know what they are using. The US National Telecommunications and Information Administration (NTIA) has been encouraging the adoption of SBOMs throughout organizations and the software supply chain process. The Linux Foundation’s Software Package Data Exchange (SPDX) format is a SBOM format by many. Once you get SBOM information, examine the versions that are included. If the software has malicious components, or components with known vulnerabilities, start asking why. Some vulnerabilities may not be exploitable, but too many application developers simply don’t update dependencies even when they are exploitable. To be fair, there’s a chicken-and-egg problem here: specifications are in the process of being updated, tools are in development, and many software producers aren’t ready to provide SBOMs.  So users should not expect that most software producers will have SBOMs ready today. However, they do need to create a demand for SBOMs.

Similarly, software producers should work towards providing SBOM information. For many OSS projects this can typically be done, at least in part, by providing package management information that identifies their direct and indirect dependencies (e.g., in package.json, requirements.txt, Gemfile, Gemfile.lock, and similar files). Many tools can combine this information to create more complete SBOM information for larger systems.

Organizations should invest in OpenChain conformance and require their suppliers to implement a process designed to improve trust in a supply chain.  OpenChain’s conformance process reveals specifics about the components you depend on that are a critical first step to countering many supply chain attacks.

Conclusion

The attack on SolarWinds’ Orion will have devastating effects for years to come. But we can and should learn from it. 

We can:

1. Harden software build environments
2. Move towards verified reproducible builds 
3. Change tools & interfaces so unintentional vulnerabilities are less likely
4. Educate developers (such as the free courses from OpenSSF on edX)
5. Use vulnerability detection tools when developing software
6. Use tools to detect known-vulnerable components when developing software
7. Improve widely-used OSS (the OpenSSF is working on this)
8. Ask for a software bill of materials (SBOMs), e.g., in SPDX format. Many software producers aren’t ready to provide one yet, but creating the demand will speed progress
9. Determine if subcomponents we use have known vulnerabilities 
10. Work towards providing SBOM information if we produce software for others
11. Implement OpenChain 

Let’s make it much harder to exploit the future systems we all depend on. Those who do not learn from history are often doomed to repeat it.

David A. Wheeler, Director of Open Source Supply Chain Security at the Linux Foundation