Securing the Modern Software Supply Chain
Third-party dependencies are a fact of life for modern software development. The code that runs the world today is rarely written from scratch. In 2021, developers borrowed more than 2.2 trillion open-source packages or components from third-party ecosystems. 99% of codebases contain open-source code, and between 85% to 97% of enterprises leverage open-source software.
It’s not just the code, either. Third-party development tools, pipelines, and infrastructure proliferate modern development environments—from public and private repositories and source code management systems to build servers, artifact registry systems, and Infrastructure-as-Code.
Together, the long chain of code, library, and package dependencies and third-party tooling employed during the software development lifecycle comprise the modern software supply chain.
They also constitute an enormous vulnerability and attack surface—and one bad actors are increasingly taking note of and exploiting. In 2021, supply chain attacks increased 650% from the year prior, and Gartner estimates that by 2025, 45% of all organizations will experience attacks on their software supply chain.
Supply chain threats are especially pernicious because of the number of attack vectors modern security and engineering leaders need to defend: Compromise in any link in the long chain of dependencies that goes into the development and deployment of an enterprise’s software—significant parts of which exist outside the enterprise’s own development environments—can propagate downstream, impacting the organization and hundreds or even thousands of similar targets.
In recent years, notable attacks like Event-stream, Codecov, and SolarWinds have illustrated the magnitude of the issue. One compromised an open-source package, another a developer tool, and a third proprietary software. But in each case, attackers were able to breach thousands of targets via a single link, several degrees of separation away from their eventual targets.
The proliferation of these types of widespread and cascading breaches has brought the discussion of software supply chain security to the top of many CISOs’ priority lists for 2023, and even the desk of the President of the United States in the past year. According to Gartner, 44% of the companies surveyed reported they will increase supply chain security substantially 5–10% and 18% expect to increase more than 10%.
At Menlo, we see the shifting threat landscape as an opportunity for next-generation software supply chain security providers to emerge and build modern solutions for the novel and expanding threats today’s enterprises face. In this article, we’ll outline the key vulnerabilities these vendors must defend, address why legacy solutions have proven insufficient, and highlight the most promising early-stage players we’re currently tracking in the space.
Key Vulnerabilities
Broadly speaking, software supply chains are susceptible to three main categories of threats that occur across the entirety of the software development lifecycle:
- Source threats: These threats impact the integrity of source code, and reflect the potential for any adversary (e.g., bad external actors, malicious insiders, or even negligent insiders) to introduce a change to the source code that does not reflect the intent of the software producer.
- Build threats: These threats compromise software when companies build or package it, or trick consumers of upstream software into using a bad version during the build process. They reflect the potential for an adversary to introduce behavior to a package that is not reflected in the source code, or to build from a source, dependency, or process not intended by the software producer.
- Deployment and runtime threats: These threats reflect vulnerability or misconfigurations in deploying or running software, and are often closer to the domain of traditional application security.
The popular SLSA framework further breaks these three categories into eight distinct types of vulnerabilities in enterprise software supply chain integrity:
In years past, the primary threat among these have been passive adversaries focused on the “lowest hanging fruit”—which meant exploiting vulnerable dependencies, and especially publicly disclosed open-source dependencies.
The infamous 2017 Equifax data breach is perhaps the most well-known example of this. The company had been using the open-source Apache Struts as its website framework for systems handling credit disputes from consumers, and had failed to update this software with a key security patch after the discovery of a critical exploit in the code. Within two months of the patch’s announcement, hackers had discovered and then used the vulnerability to gain access to Equifax’s internal servers and expose the private records of over 143 million Americans, ultimately costing the company up to $700 million in fines and perhaps more in reputational damage.
Today’s hackers are both more aggressive and creative. Bad actors are no longer waiting for public disclosures to pursue an exploit, but instead taking the initiative to inject new vulnerabilities into open-source projects and other third-party code upstream of their ultimate targets. They are also branching out from code-level vulnerabilities to targeting misconfigurations and other security gaps in development pipelines and SDLC systems themselves.
A few examples of these more recent attacks include:
- SolarWinds: In one of the largest cyber-espionage attacks of all time, bad actors gained access to builds of SolarWind’s Orion software, which several U.S. federal agencies and large enterprises like Microsoft and Cisco use for infrastructure observability. The hackers injected malicious code into the software’s update binaries, and over 18,000 customers were impacted when they pulled the bad updates from a source they trusted.
- Codecov: Bad actors compromised a Codecov server with leaked credentials to inject malicious code into a Bash Uploader script, which installs the tool in customer CI environments. Over a period of two months, Codecov’s customers—including Twilio, Monday.com, and Hashicorp—installed compromised artifacts instead.
- LastPass: Using developer account credentials, bad actors gained access to portions of the LastPass developer environment and partially exposed the company’s source code and other proprietary technical details.
The Software Supply Chain Landscape
The Traditional Approach
Part of the reason why the software supply chain has become such an exposed attack surface recently is because of how ineffective traditional approaches toward addressing modern vulnerabilities have proven to be.
Many of the most popular solutions used by enterprises today focus narrowly on code scanning at defined points in time during the SDLC. These include:
- Software composition analysis (SCA): SCA solutions scan open-source code against libraries of known vulnerabilities like NVD (National Vulnerability Database) and CVE (common vulnerabilities and exposures).
- Static application security testing (SAST): SAST also scans source code for weak or insecure coding, but focuses on an enterprise’s proprietary code vs. open-source components. In the SDLC or CI/CD pipeline, these solutions often act as quality gates for components to be promoted to the next stages of the pipeline.
- Dynamic application security testing (DAST): DAST performs security testing on running applications, testing for a variety of real threats like SQL injections, OS injections, and cross-site scripting without requiring access to source code. This is a form of black box testing, which can identify application vulnerabilities from an attacker’s perspective.
A number of market-leading vendors in these categories, including Checkmarx and Black Duck (acq. by Synopsys), were founded almost two decades ago, before the widespread adoption of cloud-native software or DevOps and DevSecOps methodologies. But even solutions from more modern vendors like the developer platform Snyk have proven to be a necessary but by no means sufficient defense against modern software supply chain attacks. The primary issues with these solutions are that they are:
- Reactive instead of proactive: Vulnerability scans and application security testing only identify known code vulnerabilities (e.g., NVDs, CVEs) and insecure code patterns (e.g., OWASP top 10 lists). They are far less adept at identifying novel source threats or proactively minimizing attack surfaces.
- Incomplete in coverage: SCA can scan source code for dependency vulnerabilities, but modern software development often moves so fast that tools only scan deltas in code changes to keep pace with modern release cadences—creating SDLC assets not covered by SCA scanners. SCA, SAST, and DAST tools also only function at SDLC checkpoints defined by QA engineers or developers, but are blind to supply chain attacks that might occur during debugging or runtime (e.g., Log4shell in production environment). Finally, all scanners are limited to code-level only visibility vs. pipeline-level security.
- Noisy and time-consuming: Scanners provide a long list of nested issues that both contain a high number of false-positives, and are difficult to investigate and address. This not only leads to slower development timelines and a high-friction experience for developers who are tasked with separating signal from noise, but also represents a haphazard approach to addressing critical security issues during development.
Emerging Players
Fortunately, a new generation of early-stage vendors are emerging to address the security challenges exposed by more recent supply chain attacks. Often complementary to the widely deployed scanners that enterprises use today, these solutions can be broken into two segments:
Code-level solutions: Addresses vulnerabilities at the code level, including by inventorying dependency vulnerabilities, reducing bloated attack surfaces (i.e., container images), enforcing and verifying provenance, and continuously scanning for compliance in production. These solutions are part of the “shift left” movement that has gained momentum in security circles in recent years, as organizations look for risks in development versus only in production. The VP of Engineering often has significant input into these buying decisions, if they are not the owner.
Pipeline-level solutions: Addresses vulnerabilities and misconfigurations in end-to-end SDLC pipelines not usually covered by code scanners, including development tools, code repositories, source code management systems, build servers, artifact registry systems, third-party apps, and IaC. These solutions are often operated by the DevOps or DevSecOps team, and sponsored by the CISO.
Below, we’ll dive deeper into a selection of categories within each of these two segments.
Code-level solutions
Container images. One way to address the increasing number of vulnerabilities at the code level is to make the software development lifecycle secure by default, as opposed to trying to constantly scan for and hunt down vulnerabilities in development.
To be sure, this is no easy task. “Secure by default” requires a fundamental shift in how software is built, starting at the very beginning—with the container base image. As the bottom layer of the container stack, base images are crucial because software applications inherit all the security properties of this base image, including any security debt.
Many enterprises today use open-source base images like Alpine, Ubuntu, and Debian—options that tend to lag behind upstream updates, sometimes with tens to hundreds of known vulnerabilities.
Players like Chainguard are innovating in this space with their introduction of “quiet” base images—or ones without extraneous packages and maintained to have no known vulnerabilities. Developers that build with Chainguard can thus build “secure by default,” minimizing the noisy din of security scanners that are typically needed to catch and remediate all the vulnerabilities that exist in more popular open-source base images.
Container optimization. Organizations often deploy containerized software to production containing significant amounts of unnecessary and unused software, including libraries, OS binaries, tools, package managers, and more. The next step in building secure by default then is to minimize the code footprint companies ship to production—and thus reducing code complexity, software attack surface, and aggregate risk.
Slim.ai and Rapidfort provide two examples. Both companies automatically optimize containers to only what is needed. Developers can provide fine-grained configurations or use vendor-provided recommendations, and the solutions offer post-optimization analyses detailing which files, packages, and vulnerabilities were removed.
Provenance: Software Bill of Materials (SBOM) are an increasingly expected requirement from software releases. The term refers to a static list of mapped open-source and proprietary software dependencies, including name, version, license, and vulnerabilities. The tool is used to establish provenance and provide greater transparency, expediting resolution of security and compliance issues.
More recently, the term has inspired a raft of other variant “BOMs”: Ox Security coined the term PBOM (Pipeline Bill of Materials), which is a dynamic list of software lineage—essentially a continuously updated map containing everything an SBOM would contain, plus a record of all the infrastructure a piece of software has passed through (e.g., pipeline branches, builds, pull requests, tickets, known issues). The purpose here is fuller traceability. On the other hand, Rapidfort introduced the “RBOM” (Real Bill of Materials), which is an SBOM post-container optimization—where the purpose is more noise reduction to minimize vulnerability alerts during scanning.
As one founder put it to us, however, everyone might be producing their own “BOM”—but ultimately, “BOM” generation is less interesting than how the tool is consumed. Chainguard’s SBOMs—which are generated as part of the build process—are leveraged for continuous verification, ensuring packages in development remain in compliance with no vulnerabilities even post-deployment. Legit Security’s SBOMs serve more security, compliance, and observability use cases—inventorying software components and facilitating vulnerability scanning for faster security responses and license tracking.
Dependency management. These developer tools automatically evaluate dependencies for security and operational risk and provide guardrails for developers to select safer dependencies in sourcing third-party code.
Endor Labs accomplishes this algorithmically by scoring dependencies with its proprietary engine. Tidelift does this by defining open-source standards and policies and maintaining a catalog of pre-vetted and approved open-source components, and partnering directly with OSS maintainers to ensure their projects continue to meet enterprise standards.
Secrets detection and mitigation. “Secrets” are anything that allows access to a system—including API keys, private keys, database credentials, or security certifications—and unmanaged secrets proliferate all aspects of the modern development environment, from source code to build logs, kubernetes clusters, and even unpublished version histories. Especially nefarious are “hard-coded secrets,” or plain-text credentials embedded into source code, which allows lateral movement for any attackers that manage to compromise a company’s code repository.
Companies like Truffle Security and GitGuardian are tackling this issue by offering—among other tools—pre-commit hooks to scan staged changes for secrets, allowing developers to self-remediate potential vulnerabilities before they appear. Arnica, on the other hand, observes secrets organization-wide versus at the developer workstation, enabling comprehensive coverage and real-time enforcement of a zero-new hardcoded secrets policy beyond just the parts of the CI/CD pipeline protected by pre-commit hooks.
Doppler takes more of a management approach, offering a centralized secrets storage that developers then securely reference versus the traditional approach of hard-coding secrets. The company also offers enterprises the ability to automatically rotate their database and API key secrets on a cadence without downtime through infrastructure integrations, further reducing the risk of a leak or breach.
Capability detection. The final category in emerging code-level security solutions is a new approach to static analysis that, unlike traditional vulnerability scanners, offers package behavioral analysis in real time to detect unknown vulnerabilities versus just NVDs and CVEs. Socket, for instance, examines open-source code to flag when dependency updates introduce new risky API usage such as network, shell, or filesystem.
Pipeline-level solutions
CI/CD posture management. Today, software supply chain security often stops at the code level. Few enterprises even have visibility into their full SDLC systems, and those that do often have to track the infrastructure in use within their development environments manually. The core innovation of CI/CD posture management vendors is to bring security to the pipeline level, automating the SDLC discovery process and enforcing security best practices throughout.
For vendors like Tromzo, this starts with addressing vulnerable misconfigurations in the SDLC and IaC: The vendor’s CI/CD posture management solution ensures build servers require authentication; limits the ability to create public repositories; limits publicly writable S3 buckets; and sets security keys to expire by default. The company also addresses potential pipeline vulnerabilities by constraining risky development practices like executing third-party resources before verification (e.g., a checksum or manual review) or referencing images in a build that may be changed externally.
Developer access management. In addition to CI/CD posture management, Legit Security and Arnica have both also extended pipeline security controls to developer access management.
Legit ensures multi-factor authentication is required to access source code, protects source code with code reviews and branch protection, regularly audits third-party integrations with access to source code, and sets up alerts for potentially insecure activities like bypassing a code review.
Arnica additionally adds behavioral components to access management to account for the dynamic nature of source code repository contributions—reducing developer permissions based on historical patterns of access, and automating the process for developers to regain permissions via communication tools like Slack. This allows companies to reduce the blast radius of a potential attack by an account takeover or insider threat.
Anomalous behavior detection. Arnica’s behavioral capabilities extend beyond developer access management to detection and remediation across the full CI/CD pipeline. One of Arnica’s key differentiators is that the vendor establishes behavioral profiles for all developers within an environment and flags anomalies detected in either the systems they interact with or the code patterns within their commits. If anomalous behavior is detected, the company also catalyzes automated workflows to mitigate developer risks in real time.
What’s Next
Since 1976, we at Menlo have had the privilege of supporting some of the earliest and most iconic cybersecurity companies to emerge from the Silicon Valley ecosystem. Our cybersecurity portfolio includes investments in IronPort (acquired by Cisco), Q1 Labs (acquired by IBM), and BitSight—and more recently, companies including Abnormal Security, Cequence Security, Strata, and others.
We invest in founders seeking to build large and enduring companies, and we believe the software supply chain security space is primed for just that type of disruptor today. If you are a founder building in this space, shoot us a note. We’d love to hear from you:
Venky Ganesan (venky@menlovc.com)
