What is Software Supply Chain Security and why it is important?

Guy Podjarny: 

On December 8th, 2020, FireEye, one of the most respected cybersecurity firms in the world, announced they were the victim of a nation-state attack, and that their Red Team toolkits, the primary tools used to perform their ethical hacking, were compromised. 5 days later, they reported finding evidence that the attackers got in through a backdoor in SolarWinds’s Orion software, Solarwinds is a vendor they used to monitor their IT systems across the company. In the following weeks and months, SolarWinds confirmed the breach, and hundreds of similar victims came up, ranging from tech giants to critical infrastructure to governments, all breached because this vendor they used – and trusted – was compromised upstream. 

Roughly 4 months later, on April 15 2021, a company called CodeCov reported it was breached. CodeCov’s tools integrate into customers’ software pipelines to help developers assess how much of their code is actually exercised by their automated tests. The attackers managed to compromise the CodeCov components that run inside the build system itself, granting the attackers access to the build systems of hundreds of customers of Codecov and allowing them to steal precious credentials, which they’ve done and potentially modify shipped code from those customer’s pipeline which, fortunately, it appears they haven’t done.

Adding fuel to the fire, late last year, on December 10th, 2021, an extremely critical vulnerability was found in a massively popular open source library called Log4j. Log4j is used to log system and user actions in Java applications, and is extremely prevalent, it used by practically any company with a bit of tech mileage and using Java. The vulnerability, dubbed Log4Shell, is a Remote Command Execution vulnerability, and the easy exploit made it trivial for attackers to run commands on victim’s servers. This combination of the widespread usage and ease of exploitation sent much of the security industry into a spin, a lot of sleepless nights trying to quickly find and patch all the places this library was used.  

These three anecdotes, and many like them, are all examples of software supply chain security issues. They’re each quite different from one another, but they share the fact that the organization was put at risk because of a piece of software they were consuming, as opposed to something they wrote. It was an upstream software component the company was using, be it open source or commercial that had a security problem, be it a vulnerability or a breach, and therefore the company itself is now at risk. 

[00:02:55] ANNOUNCER: Hi. You’re listening to The Secure Developer. It’s part of the DevSecCon community, a platform for developers, operators and security people to share their views and practices on DevSecOps, dev and sec collaboration, cloud security and more. Check out devseccon.com to join the community and find other great resources.

This podcast is sponsored by Snyk. Snyk’s a developer security platform helps developers build secure applications without slowing down, fixing vulnerabilities in code, open source containers, and infrastructure as code. To learn more visit snyk.io/tsd. 

[00:03:41] Guy Podjarny: The growing number and impact of software supply chain attacks and vulnerabilities has put it firmly in the headlines. From a US government executive order, through a fast growing open source foundation to a lot of open source and standards activity, software supply chain security is a hot topic, and it’s moving fast. 

It’s also very confusing. The breadth of the space, the urgency to do something about it and the rapidly changing landscape combine very poorly, make it very hard to feel you have a handle on the topic, and know what you need to do. 

Which is precisely where this mini series comes in!

In the next few episodes, we’ll combine our analysis of this space of supply chain security with a series of interviews that we’ve had a chance to do with other supply chain security experts. And we are going to try and create a clearer picture of what this topic involves, and what’s the state of the land. And try to help you understand what you should be doing about it.

In this first episode, we’ll focus on defining the problem. We’ll break up the key pillars of Supply Chain Security, and talk about what you should care about most – and why. 

The second episode will get specific, covering the key terms you should know and players you should track, as well as talk about some of the most prominent or promising projects in this space, so you can deep dive.

In the third episode, we’ll give examples from practitioners actually implementing supply chain security in their organizations so that you can learn and choose which of these practices you want to adopt, and we’ll talk a bit about maturity levels, how you get started vs how you continue. 

Then lastly, in the fourth episode we’ll cast our eyes forward, and talk about industry motions, what can and is being done to help the ecosystem deal with this problem, and what key changes you might expect to come down the road.

[00:05:37] Simon Maple: To be able to address any challenge, we must first understand the scope of what we are trying to solve. That’s why it’s very important to define what software supply chain security is. What people should think about when someone else says software supply chain security, what’s the scope that people are covering now? Now Guy Podjarny, Founder of Snyk and Eric Brewer, Google fellow, VP, and longtime faculty at UC Berkeley will take us through what software supply chain is really all about.

[00:06:02] Guy Podjarny: When we say supply chain security for brevity. And you know, most people do. But really we’re talking about software supply chain security and ignoring, maybe some other topics of supply chain security, literally of how has someone physically brought in some piece of equipment to your office or something like that.

Within software supply chain security? There’s another sort of subdivision which is commercial and open source which we’ll get into. But at a high level software supply chain security is how secure is your use of the software that you are consuming versus what you are producing, and the journeys that they undergo. It’s a very meta interpretation. it includes any commercial and open source software that you are consuming, knowing what you’re using, knowing what are the known security flaws in it, whether it’s vulnerabilities or it’s more sort of incident style, you know, compromises and and responses to them. Knowing the journey that they have kind of undergone. So, knowing whether you are consuming what you actually think that you’re consuming, you know where they sort of build securely. Is nobody tampering with this piece of software that you’re consuming as you go through it.

[00:07:16] Eric Brewer: Well, like any supply chain, there’s many things that go into something you use, like your car. Like, why do you trust the car? It’s safe to drive in at very high speeds. And it’s not because you know everything about it. It’s because you kind of trust indirectly through a chain of vendors who actually then review each other and build a layer of trust. That’s more of a social process.

For software, we need the same kind of thing, especially in open source where we have a huge number of dependencies and we’re pulling from all kinds of suppliers. Where, honestly, we don’t typically know much about our supply chain today. And that has become a critical threat to the nation and the world. 

[00:07:52] Jonathan Meadows: This scope is huge when you look at supply chain security, right? And it’s really the way that you ingest, build, ingest, and deploy software as an entity. And really, your supply chain really does come or start right down at the initial dependency level, all of your dependencies for the application through all of the components that you bring into an enterprise, whether that’s within routers, or databases, or the applications that you then build. And then you need to continue that journey all the way through to how you distribute that software to your clients.

And it’s a true chain, so that multiple different enterprises within that chain, need to look at that supply chain and you need to be aware of what that looks like as well, from effectively who’s building the chipsets in your servers, through to who’s putting the firmware onto them. Applications on top of those, through to which you can see that as an enterprise and then delivering an application to your customer.

So, it’s a really broad problem that obviously multiple different industries also tackle. And the difficulty with a supply chain and supply chain security is it really is as secure as the weakest link. So, you really do need to get a good grip on where you’re getting your software ingredients from, as you’re looking at that problem.

[00:02:34] Adrian Ludwig: It’s a multi-armed octopus. I’m not sure what the right metaphor is here. Probably for everywhere at this point is that the vast majority of the software that you actually ship comes from somewhere else. The percentage of the code that’s written by developers inside of your environment and under the remit of your specific organization is maybe 10%, maybe 15% of what it is that you actually ship. Whether you’re in the cloud, or whether you’re producing a mobile device, it’s a conglomerate of third-party services, third-party software. 

Could be licensed software through a negotiated license, could be open-source software. The vast majority is probably open-source at this point, which is super interesting, because they don’t even know that you’re using it, so they don’t know that they’re part of your supply chain.

[00:09:52] Simon Maple: On the secure developer we speak to many experts and they all pretty much entirely agree that Software Supply Chain security is the aggregation of all of the complete journeys of every piece of software that you consume from inception, and that includes all the systems they traveled through, all the people that touch them along the way. When we spoke to Jim Zemlin, Executive Director at Linux Foundation, he took us through this journey of the pieces of software

[00:10:19] Jim Zemlin: The most important thing to understand before you start talking about software supply chain security for someone who’s not familiar with the space is to understand how code flows around the world. It sorts of starts with a developer who works in a version control system writing software. A lot of people think writing software is just sort of math, and if, this, then, that type of thing. But there’s a real art to software and a real elegance. And you have these thousands and thousands of creators out there in the open source community that write software. And so they do a lot of that work on GitHub. But they also do it in places like GitLab or other version control systems that are used every day by millions of developers all over the world. 

Once they write that code, they package it up while they build the software. They use a build system to compile the software so it’s like actually usable and can run. And then they package that up, combine it with other packages. Today, there are package managers, little software factories that kind of combine all this stuff together. And then that’s integrated together and off to an IoT device, an operating system, a cloud service, whatever it is. 

But when I think of the supply chain, I think of starting with a developer writing code, compilation, package management, all the way to an end user, right? And most people, I think, when they think of the software supply chain, they think somebody writes some software and all of a sudden, magically, I’m using it on my Android device, or my Chromebook, or as a cloud service with the website I happen to be logging into, or whatever it is. And there’s a lot of different points in between that. And that’s how I like to think about supply chain security, in terms of how code flows. 

The second thing you start thinking about is, at every point, when software is handed off down that production frame, as it flows from the upstream and open source, which is the developers working on it in a repo, or in a version control system, to build system to a package manager and so forth. At each point in that process, crazy things can happen. As you know, your company specializes in understanding all of those places. You can have a bug in software. This is sort of the most common way that vulnerabilities happen. It’s not some malicious plan. It’s just a developer is writing a lot of software. And they write it in a way that there’s a mistake that that might have happened, or they may not have fully understood security aspects of software development and really like, “Hey, I got to get this next feature written and ship this code.” That goes out. And just that unintentional bug can produce a real security problem. But that’s only the beginning of how code flows, right? So I made that problem there. 

You can then, when you’re combining that software with other packages, bring in a package unintentionally that may have malicious code in that. People can type squad names of common packages that maybe a developer typed in the dependency name wrong for a package they wanted to use when they combine it all together into some code. And there’s a malicious actor out there like kind of squatting on JavaScript package that all of a sudden produces a vulnerability. You can inject vulnerabilities at different points in that supply chain. There’s just a ton of different attack vectors as you go down throughout the supply chain. And the that’s why everybody talks about the supply chain in that way, bypass code review, compromise source control systems. You can compromise at the build pass paths form, you can bypass CICD, you can use a bad dependency, as I mentioned, you can use a bad compromised package repo and so on and so forth. 

[00:14:40] Guy Podjarny: The vast vast majority of attention and practical use of software supply chain security topics is around open source usage, and the reason for that is that if if a vendor has written some piece of software and they have a vulnerability inside of it, then It really is a very localized problem, and it comes back to the typical trust that we put in our vendors, which is, hey? Just kind of write, secure software and and be on the hook of being compromised.However if a vendor is using an open source component, the consumer of that might actually know elements, or have an opinion about how secure or insecure that component is, and all they really needed the vendors to sort of inform them about the component that is inside. 

[00:15:21] Simon Maple: A lot of what you are hearing around open source security isn’t really new. We’ve been using open source. We’ve been using third party libraries and commercial products for a  long, long time now.  We’ve had things in pipelines for decades now as well. So why is there all of a sudden a greater concern? Has there been something that really sparked either the attackers to target supply chains, more, or something that triggered within organizations perhaps a, a regulatory or a compliance related need?

Guypo and Nicole Perlroth, author of the New York Times bestselling book “This Is How They Tell Me The World Ends,” had some interesting insights on this topic 

[00:15:59] Guy Podjarny: We need to understand how we got here. I think there is sort of the slow burner and the fast flash that combined together. The slow burner is that over the many years the amount of proprietary software versus sort of open source software or consumed software that we have in our applications has greatly changed. It has gone from, some twenty thirty years ago. If you were to sort of buy a software product, the vast majority of it was inside of that was written by that vendor and the supply chains were basically less complicated. 

While today, if you buy a piece of software, it’s most likely that ninety percent of the software that is inside is not actually written by a vendor, but they themselves consume that out of open source components or their own upstream vendors. This supply chain, just like in the physical world, allows us to produce more, to be more productive. It’s a specialization, but it means that the risk of any one of these components being weak has increased, and it’s very similar to maybe what we’re seeing now with how the pandemic has affected supply chains, physical supply chains in the world, 

You know, there’s this chain of dependencies that over the years we have optimized to be most productive in, because, every part of the ecosystem knows how to produce their piece and sort of the cheapest and best way. But it also makes it fragile, and once one of these pieces gets out of balance it throws the whole system into a loop. 

One trend that has happened is really that software is more dependent on supply chains, and so software supply chains are just a more of a focus of attention, and that has happened incrementally over the years. 

And then the flash kind of a event that has happened to doing. It is just this series of breaches, and specifically Solar Winds, I think, was a big eye opener for people  Nicole Perlroth on the interview here has kind of called it, the watershed moment has equated it to maybe the discovery of the Stuxnet that really shown the light to the world around offensive cyber from nation states, and the fact that is a thing. And so I think what solar winds has done, especially because of the high profile targets that it had and the fact that it was, presumably, had the nation state behind it. It opens people’s eyes to the problem that occurs. We know for a fact, it’s not that these attacks on supply chains are new, but once you have something that has this type of high profile. It triggers things into action.

[00:18:23] Nicole Perlroth: This attack really exposed our blind spot, because they staged it at servers inside the United States where the NSA can’t look. It was only discovered, because Mandiant or FireEye discovered that they had been breached. Out of the goodness of their own hearts, disclosed that breach of – it didn’t touch PII, Personally Identifiable Information, which would have triggered state data breach notification laws. They raised the red flag and thank God that they did.

Then in rewinding the tape, discovered, okay, they started with the update that we did from SolarWinds, and we’re able to flag that for SolarWinds and for everyone else. Otherwise, we might not have ever known about this breach. They were inside our systems, I believe, for as long as nine months before anyone knew about it. The final thing I’ll say about SolarWinds is what was stunning to me as a reporter was calling up all of these companies listed and government agencies listed on SolarWinds website as customers and trying to confirm that they had been impacted, or were at least in the process of trying to forensically understand whether they had been impacted. Half of them didn’t even know that they had been using SolarWinds. 

[00:19:38] Simon Maple: Now the solar winds attack, of course, in 2020 really got the world’s attention. Everyone was thinking about software supply chain from that day, and we were grappling with the challenges of managing third party vendors and the risks associated with them. Of course, after that, in 2021, we saw a further breach, the CodeCov breach, and we had the of having Jared Engelberg who is the CEO of Code Cov and Eli Hooten, who is the CTO, and they shared with us some of the insights on that very breach. 

[00:20:07] Eli Hooten: Yeah, so what the attacker did essentially was enumerated the environment where the CI was running, basically running an ENV command, to see environment variables printed. The issue that arises is that in some CI pipelines, these environment variables may represent secrets or credentials that access other parts of a company’s infrastructure, right? Perhaps maybe, as for example, there is an environment variable that’s a token to access AWS, or some infrastructure, maybe there’s a Google Cloud Platform service account credential, right or something in there. That can then be extracted from the environment and use in a follow on attack, specifically targeted at that company.

The ENV was taken and it was piped to a third party server where the attacker could then review later at their leisure. So how that impacted customers was depending on the configuration of their CI, what they had in their environment, how things were set up. They could leak credentials to this attacker.

Of course, if you’re an open source library, and you’re used to your CI being totally open and everything being visible, you may not have anything of consequence in there, there might not really be any real reason to worry about it. If you’re in a closed source case, or you’re potentially putting a lot of secrets in your CI or CI processes doing a lot of work across your infrastructure or your stack, there could potentially be quite a few sensitive secrets or environment variables there, that will be concerning it leaked.

[00:21:27] Simon Maple:  So this all sounds like the perfect storm as a society. We’ve really changed how we’re building our applications, and I think  there’s a cultural piece here whereby, you know, many years ago when certainly I was building software, if I wanted to introduce something back into my pipeline or an application like a third party library I would have to stop and think about it much, much more back then because, It was a far rarer, less common thing to do compared to today when a developer will happily add a new library, and just because it, it performs the piece of functionality that they need, it’s a much more well  trodden path today compared to before.

And software obviously is relying upon open source much, much more these days than ever before. As, as Jim Zemlin shared, society is relying on software

[00:22:12] Jim Zemlin: In terms of ranking it from a cybersecurity perspective, it’s pretty significant. In terms of security in general, or like national security and so forth, it’s pretty significantly high. But in context of the conflict in Ukraine or other security issues around the world, it’s adjacent. And we’ve certainly seen physical conflicts in places like Ukraine, also have a cyber component to it. I think a lot of people, when they think about cybersecurity risk, they think of it as non-impactful in a physical sense. And that’s likely a somewhat naive assumption. And that, certainly, if you’re in an active war zone, this is like your top less frightening and horrifying situation. 

But if you’re the victim of a cyberattack that cuts off fuel, and electricity, communications, and critical infrastructure that you rely on, let’s say, in a winter time environment, that also can be immediately physically threatening to your security as well. And as society has come to rely so much on software to manage critical infrastructure, energy distribution, transportation services, telecommunication services, medical devices, and so forth, cybersecurity and physical security are starting to converge. And so that’s, why as of late, it’s critically important for us to collectively secure from a cybers perspective, both software and digital components that it runs on.

[00:24:01] Simon Maple: So we spoke about how visibility into open source libraries is a big problem. Particularly when looking at something like Log4Shell, when that happens, a lot of organizations would probably question how much visibility they actually do have over their current estates and understanding whether you actually know everything that you’re seeing or whether there’s some truly hidden bits under the covers.

Brian Behlendorf  who is the general manager of the Open Source Security Foundation, shares his views on open source software and how this has changed over the years. 

[00:24:35] Brian Behlendorf : Well, I think open source software in particular, but you can say this with the whole of the software industry, came about at a time when it was small enough, and the Internet was small enough, that everybody could had a reasonable shot of at least getting to know software by getting to know the people behind it and getting to understand how teams work and all that kind of stuff.

So we operate in a very high-trust environment. And we just don’t have that environment anymore. So some people might argue we never did. But the cost now of blindly trusting things you download off the Internet or things you find in the GitHub repo or others to be truthful to the label on the packaging of what it is. That trust is now under a lot of threat right now. We also tended to default trust that software developers, open source developers, were taking certain proactive measures, that they would respond quickly when somebody found a security hole in their work or whatever. And that’s not always the case. There are a lot of folks who aren’t as diligent about fixing things or where there’s code that’s been used that hasn’t been updated in a while. And maybe the original developer has moved on to other things.

And so this is something that now we’re just finding so many more places where that kind of default trust is being taken advantage of, whether in things like typo squatting in package environments, or dependencies that aren’t getting updated as frequently as they could be or should be, or other just kinds of behaviors in terms of how the software gets built. I mean, there’s so many single developer packages out there and places where something like Kubernetes pulls in thousands of underlying dependencies where we can’t scale based on the social tools that we had before. 

[00:26:03] Simon Maple: The increased adoption of open source in our applications adds complexity to our software supply chain security. However, open source is still just one piece of the overall software supply chain picture. As Jim Zemlin explains, there’s a lot more to software supply chain security than first meets the eye.

[00:26:24] Jim Zemlin: The second thing that happened is you now have a lot more critical points in that software supply chain with these package managers that have not matured as quickly as people who are trying to engage in cyber attacks have. Cryptographic signing for a lot of these package managers really wasn’t a thing, and in some cases still isn’t a thing for a long time. And they really don’t have fundamental security in those components as well. So now we’re not just looking at what are the most critical open source projects. We’re looking at, “Oh, how do we shore up the distribution mechanisms, and the CI/CD infrastructure, and version control systems?” This thing gets a lot more complicated a lot more quickly. 

[00:27:07] Simon Maple: Software supply chain security is not only complex because of the various components that are part of it, but also because of the scalability of the supply chain attacks as Eric Brewer shared.

[00:27:17] Eric Brewer: The important shift for supply chain security is that an attacker can take one of those vulnerabilities and turn it essentially into a repeatable attack machine, where by exploiting a vulnerability in the supply chain, all the groups that use that supplied software, or those tools, now have their things in jeopardy. And so a good supply chain attack can get 20,000 victims with 20,000 sets of credentials to do further attacks. And so there’s a great widening of power through this kind of attack, which is why we’re going to see more of it.

[00:27:55] Lena Smart There’s so many different definitions out there that what I tend to do is I go back to my government roots and I went back to NIST, because I love their definitions of things.

Really, I took what they had defined as supply chain security and distilled it into the elevator pitch version. Processes that protect and secure the efficient integration of suppliers to allow commodities to produce, to be produced and distributed as planned. I mean, there’s a lot of big words and it’s a bit of a word salad. If you actually look at it and digest it, to me that’s exactly what supply chain security is.

If you don’t have policies in place, it’s not going to be efficient. If it’s not efficient, you’re not going to be able to integrate anything, because if there’s inefficiencies, people don’t want anything to do with that process. If you don’t have any efficiencies, you don’t have any suppliers, you have no supply chain. It falls apart very quickly. Also, I think the important work there is to allow commodities to be produced and distributed as planned. There has to be a plan in place. I don’t know what, these sound all very matter of fact and common sense, but that sentence, that elevator pitch probably took me maybe an hour or two of just actually thinking and drawing out and writing and having a clear understanding.

I don’t know if you’ve ever heard any of Richard Feynman’s books, the physicist, but he had this whole thing about where you can choose a topic and you can study it. The first step is explain the topic to a child. I wanted to try and explain supply chain security to my mom, who’s not a child, obviously, but she’s not in security. I told her this. I said, “Do you understand what I’m talking about here?” She’s like, “Yeah, I do, actually. Yeah.”

[00:29:39] Simon Maple: As Lena Smart, CISO of MongoDB said, there are many definitions of software supply chain and we have had the benefit of hearing some great ones on the podcast. We do hope that after this episode, similar to what Lena did for her Mom, we were able to unravel what a software supply chain is for you, with an overview of its core components and why we need to pay attention to it. In the next episode, we will be talking about some key terms you need to know and some prominent players and projects in this space worth knowing about. 

[00:30:38] ANNOUNCER: Thanks for listening to The Secure Developer. That’s all we have time for today. To find additional episodes and full transcriptions, visit thesecuredeveloper.com. If you’d like to be a guest on the show, or get involved in the community, find us on Twitter at @DevSecCon. Don’t forget to leave us a review on iTunes if you enjoyed today’s episode.

Bye for now.

 [END]