Malicious Packages and Malicious Intent

EPISODE 123

“It’s not like we’ve seen a lot of those types of squatting and malicious packages happening on recently, or things like AI and Ruby gems. It’s definitely not one of specific ecosystem that people target. It’s, I think, just the lucrative nature and just low-hanging fruits where a lot of JavaScript developers do this, it’s a front-end, it’s a back-end thing in the attack surface for it, like the impact radius is pretty large.”

[INTRODUCTION]

[00:00:29] ANNOUNCER: Hi. You’re listening to The Secure Developer. It’s part of the DevSecCon community, a platform for developers, operators and security people to share their views and practices on DevSecOps, dev and sec collaboration, cloud security and more. Check out devseccon.com to join the community and find other great resources.

This podcast is sponsored by Snyk. Snyk’s developer security platform helps developers build secure application without slowing down, fixing vulnerabilities in code, open-source containers and infrastructure as code. To learn more visit snyk.io/tsd. That’s S-N-Y-K.IO/TSD. 

[INTERVIEW]

[00:01:17] Guy Podjarny: Hello, everyone. Welcome back to The Secure Developer. Thanks for tuning back in. Today, we’ll do a bit of a special episode here focusing on all things malicious libraries. To help us understand this interesting world, or evolution is Liran Tal, who heads up developer advocacy here at Snyk. Liran, thanks for joining us here on the show.

[00:01:36] Liran Tal: Thanks for having me, Guy. It’s fun.

[00:01:38] Guy Podjarny: Liran, help us contextualise here first, and talk about – we’re talking about these malicious libraries. What is a malicious library? Maybe, can you give us an example of such a case?

[00:01:49] Liran Tal: Sure. Essentially, a malicious library, or a version of a original militia of an original package is probably DOTS, which was created with an intent to specifically attack and target a specific profile of users. If you had installed it, it’s probably trying to do harmful things when you do that. We can probably go back a few years to one of those very high-profile incidents that happened on the NPM ecosystem, with this package called EventStream, which is an interesting case of malicious packages, because it wasn’t the normal case, which we see today, which is NPM install DOTS on the registry. Which we can talk about it in a bit.

If we focus for a second on the EventStream, I think that’s an interesting case for a specific attack profile has been done there. EventStream itself is package that existed on the NPM ecosystem for four years, probably a decade or so, before this whole thing happened. It’s not something we usually as developers know about, because it’s one of those transitive dependencies, which is very invisible in your projects.

[00:03:03] Guy Podjarny: This is a library that is used by a library that is used by a library that you drop. It’s like, somewhere in the dark corners of your dependency tree.

[00:03:13] Liran Tal: Definitely. Now, it’s been maintained over the years, just fine. One of its maintainers, Dominic Tarr has been very prolific developer on the NPM ecosystem. They had something like 400 NPM packages. They were very good citizens of open-source and the NPM, and the JavaScript ecosystem as a whole. To an extent, I think, you could think of this package, like entering into maintenance mode, because it didn’t really have much releases and code going into it. It was just fully functional, fully completed, you could say that after about 2015, or 2016.

By 2018, someone goes into this open-source repository, and they – and other issues, or pull requests, they try to get engaged, they open issues, they close some of them. They’ve actually been trying to interact with some maintainer and help them resolve issues. The maintainer, Dominic, is a super busy person. You could imagine doing a whole lot of things in his life pulling open-source. At some point, this other person, which got the nickname of rights control is the handle, they started getting more and more involved with a project. To an extent, one of those involvement was to get a specific functionality onto EventStream.

When they did that, they did that as actually fixing, or adding a feature but then modularising its own NPM package, which now they control. They did that. They added it and now this extra package is now part of EventStream. From this day on, when EventStream gets installed, it actually pulls in this other package. They’ve done that. They’ve gained the trust of Dominic and this package. They’ve also been able to get access to specific merge source code changes and publish new packages and all of this, because they were acting as a good citizen.

Only problem though, is – a bit of a curveball here is essentially, open-source ecosystem is, since they control this, this other package that now is part of the EventStream that it pulls when it gets installed, they have added a specific backdoor trojan into it, which targets a set of very specific targets of developers of a specific profile. That actually happens. Now, since they’ve released this new version of that sub-dependency, which is again, a dependency of a dependency of a dependency, that gets also installed. New installation of EventStream that actually pull new versions of this other package called flatMap stream, which is now malicious, and has some code that potentially could be harmful for me, or you, or an Android developers who have installed it.

[00:06:00] Guy Podjarny: This is a new installations, but also, I guess, very common in the NPM ecosystem and in many others to take the latest, as long as it’s a patched version, or not a breaking change. For many people, who never knew that used EventStream before, and never knew they used EventStream after, until they learned about the malicious behaviour. They just naturally upgraded without any provisions rebuilt, and they naturally got the some of the updated version, right?

[00:06:27] Liran Tal: Yup. That’s the story. That’s how people get this malicious versions of packages onto their dependency tree.

[00:06:35] Guy Podjarny: Yeah. Super interesting story. They’re on it. It’s interesting to hear about the sophistication of the attacker. It’s not like, this isn’t really rocket science on it. it’s just basically, be a good Samaritan for a bid on an open-source project that is popular enough, but low activity enough that you can bubble to the top, or you can gain confidence, and especially that element of adding a dependency, which is really super common to compartmentalise capabilities.

[00:07:04] Liran Tal: Yeah, especially in open source, and in JavaScript, which is importing modular.

[00:07:09] Guy Podjarny: This is at the time EventStream, and also the time that a lot of the concept of typosquatting came to light a fair bit as well. That’s another avenue of attackers getting a malicious library in. Can you say a couple of words about what that is?

[00:07:21] Liran Tal: Yeah, sure. Typosquatting is a sense of you have domain squatting. Essentially, people, to begin with, I guess, open-source ecosystems like NPM and others are open by nature and inviting and welcoming, so they allow anyone who just have an email and publish new packages to the registry. If you take an advantage of that, and telling the malicious sense of it, I could actually create a package with a very small typo. If I want to target React developers, I can maybe have React with a double E, or something. If I’m able to publish that to the ecosystem, to the registry, and that is doing something maybe malicious behind the scenes, and potentially someone just have a typo, or it’s been typo’d in a blog post, or a StackOverflow comment or something, and people just now blindly go and do NPM install for that React with a typo or something around this, then they end up with no controls, just essentially, just blindly installing this malicious package.

It’s easy to just have those typos. For some, actually, for – I think we actually have very popular ones. It’s hard to imagine someone literally typoing that. If you think about other naming conventions, say React Native, that tool chain, is it with a dash? Is it without a dash? To count towards and stuff like that. Those have been very popular in terms of just typosquatting and that find the ecosystem. Again, this is something that transcends NPM. We’ve seen these happening on for Python developers and Pipe BI and other ecosystems.

[00:08:55] Guy Podjarny: Yeah. We have a properly malicious library that injected itself invisibly for the consumer, then we have the typosquatting is another path of it. I guess, just to wrap the try over here, there’s also the whole dependency confusion later. Can you explain that and what’s dependency confusion?

[00:09:12] Liran Tal: Yes. Dependency confusion is a fairly new concept of attack that we’ve seen since last year. Essentially, it’s been brought into existence, or awareness from a research by this developer, or group of researchers called – one of them is Alex Birsan. What he has been actually explaining to us is some companies, quite a few I guess, they have their own IP and they manage their own libraries internally, which makes sense. That everything is truly out there in the open. When they do that, they may use package naming convention that are then published and used internally in their own private registries, in their own hosting ecosystems.

When they do that, it presents some potential issues. This is kind the story of how it goes. If I have a package that has a specific name, like Azure-something, something, and that name is actually internally used by my own team, but this name of a package is actually free to register on something like the NPM registry. That’s a free namespace over there, then at this point in time, now, several things come into play. If you have a misconfiguration of how your dev environment is set up, and you simply go off and do NPM and you install Azsure reduction of something, something name of that private package, then you essentially – a few things are happening.

By default, the NPM package manager is going to look on the upstream NPM registry, if there are new versions of it. If a new version, like 100.100, using really upwards, really high major versions. If that exists on the NPM registry, then the NPM package manager would actually go ahead and take that version, and prefer to use that one over the one that actually exists in the registry. Several things come into play.

[00:11:12] Guy Podjarny: It would prioritise the public version of the library, or the older version, private version of the library.

[00:11:20] Liran Tal: Exactly. Several things come into play. We have this specifically is a version, a name for a package that has its namespace, like ‘free to register.’ That needs to happen first. The second thing is, yeah, someone is able to register that tree namespace. The third thing is you have misconfigured. You could specifically have your internal upstream version of packages actually deny access to the NPM one, but we’ve seen a lot of just native installations, which don’t have this dock and feel more the configuration flowing into developer machines. This misconfiguration, which is very – it’s like a duck profile of misconfigurations in the cloud, but it’s actually very much relating here to just how you manage package dependencies. There’s a bunch of things that need to come into play.

This has been quite an issue that we’ve seen. We’ve actually witnessed attacks by pentesting companies, like security companies that are actually trying to simulate these kinds of ways of attacks with dependency confusion in order to penetrate a company and to infiltrate it as part of doing some security and a red teaming, or pentesting from the outside, which shows you and how this has been successful. It shows you how this is a very, very high-profile attack, and also, a low-hanging fruit that might get you into trouble.

[00:12:46] Guy Podjarny: Yeah. Because really, all you need to do is you need to guess the name. As an attacker, all you need to do is guess the name that an internal package might have, and register that name. Sometimes you don’t even need to guess, because there could be some open-source, other library that is open-source, but actually mentioned that name.

[00:13:03] Liran Tal: Front-end libraries that might be making that and all of those kinds of things.

[00:13:07] Guy Podjarny: Yeah, or norms, or people copy pasting the names, or the practices for something public. Thanks for the overview here, Liran. We talked about a little bit about what is a malicious library. Simply put, it’s just that and it’s a library that has a malicious intent beyond what you’ve intended. Then we talked about the three different paths. It’s not a comprehensive list of ways in which you can get into someone’s, a developer’s machine or in an application, but it’s – they’re probably the most common ones. We’ve been talking about NPM a lot here in the examples. It’s a pretty common punching bag, but is this problem of malicious libraries exclusive to the NPM world, or does it happen to elsewhere as well?

[00:13:46] Liran Tal: It’s not. We’ve seen a lot of those typosquatting and malicious packages happening on recently, I think, by API and RubyGems. It’s definitely not one of a specific ecosystem that people target. I think, just the lucrative nature and just low-hanging fruits for a lot of JavaScript developers, they do this, it’s a front-end, it’s a back-end thing. The attack surfaces for it, or the impact radius is pretty large. It is an easy target and easy to execute.

[00:14:15] Guy Podjarny: Yeah. I guess, especially the example of EventStream, that was just basically an open-source maintainer doing something. You’re downloading a library in a very standardised way and within the rules. There’s really no loophole here. It’s social engineering effectively that drives that, combined with the dependency chain reality.

We talked a fair bit here now about the types of ways it can get in. How one-off are these cases? We talked about EventStream, or specific examples. Is it common? Are there a lot of these malicious libraries around? Or is it just a handful of scary events?

[00:14:50] Liran Tal: It’s pretty common. I mean, there is definitely typosquatting and malicious packages. In general, they happen quite a bit on the ecosystem. I’m specifically following the JavaScript one. With the Snyk security database, now full of them every week. People definitely take advantage of that and just push some of that into it. That’s happening. I think, there’s a lot of research going on here. We’ve actually looked into dependency confusion specifically. We’ve done research, like find out more about this. What we did find out is specifically, the dependency confusion thing, very, very, high profile doc. This is something not so long ago, was a research from May 2020, there’s a viewpoints back. Found out more than 200 malicious NPM packages. One of them actually, including this cobalt slice, backdoor trojan, as part of a dependency confusion attack.

They’ve all been doing different things. One of them within that, just sending information, grabbing information from the machine itself, exfiltrating that over HTTP, or DNS requests and sending it out. We’ve seen cases where it actually opens a remote reverse shell within the cobalt slice time version of it as well. It’s definitely something that is always happening, and it’s something to be very mindful for.

[00:16:14] Guy Podjarny: Yes. I mean, definitely, it feels like now, it’s almost a regular occurrence for whether it’s the open-source researchers has had some research like that, or Snyk, or other researchers find them in droves and bulks of it. Before we deviate a little bit into what you can do about it and such, we’ve been talking about the malicious libraries, as you know, there was some explicit adversary, like ill intent, bad person villain here, who came along and tried to dupe us all. There’s another interesting spin of this whole protest where, that we’ve seen recently. Do you want to give us a bit of this example? I think it’s an interesting story.

[00:16:51] Liran Tal: Yeah, definitely. has been a term that has been coined for a recent events relating to the Ukraine and Russia war. You can go back even probably seven or so years ago, and go back and see. One of the original cases, which was so popular, and that’s called left bad. We have a case, it’s channelstrip library, effectively, probably some 30 lines of code. Not really big, but gaining millions of downloads. It’s literally just bad trains with a space on the left. That’s a little –

[00:17:23] Guy Podjarny: You get a lot of Java developers making fun of JavaScript developers for having a library that pads a string. Let’s put that aside for a sec.

[00:17:30] Liran Tal: Sure. I also feel like you’re trolling me a bit, guys. I won’t fall into it.

[00:17:34] Guy Podjarny: I’m just saying, the conversation happened. Conversation happened.

[00:17:37] Liran Tal: Of course, you are. Yeah, I mean, that’s what it does. It’s been useful. As you can guess, it’s probably again, a dependency of a dependency of a dependency of something that you use in your project. What happened there is actually, not even relating to left-pad as a package. The maintainer of this library maintained, again, hundreds of other NPM packages. He was approached by a bunch of lawyers, because one of the packages that he was using has a specific name of some instant messaging app that this other company is doing. They wanted to claim that namespace.

They approached him. They asked him to let it go. There was a whole discussion with the NPM team. They just couldn’t consolidate, or reconcile on a specific, what we do about it. They ended up giving up that namespace of these other package that they control to this company. As an act of protesting, this developer just yanked and pulled all of their packages from NPM. It’s just that unfortunately, one of those was left-pad, and no one had foreseen that. This is something that people did not foresee in the ecosystem, where big enterprises in general were not ready for this.

You do an NPM install in your CI, you get all your packages, and suddenly, one of them doesn’t exist. What do you do at this point? It did end up breaking a lot of packages that were dependent on it, like the NodeJS project was dependent on it and some sub-packages. Other CI were breaking around the world. This was on some newspapers, online media how one developer broke a lot of the Internet in a sense.

[00:19:08] Guy Podjarny: Yeah. It inspired the graphics, space of the graphics that are showing how the Internet is dependent on this little tiny pebble that is –

[00:19:15] Liran Tal: As far as central Nebraska.

[00:19:16] Guy Podjarny: Yeah, exactly. I think, just before we overlay and pick on the NPM ecosystem, there was a very similar case happening in the Chef Ecosystem, with dependencies where a chef named said Fargo was annoyed with the Chef’s business with ICE, with immigration entity at the time in the US. And as an act of protest, just pulled a very popular used Chef recipe component on it.

[00:19:38] Liran Tal: Yeah, a Ruby package.

[00:19:39] Guy Podjarny: As a Ruby packet. That also posed some fairly reel of it. Okay, so interesting. These are acts of protest by open-source maintainers using their reach to draw attention to an issue that is relevant. What was the latest one? I think that was interesting.

[00:19:56] Liran Tal: March, we got, the world was spinning, through this crisis between Ukraine and Russia. We’ve had this developer of this very fairly popular open-source package called node IPC. I did not know about it either, because it’s again, one of those dependencies of dependencies of dependencies. Never used it directly in a project, but it’s used millions –

[00:20:17] Guy Podjarny: That’s an important point to maybe to reassess, and it’s probably everybody listening can notice the pattern here, which is, generally speaking, these issues happen not in the component you’ve chosen to use, but sometimes it does, but oftentimes, it’s in something deeper in the ether there.

[00:20:34] Liran Tal: Yup. It’s not like I have millions of downloads. Apparently, it’s been used by a lot of packages. A lot of other packages are placed. What they had wanted to do is take action for in the face of this crisis, and added a specific line of code, or lines of snippets for this node IPC module, where when it’s being used, it actually starts – spins off some timers on the machine it’s being run on, and they effectively traverse these. All of the filesystem, everything on the file system on each file, and just knobs them. Actually, not just empties every file, but adds a heart emoji into it, which is a bit ironic for this thing to do that.

This was an act of protest against the whole crisis thing that was going on. They specifically had some API request to get location and to see that it only targets developers, or installations in Russia, in some other countries, etc., as to protest against this. This really happened. Definitely, when we tracked, we looked into it and I think we can responded pretty quickly take for event. When we did look at some of the numbers and the forensic of what afterwards, what happened, the whole postmortem, there was about 4,000 installations for December before that version. It is very likely to estimate about 4,000s of installations, and by this specific malware.

It actually went on to get developed into even a further issue, because then they replaced that malicious code that specifically targeted those machines and though they did, and maybe destroyed, or destroyed if you had installed it, which they move by the way, and then call it a new version again, showing us the cases of dependencies and managing versions and new releases and stuff like that.

They replaced that with this other package called ‘peacenotwar’ which does something less harmful, but the library was unexpected. It creates a file on your desktop, when you install this. Some protests were messages, calling for peace and all of this stuff. Which again, I wouldn’t want any open-source package installed to do this, so it could be then expected, yet not inherently malicious. What we’ve seen is they added this functionality as a dependency of node IPC there. Then, also released that version of node IPC, which is peacenotwar, as a minor version to one of the most downloaded major version of node IPC. Now, what this mean is this isn’t just everyone who uses the latest major version, which wasn’t very popular, but also, all the upstream projects that are dependent on node IPC. This has been very, very popular projects, like the Vue.js project that is CLI and that’s dependent on node IPC.

There’s a bunch of others, like the Unity game engine, I think has dropped some electron application and this is node IPC as well as part of it. That’s also being known to be – there’s form bulk of users bring things very talent right then, then there is about it. They say, I just think next, next, next, until it’s all finished, but suddenly, I get surprised by this new war, or crisis related thing. You file on the desktop, like, what do I do about it? If it’s a virus, whatever.

[00:23:56] Guy Podjarny: Right. These examples of protest actions are really interesting, because they’re not – They challenge a little bit the term ‘malicious’. It’s not really, I think, for much of the Western world, it’s not a type of action that they would deem malicious. Many people would relate to the goal of the protest. It’s also problematic. I mean, how do we think about that? Is it right to call these malicious libraries, just like we do the rest? Is it not? Do we have a different term for these, or to treat them differently? Maybe also related a little bit, what happened with these libraries, for instance, with the registries, that the registry is – I know that for a lot of the malicious libraries, the registry has deleted them. In these cases, is that also the case that they delete them?

[00:24:41] Liran Tal: I think some of them are a very fine line of what do we allow and what not. I think, the original node IPC source code that actually goes and delete files from your desktop, that is super malicious. There’s this, regardless of the circumstances, but the outcome of that is very harmful. That is by definition, again the terms of use for the NPM registry. That was very quick to get removed. It’s not where the module was essentially, you could install that module directly like an NPM, and you go peacenotwar.

Once you’ve been installing that, you get a new file on your desktop. You want to call it a feature, that’s fine. It’s probably not that’s malicious, but it’s an unexpected turn of events. I don’t think you’d expect that would be happening. That’s like, I’m not entirely sure where that holds within terms of use for the NPM registry is probably going to build very grey areas, or you might want to remove it, and then it’s up to them in general about potentially what they’ve done. That’s for extent. I think for many of the malicious packages that are obviously malicious, where they’re like, dependency confusion, or even no red teams, red teaming on companies open-operation testing, companies doing dependency zero. Out of that was calling, or someone or waiting to get in, and just publishing that to the NPM registry, because they want a way to get into the company.

Those are, again, there’s probably against the terms of use for what you would want to put into the NPM registry, because that’s actually for public consumption. It’s not like they’re publishing this retail company.

[00:26:16] Guy Podjarny: Liran, these are super interesting stories. Sometimes I like to say that security is boring, but hacking is fun. Sometimes the stories around them are curious and all that. But they’re not. At the end of the day, when we’re doing our job as defenders, we want to do something about those. Maybe we spend the last part here of the podcast, of the show, talking about what we can do about them. Maybe, start us off. This is happening. Attackers are now noticing this attack avenue more, or at least maybe we’re more conscious of the fact that they are. As a consumer of open source, and someone who cannot just not use open source, what can I do about it? What can people do, and organisations do?

[00:26:57] Liran Tal: Right. I mean, the first thing is knowing about the problem. This is usually 30% of the solution. We say, it’s knowing that you have this issue. Obviously, when those events are happening, and by the way, that’s usually within a delay of a bit of time. The media breaks out, and you find maybe press on Twitter, or Reddit, but not everyone, everybody are very, promote the social media and all of those. I definitely had basic work, working with developers and they never heard about node IPC. They never heard about that.

While we think about it, there’s the media has talked about it, and itself of the headlines, news and stuff like that. Not everyone really knows about this. You should be using the tools to find those issues, or transitive dependency, direct dependencies throughout your project, or deployed packages, or apps. You should really scan and understand that when those things are happening, malicious packages, typosquatting, whatever is the case, you really want to find out about it as soon as possible, like the case number one.

[00:28:03] Guy Podjarny: Cool. That’s an important first bit, which is just know about the problem happening, which I guess, requires you to know which components you are using, right? Because especially if there are now hundreds of these, you’ll pretty quickly get alert fatigue and start ignoring just a laundry list of these are all the recently, all the malicious libraries found this week. You need some solutions. Hence, Snyk is clearly one of them. You can pick the tool of choice. Do you need some solution to track what is it that are the components are using, and that has the ability, or the capability to alert you when one of those things turns out to be malicious?

[00:28:39] Liran Tal: Definitely. That’s not an issue of like, there’s malicious packages. Also, generally, a good approach is the idea of being able to find them. I think, being able to mitigate them as fast as possible as well. Being able to mitigate a vulnerability is essentially using an upgrade in your version. You want a minimal, like extend the range possible to be able to upgrade really fast, without breaking the aptitude to make a version. Without them telling them how to do that. I think, we’ve got to really develop – potentially a lot of alerts and frustrations that developers have, but also, the fact that sometimes vulnerabilities and could potentially be invisible, because they’re happening in dependencies that we’ve done to ourselves. They’re not in the right dependencies.

They can also be invisible to other cases, where you have cases where your tooling might not find all the tools that you actually have today. Definitely, there’s a lot of – I feel like we’re transitioning into supply chain security as a whole here, but it’s definitely one thing to keep in your mind as you’re thinking about this entire landscape.

[00:29:44] Guy Podjarny: Yeah. I think those are a couple of super important things you said. First of all, one of the transitive things, because it’s oftentimes so frustrating that it feels easy to track which dependencies you are using, because you just look at the libraries that are in the file. I mean, there’s an easily parsable file that states which libraries you’re using. Except the reason, because those libraries in turn use others, that there’s others. A lot of this drama happens actually in the components that you pulled in indirectly, the transitive dependencies.

You said it well. Just feels like, it’s really worth emphasising whatever is the approach you’re using to track your components, just make sure you cover transitive dependencies. Otherwise, you’d basically be left fairly bare in terms of the coverage that you’re actually getting.

[00:30:25] Liran Tal: Yeah, definitely.

[00:30:26] Guy Podjarny: Yeah. We started off knowing what we’re doing, we’re covering the dependencies that we have and we’re making sure that we do transitive dependencies as well. We’re looking at vulnerabilities as well, because fundamentally, this is the broader supply chain security issue. We use some service that tells us about malicious components. Now we’re in that state. What more can we do to not get caught up moving forward?

[00:30:49] LT: Right. One of the things that I think is very characteristic of developer and especially in the JavaScript ecosystem is, we like living on the bleeding edge. We upgrade too quickly. We use the latest frameworks. Not a winter goes by without the new front-end framework getting born, which is, it’s fun. I love that part of –

[00:31:08] Guy Podjarny : Did I just hear you make fun of the JavaScript ecosystem, Liran, here? You know this is going to come back to haunt you.

[00:31:16] Liran Tal: Maybe. Maybe so. Yeah, it’s only good. It’s getting good faith. We love to experiment and do some of that stuff, a lot of new things and just love study as we go on. The thing is, it’s not always the best thing to do when it comes to security. When there’s a security vulnerability, you obviously want the weakest fix to just land in your pull request, and you want to merge it in and that’s – I mean, the whole thing. That’s a given everyone, or probably, no doubt and we’re receptive to this.

The thing is, you want to do a bit of the opposite when it’s just general upgrading. Here is the thing. If you had used node IPC, or it shows you BCDI. You were continuously updating new versions and stuff like that. We could apply it, by the way, to an event frame and probably, all the other malicious packages we talked about and have been in the ecosystem. What would have happened is that if you are always upgrading too quickly, you would also be getting those malicious burdens, because you’re living on a bleeding edge.

If you’re doing NPM install all the time in your CI, so the latest version, just because you want to test your application doesn’t break, or whatever the reason, you’re opening yourself to this. I think unnecessarily risk of developers that are getting these products were under these, or malicious attackers doing that for end users. Instead of saying that, what I propose is don’t always update too quickly. Just give us a bit of time out before you operate on the latest version.

Actually, it was with the security games, or do a whole research on this. We looked at the database timeline of what happened in prior cases. We’ll send out and protect those, those vehicle system liquidity, or security companies and actually, found out that there is malicious version to what’s happening within usually, days, maybe hours, sometimes even the updates for dependency version, even unrelated to the security issues.

When you want to update your dependencies, which make sense, what we do is we have this inherent delay of about 21 days. Before, we’ll propose you an upgrade of a newer version, of a specific version that you use. Maybe you want to get to the testing library, or something else, to have this inherent delay, which gives you a bit of buffer times. It’s essentially protected by the cover time to protect you in case something is really bad happening there, and give you some time to file everything else. Throw it out, curse this out and everyone to find out about it before something bad is actually happening to you.

[00:33:51] Guy Podjarny: Yeah. I love that hack. Really, in terms of the path. We are as an ecosystem, when we find a malicious component, we actually can deal with it centrally, unlike the vulnerabilities, which tend to continue to exist forever, because they need everybody to stop taking them. Malicious components actually get removed from the registry. If you lagged a bit, you let the world delay, or give the ecosystem an opportunity to assess those components and deem them safe before you actually consume it.

Naturally, some people will be pulling a library for the first time. There are definitely companies like Snyk that try to assess the library, so maybe others will find it that way. You really are doing it quite safe. I guess, that’s the type of walk that we have here, which is if you upgrade, it’s a risk. If you don’t upgrade, it’s a risk. Trying to be intelligent about it. If there is a new vulnerability, be quick to upgrade. If there isn’t, be slow to upgrade. Eventually upgrade. You don’t want this to go stale and make that eventual upgrade when the vulnerability comes along late, because there’s a similar one with log files. What’s your view around using log files versus not using log files here? Does that matter?

[00:35:00] Liran Tal: Yeah. It’s a very healthy debate in the ecosystem between maintainers. There are cases where you want to use them as an application developer, or whether you want to use them as a maintainer of libraries. There are definitely several use cases for them. I’m generally on the camp of always using log files, which is, it’s for several reasons. One of them is obviously, the deterministic builds. You know everyone on your team are always going to get the same version and nothing’s going to be runtime, or equal time I evaluated, and now someone pulls the rug from underneath your feet.

There’s also security issues that is told. This is some exact spaces, like between down with larger direct dependencies, but also your indirect dependencies. It’s to an extent, protect you from some of this stuff happening. Ironic or not, we have found out this at Snyk, we disclosed the research about potential security issues with log files and the way they work, because someone could inject actually, the machine generated log files on their own control URL resource of actually, where they get the cargo for the dependency. You should also be able to manage log files in a very secure way. I’ll say, we definitely should do that. Then use log files.

[00:36:15] Guy Podjarny: Yeah. I think, I’m in favour of using log files as well, but using them cautiously. I think, it’s also important to remember that there are a random, like for the case of malicious libraries, there are a bit of a random solution, because what would happen is, you’re using log file, so fine. If you just rebuild, you wouldn’t have gotten the malicious components. Then totally unrelated, you would update one of your dependencies versions, you will recreate the log file, because that’s what you do. There’s no way that you notice that this four levels down dependency has upgraded a patched version. You just take it.

Unlike the maybe more thoughtful control that says, hey, if it’s 21 days, don’t take it, unless it’s 21 days. Such those types of upgrades would just – they will just update the log file, and you would just be as blind. You added a little bit of randomness to upgrade, you’re a tiny bit safer, but it’s probably not the right strategy to address this.

Good. Okay. If I recap a little bit for the consumer, know what you’re using, make sure you cover the transitive dependencies as well. Be subscribed to something that tells you whether any of these things are malicious, so that you’d know about that quickly. Of course, make sure that you address vulnerabilities in those components as well. It’s all a part of supply-chain security, and institute a smarter upgrade approach, where if there isn’t a vulnerability, maybe delay, maybe wait a little bit before you upgrade. Give the ecosystem an opportunity to flush out some of these malicious players. Any other key tips here about what you could do?

[00:37:38] Liran Tal: Yeah, I think there’s more. There’s some other things in there, like you probably want to be more mindful to the fact of health. Essentially, the idea of we’re somewhat present to the node IPC and the chord, the paper, like the fact of all the little details. Their community around them, they just want to get supported. You’re like, what they’re really going to go for it, they’re just going to be deprecated in six months, and you could have known now. Now, rather than finding out about it when you’re already with that stuff. Now, I can build, even finding a tooling that helps you find out about getting an ethical score, like say, 50 units a second.

[00:38:12] Guy Podjarny: I give similar advice. Although, it’s worth noting that it’s a bit hard as well to think about, okay, if again, if you knew that, some dependency four levels down, had some poor maintenance practice, or has a new maintainer, how really able you are to do anything about that. You want to listen for primarily, for the glaring problems. It’s worth noting that services like Snyk will – we notice those as well. If some funky action, or maintainer join some repo or such, will monitor them as well to tell you if we deem any of those activities malicious.

[00:38:42] Liran Tal: Yeah. That has been part of this new open stuff, new foundation and you build those for years from, what’s been going under.

[00:38:49] Guy Podjarny: Yeah. I think that’s an important addition, indeed. Because a lot of what we’ve discussed was what maintainers, or not maintainers, but rather, consumers of open-source components can do to protect themselves. I think the only thing, by the way, there that maybe is worth adding is to just prepare, play through the scenario of a malicious component turnout in your system, practice the incident response. I think that a lot comes well, the episode with cold cove was talking about themselves compromised, but I think in that episode there, they shared some great practices on that.

Then there’s the, what can the ecosystem do about that? There’s a lot of work happening right now in the open-source security foundation, or OpenSSF. We’re going to have a bit of a miniseries about this topic here in this podcast in the coming couple of months. We’ll cover a lot of activity there. There’s definitely work happening in the ecosystem to try and secure the pipelines and have better at the station, better signed affirmations that the data you’re downloading is indeed what you think it has some propagation of information about what has been properly signed and such. In some standardised — there’s a scorecard project that helps standardise the way that we assess repositories. A lot of those come along. They’re all very nascent in the sense that there’s great technologies and they’re advancing very rapidly. Great conforming, and so a lot of the industry trying to come together and create a standard.

There’s nothing really, like as a consumer of it, most of it, unless you’re a little bit higher end, isn’t as immediately consumable for you when it comes to malicious components. I think, watch that space. I don’t think it’s the most complete actions that we have. This has been a great conversation. Thanks for coming on and doing it. There’s a lot of stories and interesting stories to cover here. As you point out, thank you for doing all the great work and raising awareness to it. In general, people need to know about this problem and be mindful of it, so they try not to fall for it. Thanks for sharing all the knowledge.

[00:40:40] Liran Tal: Thank you for helping us advance this knowledge and share this awareness with everyone else as well.

[00:40:45] Guy Podjarny: Thanks, everybody, for tuning in. I hope you join us for the next week.

[END OF INTERVIEW]

[00:40:52] ANNOUNCER: Thanks for listening to The Secure Developer. That’s all we have time for today. To find additional episodes and full transcriptions, visit thesecuredeveloper.com. If you’d like to be a guest on the show, or get involved in the community, find us on Twitter at @DevSecCon. Don’t forget to leave us a review on iTunes if you enjoyed today’s episode. Bye for now.

[END]